You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "aminghadersohi (via GitHub)" <gi...@apache.org> on 2026/09/01 16:41:13 UTC
[PR] feat(mcp): add typed Sunburst chart support [superset]
aminghadersohi opened a new pull request, #43771:
URL: https://github.com/apache/superset/pull/43771
### SUMMARY
Adds complete typed MCP support for the frontend ECharts Sunburst plugin using the exact `sunburst_v2` viz tag and the typed `chart_type="sunburst"` discriminator.
- Registers a typed Sunburst plugin/schema with hierarchy, saved/SIMPLE/SQL primary and secondary metrics, filters/time controls, ordering/limits, and presentation controls.
- Canonicalizes dataset columns and saved metrics case-insensitively, validates hierarchy/metric roles and output-label uniqueness, and round-trips native saved form data.
- Rebuilds Sunburst query contexts faithfully for compile, `get_chart_data`, saved previews, and unsaved previews, including the frontend-only `sort_by_metric` transform and secondary metric promotion.
- Gives `generate_chart`, `update_chart`, and `update_chart_preview` parity while preserving omitted saved UI/filter state and honoring explicit false/null/empty values.
- Adds schema discovery, tool guidance, recommendation metadata, ASCII hierarchy previews, and product-path/regression coverage.
Shortcut: [SC-119162](https://app.shortcut.com/preset/story/119162)
Agor worktree: [sc-119162-mcp-sunburst](https://agor.sandbox.preset.zone/ui/w/01a05dae50f0741f888abd48/)
Coordination: this is independent of the open typed [Bullet PR #43770](https://github.com/apache/superset/pull/43770) and community chart-plugin PRs such as [Sankey #43573](https://github.com/apache/superset/pull/43573), although those PRs touch the same registry/schema/docs surfaces and may require normal rebase sequencing. The merged generic fallback work in [#37969](https://github.com/apache/superset/pull/37969) did not provide typed Sunburst support.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable; this is MCP backend/schema/query support with no frontend UI change.
### TESTING INSTRUCTIONS
```bash
# SQLAlchemy 2.0.52 compatible environment
python -m pytest -q tests/unit_tests/mcp_service/chart/test_sunburst_chart.py \
tests/unit_tests/common/test_form_data_query_context.py \
tests/unit_tests/common/test_query_context_factory.py
python -m pytest -q tests/unit_tests/mcp_service/chart
python -m pytest -q tests/unit_tests/mcp_service
pre-commit run --from-ref origin/master --to-ref HEAD
```
The focused/query-context run passes 109 tests, the full MCP chart run passes 1,391 tests, the broader MCP run passes 3,725 tests, and all pre-commit hooks pass under SQLAlchemy 2.0.52.
Manual MCP verification when a server is available:
1. Call `get_chart_type_schema(chart_type="sunburst")` and confirm `viz_type` is fixed to `sunburst_v2`.
2. Generate a preview with a two-level hierarchy, primary metric, and optional secondary metric; inspect the Explore URL/query for both metrics, all hierarchy columns, and primary-metric descending order when requested.
3. Update the preview and saved chart while omitting presentation/filter fields, then repeat with explicit `false`, `null`, and `[]` values.
The branch environment did not expose a live server on `localhost:8088`; the automated app/unit suites above were used instead.
### ADDITIONAL INFORMATION
- [x] Has associated issue: [Shortcut SC-119162](https://app.shortcut.com/preset/story/119162)
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3906353803
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -1045,6 +1045,323 @@ def reject_sql_expression_on_dimensions(self) -> "PieChartConfig":
return self
+class SunburstChartConfig(BaseChartConfig):
+ """Config for the ECharts Sunburst plugin (viz_type ``sunburst_v2``).
+
+ ``hierarchy`` follows the frontend ``columns`` control: the first entry is
+ the innermost ring and each later entry adds a child level. The primary
+ metric sizes arcs; an optional secondary metric colors arcs by the
+ secondary/primary ratio.
+ """
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ chart_type: Literal["sunburst"] = "sunburst"
+ viz_type: Literal["sunburst_v2"] = Field(
+ "sunburst_v2",
+ description="Exact Superset frontend visualization tag",
+ )
+ hierarchy: List[ColumnRef] = Field(
+ ...,
+ min_length=1,
+ description=(
+ "Hierarchy dimensions in ring order, from the innermost/root level "
+ "to the outermost/leaf level"
+ ),
+ validation_alias=AliasChoices("hierarchy", "columns", "groupby"),
+ )
+ metric: ColumnRef = Field(
+ ...,
+ description=(
+ "Primary metric used to size arcs. Use aggregate for a SIMPLE "
+ "adhoc metric, saved_metric=True for a dataset metric, or "
+ "sql_expression plus label for a SQL metric."
+ ),
+ )
+ secondary_metric: ColumnRef | None = Field(
+ None,
+ description=(
+ "Optional metric used to color arcs by secondary/primary ratio. "
+ "When omitted, colors are categorical."
+ ),
+ validation_alias=AliasChoices("secondary_metric", "secondaryMetric"),
+ )
+ filters: List[FilterConfig] | None = Field(
+ None,
+ description=(
+ "Structured WHERE filters (column/op/value). An omitted list is "
+ "preserved on updates; an explicit [] clears saved filters."
+ ),
+ )
+ time_range: str | None = Field(
+ None,
+ min_length=1,
+ max_length=1000,
+ description=(
+ "Superset time range, for example 'Last year', "
+ "'2025-01-01 : 2025-12-31', or 'No filter'"
+ ),
+ )
+ time_grain: TimeGrain | None = Field(
+ None,
+ description="Optional bucket for temporal hierarchy columns",
+ validation_alias=AliasChoices("time_grain", "time_grain_sqla"),
+ )
+ sort_by_metric: bool = Field(
+ False,
+ description=(
+ "Order hierarchy rows by the primary metric descending before "
+ "applying row_limit, matching the frontend buildQuery transform"
+ ),
+ )
+ row_limit: int = Field(10000, description="Maximum hierarchy rows", ge=1, le=50000)
+ color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Categorical scheme used when secondary_metric is omitted",
+ )
+ linear_color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Sequential scheme used when secondary_metric is present",
+ )
+ show_labels: bool = False
+ show_labels_threshold: float = Field(
+ 5,
+ ge=0,
+ le=100,
+ description="Minimum arc size in percentage points for showing a label",
+ )
+ show_total: bool = False
+ show_null_values: bool = Field(
+ True,
+ description="Keep null-valued hierarchy nodes in the rendered tree",
+ )
+ label_type: Literal["key", "value", "key_value"] = "key"
+ number_format: str = Field("SMART_NUMBER", min_length=1, max_length=50)
+ date_format: str = Field("smart_date", min_length=1, max_length=50)
+ currency_format: CurrencyFormat | None = None
+
+ @staticmethod
+ def _looks_like_native_form_data(data: Any) -> bool:
+ """Identify saved Explore payloads without weakening typed typo checks."""
+ if not isinstance(data, dict) or data.get("viz_type") != "sunburst_v2":
+ return False
+ metric = data.get("metric")
+ return (
+ any(
+ key in data
+ for key in (
+ "adhoc_filters",
+ "annotation_layers",
+ "datasource",
+ "extra_form_data",
+ "since",
+ "slice_id",
+ "standardizedFormData",
+ "until",
+ )
+ )
+ or isinstance(metric, str)
+ or (isinstance(metric, dict) and "expressionType" in metric)
+ )
+
+ @staticmethod
+ def _coerce_native_metric(value: Any) -> Any:
+ """Accept saved metric names and native SIMPLE/SQL metric objects."""
+ if isinstance(value, str):
+ return {"name": value, "saved_metric": True}
+ if not isinstance(value, dict) or "expressionType" not in value:
+ return value
+
+ expression_type = value.get("expressionType")
+ label = value.get("label") if value.get("hasCustomLabel", True) else None
+ if expression_type == "SQL":
+ return {
+ "sql_expression": value.get("sqlExpression"),
+ "label": label,
+ }
Review Comment:
**Suggestion:** Valid native SQL metrics with `hasCustomLabel=false` lose their frontend-generated label, then fail `ColumnRef` validation because SQL metrics require a label. [api mismatch]
**Assessment:** ๐ `Major` ยท ๐ `Occurrence: Sometimes`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=8582918877fb475082df3e3ec5ec8543&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=8582918877fb475082df3e3ec5ec8543&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent ๐ค </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/mcp_service/chart/schemas.py
**Line:** 1179:1183
**Comment:**
*Api Mismatch: Valid native SQL metrics with `hasCustomLabel=false` lose their frontend-generated label, then fail `ColumnRef` validation because SQL metrics require a label.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F43771&comment_hash=15b5652b8c7ddde9c29280a29f5426c9eec03698ff69523fa032a3a1e0281d60&reaction=like'>๐</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F43771&comment_hash=15b5652b8c7ddde9c29280a29f5426c9eec03698ff69523fa032a3a1e0281d60&reaction=dislike'>๐</a>
##########
superset/mcp_service/chart/tool/update_chart_preview.py:
##########
@@ -240,6 +245,11 @@ def update_chart_preview( # noqa: C901
if previous_form_data:
merge_table_column_config(previous_form_data, new_form_data)
merge_interactive_pivot_ui_config(previous_form_data, new_form_data)
+ new_form_data = merge_form_data_for_update(
+ previous_form_data, new_form_data, config
+ )
Review Comment:
**Suggestion:** When changing a cached preview to another chart type, this merge retains the old chart's query fields, so stale dimensions or metrics can affect the new chart. [api mismatch]
**Assessment:** ๐ `Major` ยท ๐ `Occurrence: Sometimes`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=c708324bcd28473b8dca3df6abd8acb2&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=c708324bcd28473b8dca3df6abd8acb2&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent ๐ค </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/mcp_service/chart/tool/update_chart_preview.py
**Line:** 248:250
**Comment:**
*Api Mismatch: When changing a cached preview to another chart type, this merge retains the old chart's query fields, so stale dimensions or metrics can affect the new chart.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F43771&comment_hash=9199b4dccba852ca41079eb0367963f5236ed77aa6f77c142e908f2894546a78&reaction=like'>๐</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F43771&comment_hash=9199b4dccba852ca41079eb0367963f5236ed77aa6f77c142e908f2894546a78&reaction=dislike'>๐</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3927151350
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -167,6 +175,110 @@ def _rejected_requested_filter_columns(
}
_MAX_RECOMMENDATIONS = 4
+_MAX_COLUMN_PROFILE_CELLS = 100_000
+
+
+@dataclass
+class _ProfileBudget:
+ """One iterative node budget shared by every profiled column."""
+
+ nodes: int = 0
+
+
+def _canonical_profile_key(
+ value: Any, budget: _ProfileBudget
+) -> tuple[Any, ...] | None:
+ """Build a bounded identity from canonical JSON primitives, iteratively."""
+ tokens: list[Any] = []
+ stack: list[tuple[str, Any]] = [("value", value)]
+ while stack:
+ action, item = stack.pop()
+ budget.nodes += 1
+ if budget.nodes > _MAX_COLUMN_PROFILE_CELLS:
+ return None
+ if action == "end":
+ tokens.append(item)
+ continue
+ if type(item) is list:
+ tokens.append(("list", len(item)))
+ stack.append(("end", "list_end"))
+ stack.extend(
+ ("value", list.__getitem__(item, index))
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if type(item) is dict:
+ entries = sorted(dict.items(item))
+ tokens.append(("dict", len(entries)))
+ stack.append(("end", "dict_end"))
+ for key, child in reversed(entries):
+ stack.append(("value", child))
+ stack.append(("end", ("key", key)))
+ continue
+ value_type = type(item)
+ if value_type not in {type(None), bool, int, float, str}: # noqa: E721
+ raise TypeError("query result value was not canonicalized")
+ tokens.append((value_type.__name__, item))
+ return tuple(tokens)
+
+
+def _build_data_columns( # noqa: C901
+ data: list[dict[str, Any]],
+ raw_columns: list[str],
+ coltypes: list[Any] | None = None,
+) -> list[DataColumn]:
+ """Profile canonical rows under one nested-node budget."""
+ column_count = max(len(raw_columns), 1)
+ profile_row_limit = max(3, _MAX_COLUMN_PROFILE_CELLS // column_count)
+ budget = _ProfileBudget()
+ profiled_values: dict[str, list[Any]] = {column: [] for column in raw_columns}
+ profiled_keys: dict[str, set[tuple[Any, ...]]] = {
+ column: set() for column in raw_columns
+ }
+ sampled_rows = 0
+ for row_offset in range(min(len(data), profile_row_limit)):
+ row = list.__getitem__(data, row_offset)
+ row_values: list[tuple[str, Any, tuple[Any, ...]]] = []
+ for col_name in raw_columns:
+ value = dict.get(row, col_name)
+ identity = _canonical_profile_key(value, budget)
+ if identity is None:
+ break
+ row_values.append((col_name, value, identity))
+ if len(row_values) != len(raw_columns):
+ break
+ for col_name, value, identity in row_values:
+ profiled_values[col_name].append(value)
+ profiled_keys[col_name].add(identity)
+ sampled_rows += 1
+
+ columns: list[DataColumn] = []
+ for idx, col_name in enumerate(raw_columns):
+ values = profiled_values[col_name]
+ sample_values = [value for value in values[:3] if value is not None]
+ data_type = "string"
+ if coltypes:
+ data_type = _GENERIC_TYPE_MAP.get(coltypes[idx], "string")
Review Comment:
This was stale against the current implementation and is now explicitly reverified. Exact finite `Decimal` values are included in numeric type inference and use the bounded rational identity path; scale-equivalent Decimal/int/float values coalesce intentionally. `test_decimal_cardinality_uses_exact_cross_numeric_value_semantics`, the exponent-cap regression, and hostile Decimal-subclass tests pass at 5306ee42.
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -419,15 +447,16 @@ async def get_chart_data( # noqa: C901
)
try:
cached_form_data_dict = utils_json.loads(cached_form_data)
- except (TypeError, ValueError) as e:
+ except (TypeError, ValueError) as exc:
+ if type(exc) not in {TypeError, ValueError}:
+ raise
Review Comment:
Fixed in 27f6a119. Ordinary parser failures now recognize the exact built-in TypeError/ValueError and exact `simplejson.JSONDecodeError`, while unexpected/hostile subclasses still propagate to the fixed outer containment layer. The classifier regression and a real registered `FastMCP Client.call_tool` invalid-cache probe verify bounded `ParseError` behavior without invoking hostile hooks.
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -1045,6 +1064,831 @@ def reject_sql_expression_on_dimensions(self) -> "PieChartConfig":
return self
+class SunburstStandardizedControls(UnknownFieldCheckMixin):
+ """Bounded shared controls retained by Explore across viz changes."""
+
+ model_config = ConfigDict(extra="ignore")
+
+ metrics: list[JsonValue] = Field(default_factory=list)
+ columns: list[JsonValue] = Field(default_factory=list)
+
+
+class SunburstStandardizedFormData(UnknownFieldCheckMixin):
+ """Validated shape of Explore's cross-plugin UI memory."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ controls: SunburstStandardizedControls
+ memorized_form_data: list[tuple[str, dict[str, JsonValue]]] = Field(
+ default_factory=list,
+ validation_alias=AliasChoices("memorized_form_data", "memorizedFormData"),
+ serialization_alias="memorizedFormData",
+ )
+
+
+_NATIVE_COLUMN_META_MAX_DEPTH = 8
+_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS = 128
+_NATIVE_COLUMN_META_MAX_TOTAL_VALUES = 512
+_NATIVE_COLUMN_META_MAX_KEY_BYTES = 1_024
+_NATIVE_COLUMN_META_MAX_STRING_BYTES = 16 * 1_024
+_NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES = 64 * 1_024
+_NATIVE_COLUMN_META_MAX_INT_BITS = 64
+_NATIVE_METRIC_ALLOWED_KEYS = frozenset(
+ {
+ "aggregate",
+ "column",
+ "datasourceWarning",
+ "expressionType",
+ "hasCustomLabel",
+ "label",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_EXPRESSION_TYPES = frozenset({"SIMPLE", "SQL"})
+_NATIVE_METRIC_DISCRIMINATOR_KEYS = frozenset(
+ {
+ "column",
+ "datasourceWarning",
+ "hasCustomLabel",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_MISSING = object()
+
+
+class _UnsafeNativeMetricInputError(ValueError):
+ """Signal input that must be removed before Pydantic renders an error."""
+
+
+def _native_column_meta_string_bytes(value: str, *, key: bool = False) -> int:
+ """Return exact UTF-8 size for a trusted built-in string."""
+ kind = "key" if key else "string"
+ try:
+ size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(
+ f"Sunburst native metric column metadata {kind} is invalid"
+ ) from ex
+ limit = (
+ _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ if key
+ else _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ )
+ if size > limit:
+ raise ValueError(f"Sunburst native metric column metadata {kind} is too long")
+ return size
+
+
+def _validate_native_metric_string(
+ value: Any, field: str, max_length: int
+) -> str | None:
+ """Validate one optional exact string in the closed native metric wrapper."""
+ if value is None:
+ return None
+ if type(value) is not str:
+ raise ValueError(f"Sunburst native metric {field} must be a string")
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(f"Sunburst native metric {field} is invalid") from ex
+ if str.__len__(value) > max_length or encoded_size > max_length * 4:
+ raise ValueError(f"Sunburst native metric {field} is too long")
+ return value
+
+
+def _inspect_native_metric_expression_type(value: Any) -> tuple[object, bool]:
+ """Inspect an exact wrapper without hashing or comparing hostile objects."""
+ if type(value) is not dict:
+ return _NATIVE_METRIC_MISSING, False
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+ expression_type: object = _NATIVE_METRIC_MISSING
+ has_native_discriminator = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise _UnsafeNativeMetricInputError(
+ "Sunburst native metric keys must be strings"
+ )
+ _validate_native_metric_string(key, "field name", 255)
+ if key in _NATIVE_METRIC_DISCRIMINATOR_KEYS:
+ has_native_discriminator = True
+ if key == "expressionType":
+ if type(item) is not str:
+ if issubclass(type(item), str):
+ raise _UnsafeNativeMetricInputError(
+ "Sunburst native metric expressionType must be a string"
+ )
+ raise ValueError(
+ "Sunburst native metric expressionType must be a string"
+ )
+ if item not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError(
+ "Sunburst native metric expressionType must be SIMPLE or SQL"
+ )
+ expression_type = item
+ return expression_type, has_native_discriminator
+
+
+def _validate_native_metric_column_value(value: Any) -> None:
+ """Validate a wrapper column without dispatching container or scalar hooks."""
+ if type(value) is str:
+ _validate_native_metric_string(value, "column", 255)
+ elif type(value) is dict:
+ _validate_bounded_native_column_meta(value)
+ elif value is not None:
+ raise ValueError("Sunburst native metric column must be a string or object")
+
+
+def _validate_native_metric_item(key: str, item: Any) -> object:
+ """Validate one value after its wrapper key is known to be an exact string."""
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError("Sunburst native metric expressionType must be a string")
+ _validate_native_metric_string(item, "expressionType", 10)
+ return item
+ if key in ("label", "optionName"):
+ _validate_native_metric_string(item, key, 500)
+ elif key == "sqlExpression":
+ _validate_native_metric_string(item, key, 2_000)
+ elif key == "aggregate":
+ _validate_native_metric_string(item, key, 32)
+ elif key in ("datasourceWarning", "hasCustomLabel"):
+ if item is not None and type(item) is not bool:
+ raise ValueError(f"Sunburst native metric {key} must be a boolean")
+ elif key == "column":
+ _validate_native_metric_column_value(item)
+ return _NATIVE_METRIC_MISSING
+
+
+def _validate_native_metric_wrapper(value: Any) -> dict[str, Any]:
+ """Validate and copy the closed native metric wrapper without calling hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric must be a string or object")
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+
+ validated: dict[str, Any] = {}
+ expression_type: object = _NATIVE_METRIC_MISSING
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key not in _NATIVE_METRIC_ALLOWED_KEYS:
+ raise ValueError(f"Unknown Sunburst native metric field: {key}")
+ item_expression_type = _validate_native_metric_item(key, item)
+ if item_expression_type is not _NATIVE_METRIC_MISSING:
+ expression_type = item_expression_type
+ validated[key] = item
+
+ if expression_type is _NATIVE_METRIC_MISSING:
+ raise ValueError("Sunburst native metric requires expressionType")
+ if expression_type not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError("Sunburst native metric expressionType must be SIMPLE or SQL")
+ return validated
+
+
+def _validate_non_dict_native_metric(value: Any) -> ColumnRef:
+ """Accept an existing model while rejecting hook-bearing dict subclasses."""
+ if issubclass(type(value), dict):
+ raise _UnsafeNativeMetricInputError(
+ "Sunburst native metric must be an exact object"
+ )
+ if isinstance(value, ColumnRef):
+ return value
+ raise ValueError("Sunburst native metric must be a string or object")
+
+
+def _replace_unsafe_native_metric(
+ data: dict[str, Any], original_data: dict[str, Any] | None, key: str
+) -> None:
+ """Remove one hook-bearing value from working and Pydantic input mappings."""
+ dict.__setitem__(data, key, 0)
+ if original_data is not None:
+ dict.__setitem__(original_data, key, 0)
+
+
+def _native_column_meta_dict_children(
+ value: dict[Any, Any], depth: int
+) -> tuple[list[tuple[Any, int]], int]:
+ """Read exact-dict children while validating bounded primitive keys."""
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata object is too large")
+ children: list[tuple[Any, int]] = []
+ string_bytes = 0
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ raise ValueError(
+ "Sunburst native metric column metadata keys must be strings"
+ )
+ string_bytes += _native_column_meta_string_bytes(key, key=True)
+ children.append((nested, depth + 1))
+ return children, string_bytes
+
+
+def _native_column_meta_list_children(
+ value: list[Any], depth: int
+) -> list[tuple[Any, int]]:
+ """Read exact-list children without dispatching subclass hooks."""
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata array is too large")
+ return [
+ (list.__getitem__(value, index), depth + 1)
+ for index in range(list.__len__(value))
+ ]
+
+
+def _validate_native_column_meta_scalar(value: Any) -> int:
+ """Validate one exact JSON scalar and return its string-byte contribution."""
+ value_type = type(value)
+ if value_type is str:
+ return _native_column_meta_string_bytes(value)
+ if value_type is int:
+ if int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ raise ValueError(
+ "Sunburst native metric column metadata integer is too large"
+ )
+ return 0
+ if value_type is float:
+ if not math.isfinite(value):
+ raise ValueError(
+ "Sunburst native metric column metadata number must be finite"
+ )
+ return 0
+ if value_type not in (bool, type(None)):
+ raise ValueError(
+ "Sunburst native metric column metadata contains an invalid value"
+ )
+ return 0
+
+
+def _validate_bounded_native_column_meta(value: Any) -> Any:
+ """Validate an extensible frontend ColumnMeta object without invoking hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric column must be an object")
+
+ total_values = 0
+ total_string_bytes = 0
+ stack: list[tuple[Any, int]] = [(value, 0)]
+ while stack:
+ item, depth = stack.pop()
+ total_values += 1
+ if total_values > _NATIVE_COLUMN_META_MAX_TOTAL_VALUES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ if depth > _NATIVE_COLUMN_META_MAX_DEPTH:
+ raise ValueError(
+ "Sunburst native metric column metadata is too deeply nested"
+ )
+
+ item_type = type(item)
+ if item_type is dict:
+ children, key_bytes = _native_column_meta_dict_children(item, depth)
+ total_string_bytes += key_bytes
+ stack.extend(children)
+ elif item_type is list:
+ stack.extend(_native_column_meta_list_children(item, depth))
+ else:
+ total_string_bytes += _validate_native_column_meta_scalar(item)
+
+ if total_string_bytes > _NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ return value
+
+
+class SunburstNativeMetricColumn(BaseModel):
+ """Owned projection of the frontend's intentionally extensible ColumnMeta."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ type: StrictStr | None = Field(None, max_length=255)
+
+ @model_validator(mode="before")
+ @classmethod
+ def validate_bounded_column_meta(cls, value: Any) -> Any:
+ """Bound the full open object before projecting the fields we own."""
+ value = _validate_bounded_native_column_meta(value)
+ snake_name = dict.get(value, "column_name", _NATIVE_METRIC_MISSING)
+ camel_name = dict.get(value, "columnName", _NATIVE_METRIC_MISSING)
+ if (
+ snake_name is not _NATIVE_METRIC_MISSING
+ and camel_name is not _NATIVE_METRIC_MISSING
+ ):
+ if type(snake_name) is not str or type(camel_name) is not str:
+ raise ValueError(
+ "column_name and columnName must both be exact strings"
+ )
+ if snake_name != camel_name:
+ raise ValueError(
+ "column_name and columnName must match when both are provided"
+ )
+ return value
+
+
+class XYNativeMetricColumn(UnknownFieldCheckMixin):
+ """Closed legacy ColumnMeta projection retained for native XY axes."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ id: StrictInt | None = Field(None, ge=0)
+ type: StrictStr | None = Field(None, max_length=255)
+ type_generic: StrictInt | None = Field(None, ge=0, le=10)
+ groupby: StrictBool | None = None
+ is_dttm: StrictBool | None = None
+ filterable: StrictBool | None = None
+ verbose_name: StrictStr | None = Field(None, max_length=500)
+ description: StrictStr | None = Field(None, max_length=2000)
+ expression: StrictStr | None = Field(None, max_length=2000)
+ database_expression: StrictStr | None = Field(None, max_length=2000)
+ python_date_format: StrictStr | None = Field(None, max_length=255)
+ option_name: StrictStr | None = Field(
+ None,
+ max_length=500,
+ validation_alias=AliasChoices("option_name", "optionName"),
+ )
+ filter_by: StrictStr | None = Field(
+ None,
+ max_length=255,
+ validation_alias=AliasChoices("filter_by", "filterBy"),
+ )
+ value: StrictStr | None = Field(None, max_length=500)
+ advanced_data_type: StrictStr | None = Field(None, max_length=255)
+ uuid: StrictStr | None = Field(None, max_length=64)
+
+
+class SunburstLegacySavedFields(BaseModel):
+ """Known obsolete controls tolerated only while reading saved Sunbursts."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ country_fieldtype: StrictStr | None = Field(None, max_length=255)
+ entity: StrictStr | None = Field(None, max_length=255)
+ granularity: StrictStr | None = Field(None, max_length=255)
+ limit: StrictInt | StrictStr | None = None
+ markup_type: Literal["markdown", "html"] | None = None
+ show_bubbles: StrictBool | None = None
+
+ @field_validator("limit")
+ @classmethod
+ def validate_legacy_limit(cls, value: int | str | None) -> int | str | None:
+ """Bound the ignored legacy row limit and reject non-numeric strings."""
+ if value is None:
+ return None
+ if isinstance(value, str):
+ if not value.isascii() or not value.isdigit():
+ raise ValueError("legacy Sunburst limit must be an integer")
+ numeric_value = int(value)
+ else:
+ numeric_value = value
+ if not 1 <= numeric_value <= 50000:
+ raise ValueError("legacy Sunburst limit must be between 1 and 50000")
+ return value
+
+
+_NATIVE_FORM_DATA_MARKER = "_mcp_native_form_data"
+_SUNBURST_IGNORED_LEGACY_FIELDS = frozenset(
+ {
+ "country_fieldtype",
+ "entity",
+ "granularity",
+ "limit",
+ "markup_type",
+ "show_bubbles",
+ }
+)
+
+
+class SunburstChartConfig(BaseChartConfig):
+ """Config for the ECharts Sunburst plugin (viz_type ``sunburst_v2``).
+
+ ``hierarchy`` follows the frontend ``columns`` control: the first entry is
+ the innermost ring and each later entry adds a child level. The primary
+ metric sizes arcs; an optional secondary metric colors arcs by the
+ secondary/primary ratio.
+ """
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ chart_type: Literal["sunburst"] = "sunburst"
+ viz_type: Literal["sunburst_v2"] = Field(
+ "sunburst_v2",
+ description="Exact Superset frontend visualization tag",
+ )
+ hierarchy: List[ColumnRef] = Field(
+ ...,
+ min_length=1,
+ description=(
+ "Hierarchy dimensions in ring order, from the innermost/root level "
+ "to the outermost/leaf level"
+ ),
+ validation_alias=AliasChoices("hierarchy", "columns", "groupby"),
+ )
+ metric: ColumnRef = Field(
+ ...,
+ description=(
+ "Primary metric used to size arcs. Use aggregate for a SIMPLE "
+ "adhoc metric, saved_metric=True for a dataset metric, or "
+ "sql_expression plus label for a SQL metric."
+ ),
+ )
+ secondary_metric: ColumnRef | None = Field(
+ None,
+ description=(
+ "Optional metric used to color arcs by secondary/primary ratio. "
+ "When omitted, colors are categorical."
+ ),
+ validation_alias=AliasChoices("secondary_metric", "secondaryMetric"),
+ )
+ filters: List[FilterConfig] | None = Field(
+ None,
+ description=(
+ "Structured WHERE filters (column/op/value/clause). An omitted list is "
+ "preserved on updates; an explicit [] clears saved filters."
+ ),
+ )
+ time_range: str | None = Field(
+ None,
+ min_length=1,
+ max_length=1000,
+ description=(
+ "Superset time range, for example 'Last year', "
+ "'2025-01-01 : 2025-12-31', or 'No filter'"
+ ),
+ )
+ time_grain: TimeGrain | None = Field(
+ None,
+ description="Optional bucket for temporal hierarchy columns",
+ validation_alias=AliasChoices("time_grain", "time_grain_sqla"),
+ )
+ sort_by_metric: bool = Field(
+ False,
+ description=(
+ "Order hierarchy rows by the primary metric descending before "
+ "applying row_limit, matching the frontend buildQuery transform"
+ ),
+ )
+ row_limit: int = Field(10000, description="Maximum hierarchy rows", ge=1, le=50000)
+ color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Categorical scheme used when secondary_metric is omitted",
+ )
+ linear_color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Sequential scheme used when secondary_metric is present",
+ )
+ show_labels: bool = False
+ show_labels_threshold: float = Field(
+ 5,
+ ge=0,
+ le=100,
+ description="Minimum arc size in percentage points for showing a label",
+ )
+ show_total: bool = False
+ show_null_values: bool = Field(
+ True,
+ description="Keep null-valued hierarchy nodes in the rendered tree",
+ )
+ label_type: Literal["key", "value", "key_value"] = "key"
+ number_format: str = Field("SMART_NUMBER", min_length=1, max_length=50)
+ date_format: str = Field("smart_date", min_length=1, max_length=50)
+ currency_format: CurrencyFormat | None = None
+ # Bounded native Explore envelope/UI state. These keys occur in real saved
+ # form_data and are safe to round-trip, but remain typed so native mode does
+ # not become an escape hatch for arbitrary plugin keys or malformed nesting.
+ annotation_layers: list[dict[str, JsonValue]] | None = None
+ dashboard_id: int | None = Field(
+ None,
+ validation_alias=AliasChoices("dashboard_id", "dashboardId"),
+ serialization_alias="dashboardId",
+ )
+ dashboards: list[int] | None = None
+ datasource: str | None = Field(None, min_length=1, max_length=255)
+ extra_form_data: dict[str, JsonValue] | None = None
+ slice_id: int | None = None
+ url_params: dict[str, JsonValue] | None = None
+ standardized_form_data: SunburstStandardizedFormData | None = Field(
+ None,
+ validation_alias=AliasChoices("standardized_form_data", "standardizedFormData"),
+ serialization_alias="standardizedFormData",
+ )
+ since: str | None = Field(None, max_length=1000)
+ until: str | None = Field(None, max_length=1000)
+ time_compare: str | list[str] | None = None
+ compare_lag: int | str | None = None
+ compare_suffix: str | None = Field(None, max_length=255)
+
+ @staticmethod
+ def _looks_like_native_form_data(data: Any) -> bool:
+ """Identify saved Explore payloads without weakening typed typo checks."""
+ if type(data) is not dict:
+ return False
+ native_marker = dict.get(data, _NATIVE_FORM_DATA_MARKER)
+ viz_type = dict.get(data, "viz_type")
+ if not (
+ (type(native_marker) is bool and native_marker)
+ or (type(viz_type) is str and viz_type == "sunburst_v2")
+ ):
+ return False
+ metric = dict.get(data, "metric")
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(metric)
+ )
+ return (
+ any(
+ key in data
+ for key in (
+ "adhoc_filters",
+ "annotation_layers",
+ "datasource",
+ "extra_form_data",
+ "since",
+ "slice_id",
+ "standardizedFormData",
+ "until",
+ )
+ )
+ or type(metric) is str
+ or expression_type is not _NATIVE_METRIC_MISSING
+ or has_native_discriminator
+ )
+
+ @staticmethod
+ def _coerce_native_metric(
+ value: Any, *, allow_extensible_column_meta: bool = True
+ ) -> Any:
+ """Accept saved metric names and native SIMPLE/SQL metric objects."""
+ if type(value) is str:
+ return {"name": value, "saved_metric": True}
+ if type(value) is dict:
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(value)
+ )
+ if expression_type is _NATIVE_METRIC_MISSING:
+ if has_native_discriminator:
+ raise ValueError("Sunburst native metric requires expressionType")
+ return value
+ value = _validate_native_metric_wrapper(value)
+ else:
+ return _validate_non_dict_native_metric(value)
+
+ expression_type = value.get("expressionType")
+ label = value.get("label")
+ has_custom_label = value.get("hasCustomLabel")
+ if expression_type == "SQL":
+ sql_expression = value.get("sqlExpression")
+ return {
+ "sql_expression": sql_expression,
+ # The frontend still assigns a result label when the label is
+ # not custom. Preserve a nonempty native label first, then use
+ # the SQL expression as the same effective fallback.
+ "label": label or sql_expression,
+ "has_custom_label": has_custom_label,
+ }
+ if expression_type == "SIMPLE":
+ column = value.get("column")
+ if type(column) is dict:
+ column_model = (
+ SunburstNativeMetricColumn
+ if allow_extensible_column_meta
+ else XYNativeMetricColumn
+ )
+ column_metadata = column_model.model_validate(column)
+ name = column_metadata.column_name
+ dtype = column_metadata.type
+ elif type(column) is str:
+ name = column
+ dtype = None
+ else:
+ raise ValueError(
+ "Sunburst SIMPLE metric column must be a column name or object"
+ )
+ return {
+ "name": name,
+ "aggregate": value.get("aggregate"),
+ "label": label,
+ "has_custom_label": has_custom_label,
+ "dtype": dtype,
+ }
+ return value
+
+ @staticmethod
+ def _coerce_native_hierarchy(data: dict[str, Any]) -> None:
+ """Normalize native ``columns``/``groupby`` dimension shortcuts."""
+ hierarchy_key = next(
+ (key for key in ("hierarchy", "columns", "groupby") if key in data),
+ None,
+ )
+ if hierarchy_key is None:
+ return
+ hierarchy = data[hierarchy_key]
+ if isinstance(hierarchy, (str, dict)):
+ hierarchy = [hierarchy]
+ if isinstance(hierarchy, list):
+ data[hierarchy_key] = [
+ {"name": item} if isinstance(item, str) else item for item in hierarchy
+ ]
+ if hierarchy_key != "groupby":
+ # Saved Sunburst payloads sometimes retain an empty groupby from a
+ # previous plugin. The explicit columns/hierarchy control wins.
+ data.pop("groupby", None)
+
+ @staticmethod
+ def _coerce_native_filter(
+ native_filter: Any, data: dict[str, Any]
+ ) -> dict[str, Any] | None:
+ """Convert one native SIMPLE filter, extracting temporal controls."""
+ if not isinstance(native_filter, dict):
+ raise ValueError("Each Sunburst native adhoc_filter must be an object")
+ allowed_keys = {
+ "clause",
+ "comparator",
+ "datasourceWarning",
+ "expressionType",
+ "filterOptionName",
+ "isExtra",
+ "operator",
+ "sqlExpression",
+ "subject",
+ }
+ if unknown := set(native_filter) - allowed_keys:
+ raise ValueError(
+ "Unknown Sunburst native adhoc_filter field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ if native_filter.get("expressionType") != "SIMPLE":
+ raise ValueError(
+ "Sunburst native adhoc_filters round-trip supports SIMPLE filters "
+ "only; express filters with the typed 'filters' field"
+ )
+ operator = native_filter.get("operator")
+ clause = native_filter.get("clause", "WHERE")
+ if not isinstance(clause, str) or clause != "WHERE":
+ raise ValueError(
+ "Sunburst native SIMPLE filter clause must be 'WHERE'; "
+ "SIMPLE HAVING is unsupported because the shared query mapper "
+ "cannot preserve HAVING semantics"
+ )
+ if operator == "TEMPORAL_RANGE":
+ if "temporal_column" not in data and "granularity_sqla" not in data:
+ data["temporal_column"] = native_filter.get("subject")
+ data.setdefault("time_range", native_filter.get("comparator"))
+ return None
+ mapped_operator = "=" if operator == "==" else operator
+ comparator = native_filter.get("comparator")
+ if mapped_operator in {"IS NULL", "IS NOT NULL"}:
+ comparator = None
+ return {
+ "column": native_filter.get("subject"),
+ "clause": clause,
+ "op": mapped_operator,
+ "value": comparator,
+ }
+
+ @staticmethod
+ def _discard_legacy_saved_fields(data: dict[str, Any]) -> None:
+ """Validate and ignore only the known obsolete saved-chart controls."""
+ legacy = {
+ key: data[key] for key in _SUNBURST_IGNORED_LEGACY_FIELDS if key in data
+ }
+ if legacy:
+ SunburstLegacySavedFields.model_validate(legacy)
+ for key in legacy:
+ data.pop(key, None)
+
+ @classmethod
+ def _coerce_native_filters(cls, data: dict[str, Any]) -> None:
+ """Round-trip native saved filters into the typed filter contract."""
+ native_filters = data.pop("adhoc_filters", None)
+ if native_filters is None or "filters" in data:
+ return
+ if not isinstance(native_filters, list):
+ raise ValueError("Sunburst native adhoc_filters must be a list")
+ filters = [
+ converted
+ for native_filter in native_filters or []
+ if (converted := cls._coerce_native_filter(native_filter, data)) is not None
+ ]
+ data["filters"] = filters
+
+ @model_validator(mode="before")
+ @classmethod
+ def accept_native_form_data(cls, data: Any) -> Any:
+ """Translate native saved Sunburst form_data into the typed contract."""
+ if not isinstance(data, dict):
+ return data
+ original_data = data if type(data) is dict else None
+ data = dict(data)
+ try:
+ is_native_form_data = cls._looks_like_native_form_data(data)
+ except _UnsafeNativeMetricInputError:
+ # Pydantic includes rejected input in its rendered error. Remove
+ # hook-bearing objects before handing control back to its validators.
+ _replace_unsafe_native_metric(data, original_data, "metric")
+ is_native_form_data = True
+ data.pop(_NATIVE_FORM_DATA_MARKER, None)
+
+ if is_native_form_data:
+ cls._discard_legacy_saved_fields(data)
+ cls._coerce_native_hierarchy(data)
+
+ for key in ("metric", "secondary_metric", "secondaryMetric"):
+ if key in data and data[key] is not None:
+ try:
+ data[key] = cls._coerce_native_metric(data[key])
+ except _UnsafeNativeMetricInputError:
+ _replace_unsafe_native_metric(data, original_data, key)
+
+ # Native saved filters can be round-tripped when they use the SIMPLE
+ # controls. SQL filter clauses intentionally remain outside the typed
+ # MCP filter contract rather than being accepted as arbitrary SQL.
+ cls._coerce_native_filters(data)
+
+ if "time_range" not in data and (data.get("since") or data.get("until")):
+ data["time_range"] = (
+ f"{data.get('since') or ''} : {data.get('until') or ''}"
+ )
+ if "temporal_column" not in data and "granularity_sqla" in data:
+ data["temporal_column"] = data["granularity_sqla"]
+ data.pop("granularity_sqla", None)
+ # Native mode intentionally accepts only the explicitly typed keys
+ # above. Leave unknown keys in place so UnknownFieldCheckMixin rejects
+ # typos and stale type-specific payload rather than silently dropping it.
+ return data
+
+ @field_validator("time_range")
+ @classmethod
+ def sanitize_time_range(cls, value: str | None) -> str | None:
+ """Sanitize a human-readable Superset time-range expression."""
+ return sanitize_user_input(
+ value,
+ "Time range",
+ max_length=1000,
+ allow_empty=True,
+ )
+
+ @model_validator(mode="after")
+ def validate_roles(self) -> "SunburstChartConfig":
+ """Enforce dimension/metric roles and unique query output labels."""
+ hierarchy_names: list[str] = []
+ for index, dimension in enumerate(self.hierarchy):
+ _reject_sql_expression_on_dimension(dimension, f"hierarchy[{index}]")
+ if dimension.saved_metric or dimension.aggregate:
+ raise ValueError(
+ f"hierarchy[{index}] must be a physical dimension column, "
+ "not a metric"
+ )
+ hierarchy_names.append(dimension.name or "")
+
+ folded_hierarchy = [name.casefold() for name in hierarchy_names]
+ if len(folded_hierarchy) != len(set(folded_hierarchy)):
+ raise ValueError("Sunburst hierarchy dimensions must be unique")
+
+ metrics = [("metric", self.metric)]
+ if self.secondary_metric is not None:
+ metrics.append(("secondary_metric", self.secondary_metric))
+ for role, metric in metrics:
+ if not metric.is_metric:
+ raise ValueError(
+ f"{role} must define aggregate, saved_metric=True, or "
+ "sql_expression with label"
+ )
+
+ output_labels: list[tuple[str, str]] = [
+ (f"hierarchy[{index}]", name) for index, name in enumerate(hierarchy_names)
+ ]
+ output_labels.extend(
+ (role, _metric_display_label(metric)) for role, metric in metrics
+ )
+ seen: dict[str, str] = {}
+ for role, label in output_labels:
Review Comment:
Fixed in 27f6a119. The duplicate-label check now permits only a semantically identical primary/secondary pair (saved, SIMPLE, or SQL); hierarchy collisions and different metrics sharing an alias still fail. Result-role resolution treats the repeated physical output as categorical-color mode. Direct Generate/Update/Explore models, strict result validation, and a registered FastMCP `generate_chart` call cover the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5523019833
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5574577091
Finite disposition of Bito run #b69dc3, against the reviewed head bc8ac02f9a0dd3da8bdc3909079de1a99459efad:
- **Oversized fractional Decimal identity: retained intentionally.** The digit/exponent and aggregate sampling-work limits already bound these integers. Replacing numerator/denominator with their bit lengths would merge distinct values: 11e-4096 and 13e-4096 have equal numerator/denominator bit lengths but are different fractions. Added `test_extreme_fractions_keep_distinct_exact_identities`; existing bounded-exponent/other-column regressions also pass. No production change for this suggestion.
- **Terminal display width: addressed.** ASCII line clipping conservatively accounts for wide and combining Unicode characters, replaces terminal controls, and marks clipped content. Tests cover CJK, emoji, combining accents, tiny widths, and Sunburst footer preservation.
- **Mutating validator name: addressed.** Renamed to `normalize_and_validate_sunburst_result_data`, updated every caller/test reference, and explicitly documented in-place NULL-to-zero normalization. Existing compile/preview/data/export NULL regressions remain passing.
No additional review was requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4194249816
##########
superset/mcp_service/chart/response_preflight.py:
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Schema-preserving final wire-size gates for MCP chart responses."""
+
+from typing import Any, TypeVar
+
+from pydantic import BaseModel, RootModel
+
+from superset.mcp_service.chart.query_result import response_json_failure
+from superset.mcp_service.chart.schemas import (
+ ChartData,
+ ChartError,
+ GenerateChartResponse,
+)
+from superset.mcp_service.dataset.schemas import DatasetError, QueryDatasetResponse
+from superset.mcp_service.semantic_layer.schemas import (
+ GetTableResponse,
+ SemanticLayerError,
+)
+
+_ResponseModel = TypeVar("_ResponseModel", bound=BaseModel)
+_MAX_EXCEPTION_PARTS = 3
+_MAX_EXCEPTION_TEXT_BYTES = 256
+
+
+class UpdateChartPreviewResponse(RootModel[dict[str, Any]]):
+ """Typed wire projection for the legacy dict-shaped preview response."""
+
+
+def bounded_exception_message(exception: BaseException) -> str: # noqa: C901
+ """Extract bounded primitive exception arguments without formatting hooks."""
+ try:
+ args = object.__getattribute__(exception, "args")
+ except Exception: # pragma: no cover - BaseException provides args
+ args = ()
+ parts: list[str] = []
+ used = 0
+ if type(args) is tuple:
+ for index in range(min(tuple.__len__(args), _MAX_EXCEPTION_PARTS)):
+ value = tuple.__getitem__(args, index)
+ if type(value) is str:
+ text = value
+ elif type(value) is bool:
+ text = "True" if value else "False"
+ elif type(value) is int and int.bit_length(value) <= 4_096:
+ text = int.__str__(value)
+ elif type(value) is float:
+ text = float.__repr__(value)
+ elif value is None:
+ text = "None"
+ else:
+ continue
+ remaining = _MAX_EXCEPTION_TEXT_BYTES - used - (2 if parts else 0)
+ if remaining <= 0:
+ break
+ encoded = str.encode(text[:remaining], "utf-8", errors="replace")[
+ :remaining
+ ]
+ bounded = bytes.decode(encoded, "utf-8", errors="ignore")
+ if bounded:
+ parts.append(bounded)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ if parts:
+ return "; ".join(parts)
+ return "request failed"
+
+
+def finalize_chart_data_response(
+ response: ChartData | ChartError,
+) -> ChartData | ChartError:
+ """Preflight every chart-data result while preserving its public union."""
+ if response_json_failure(response) is None:
+ return response
+ return ChartError(
+ error="Chart data response could not be returned safely.",
+ error_type="InvalidQueryResult",
+ details="The complete response exceeded its serialization safety limits.",
+ suggestions=["Request fewer rows or use a narrower export."],
+ error_code="CHART_DATA_RESPONSE_TOO_LARGE",
+ )
+
+
+def finalize_query_dataset_response(
+ response: QueryDatasetResponse | DatasetError,
+) -> QueryDatasetResponse | DatasetError:
+ """Preflight every dataset-query result while preserving its public union."""
+ if response_json_failure(response) is None:
+ return response
+ return DatasetError.create(
+ error="Dataset response could not be returned safely.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def finalize_get_table_response(
+ response: GetTableResponse | SemanticLayerError,
+) -> GetTableResponse | SemanticLayerError:
+ """Preflight every semantic-table result while preserving its public union."""
+ if response_json_failure(response) is None:
+ return response
+ return SemanticLayerError.create(
+ error="Semantic table response could not be returned safely.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def finalize_chart_response(
+ response: _ResponseModel,
+) -> _ResponseModel | ChartError:
+ """Preserve a chart response schema or return a bounded structured error."""
+ return response_json_failure(response) or response
+
+
+def finalize_generate_chart_response(
+ response: GenerateChartResponse,
+) -> GenerateChartResponse:
+ """Preflight generate/update responses and preserve their response schema."""
+ failure = response_json_failure(response)
+ if failure is None:
+ return response
+ return GenerateChartResponse.model_validate(
+ {
+ "chart": None,
Review Comment:
This replaces a successful response with `chart=None, success=False` after `generate_chart` has already run `CreateChartCommand` (and `update_chart` has already saved). If the preview/response then exceeds the preflight bounds (for example a large `table` preview with `save_chart=true`, or an oversized `column_config` key), the caller sees a failure with no chart ID even though the chart exists, and a retry creates a duplicate. Could the committed chart identity be kept in this error response, or the size check run before persisting?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1293 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ query.setdefault("metrics", []).append(sort_metric)
Review Comment:
The sort-only metric is appended to `query["metrics"]` here, before `_timeseries_post_processing` derives `metric_labels` from `query["metrics"]`, so the rolling/cumulative operators also run over it. For an ungrouped timeseries Bar with `metrics=["sales"]`, `timeseries_limit_metric="orders"`, `x_axis_sort="orders"` ascending and a 2-period rolling sum, orders `[100, 1]` become `[100, 101]` and the sort order flips relative to sorting on the raw metric. Should the sort-only metric be added after the rolling labels are computed (or excluded from them)?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,789 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
Review Comment:
Same-viz updates now drop every key in the plugin's `query_role_keys` before overlaying the mapper output, and `order_by_cols` is one of them. For a saved Table with `order_by_cols=['["id",true]']`, an `update_chart` that sets only `columns`/`row_limit` and omits `sort_by` gets no `order_by_cols` back (Table's modeled controls don't list it), so a `LIMIT 1` query afterwards returns an arbitrary row; the previous merge kept the saved sort unless `sort_by: []` was passed. Should omitted `sort_by` keep the existing ordering, with only an explicit empty list clearing it?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1487 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+import base64
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.utils.serialization import BINARY_PREFIX
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _canonical_binary(
+ value: bytes | bytearray | memoryview,
+) -> tuple[str | None, str | None]:
+ """Render exact binary cells as UTF-8 text or ``base64:``-prefixed text.
+
+ Matches the MCP response serializer: valid UTF-8 is returned as text and
+ anything else is losslessly base64-encoded. The raw size is bounded before
+ copying so the rendered string stays within the per-cell text budget.
+ """
+ size = value.nbytes if type(value) is memoryview else len(value)
+ if size > MAX_RESULT_STRING_LENGTH:
+ return None, "an oversized binary value"
+ try:
+ raw = bytes(value)
+ except (BufferError, TypeError, ValueError):
+ return None, "an invalid binary value"
+ try:
+ text = raw.decode("utf-8")
+ except UnicodeDecodeError:
+ text = BINARY_PREFIX + base64.b64encode(raw).decode("ascii")
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized binary value"
+ return text, None
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+ if value_type is bytes or value_type is bytearray or value_type is memoryview:
+ return _canonical_binary(value)
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = list.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (list.__getitem__(item, index), item, index, depth + 1, False)
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ children: list[tuple[Any, dict[str, Any], str, int, bool]] = []
+ for key, child in dict.items(item):
+ if type(key) is not str:
+ return None, "a non-string object key"
+ if reason := _charge_text(key, budget, key=True, metadata=metadata):
+ return None, reason
+ children.append((child, item, key, depth + 1, False))
+ stack.append((item, None, None, depth, True))
+ stack.extend(reversed(children))
+ continue
+
+ source_item = item
+ if type(item) in enum_types:
+ normalized = _safe_enum_value(item, enum_types)
+ if normalized is None:
+ return None, "an unsupported enum"
+ item = normalized
+ elif any(base is Enum for base in _type_mro(type(item))):
+ return None, "an enum outside its expected metadata slot"
+
+ normalized, reason = _normalize_scalar(item)
+ if reason is not None:
+ return None, reason
+ max_string_bytes = (
+ MAX_QUERY_RESULT_METADATA_BYTES if metadata else MAX_RESULT_STRING_LENGTH
+ )
+ if type(normalized) is str:
+ scalar_size = _json_string_size(normalized, max_string_bytes)
+ if scalar_size is None:
+ return None, "invalid text data"
+ else:
+ scalar_size = _normalized_scalar_json_size(normalized)
+ if reason := _charge_json_bytes(
+ budget,
+ scalar_size,
+ metadata=metadata,
+ ):
+ return None, reason
+ if parent is None:
+ root = normalized
+ elif normalized is not source_item:
+ if type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized)
+ return root, None
+
+
+def _normalize_metadata_value(
+ payload: dict[str, Any], key: str, budget: _ResultBudget
+) -> str | None:
+ enum_slots: dict[str, frozenset[type[Any]]] = {
+ "status": frozenset({QueryStatus}),
+ "result_format": frozenset({ChartDataResultFormat}),
+ "coltypes": frozenset({GenericDataType}),
+ "applied_filters": frozenset({ExtraFiltersTimeColumnType}),
+ "rejected_filters": frozenset(
+ {ExtraFiltersReasonType, ExtraFiltersTimeColumnType}
+ ),
+ }
+ value = dict.__getitem__(payload, key)
+ normalized, reason = _normalize_value(
+ value,
+ budget,
+ enum_types=enum_slots.get(key, frozenset()),
+ metadata=True,
+ )
+ if reason is None and normalized is not value:
+ dict.__setitem__(payload, key, normalized)
+ return reason
+
+
+def _error_text(value: Any) -> str | None:
+ """Extract a bounded error from an already validated primitive tree."""
+ stack: list[Any] = [value]
+ parts: list[str] = []
+ used = 0
+ while stack and len(parts) < 3 and used < _MAX_ERROR_TEXT_BYTES:
+ item = stack.pop()
+ if type(item) is dict:
+ stack.extend(
+ reversed(
+ [dict.__getitem__(item, key) for key in _ERROR_KEYS if key in item]
+ )
+ )
+ continue
+ if type(item) is list:
+ stack.extend(
+ list.__getitem__(item, index)
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if item is None or item is False:
+ continue
+ if type(item) is str:
+ remaining = _MAX_ERROR_TEXT_BYTES - used - (2 if parts else 0)
+ encoded = str.encode(item, "utf-8")[:remaining]
+ text = bytes.decode(encoded, "utf-8", errors="ignore")
+ if text:
+ parts.append(text)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ return "; ".join(parts) or None
+
+
+def _failure_for_payload(payload: dict[str, Any], label: str) -> ChartError | None:
for key in ("error", "errors", "error_message"):
- if message := _query_error_text(payload.get(key)):
+ if key in payload and (message := _error_text(dict.__getitem__(payload, key))):
return ChartError(
error=f"{label} failed: {message}", error_type="QueryError"
)
- raw_status = payload.get("status")
- status = str(getattr(raw_status, "value", raw_status) or "")
+ raw_status = dict.get(payload, "status")
+ status = raw_status if type(raw_status) is str else ""
normalized_status = status.strip().casefold().replace("-", "_").replace(" ", "_")
if normalized_status in FAILED_QUERY_STATUSES:
message = (
- _query_error_text(payload.get("message"))
- or _query_error_text(payload.get("error_message"))
+ _error_text(dict.get(payload, "message"))
+ or _error_text(dict.get(payload, "error_message"))
or normalized_status
)
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
- if payload.get("success") is False:
- message = _query_error_text(payload.get("message")) or "request failed"
+ if dict.get(payload, "success") is False:
+ message = _error_text(dict.get(payload, "message")) or "request failed"
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if raw_status is None and "data" not in payload and "queries" not in payload:
+ if message := _error_text(dict.get(payload, "message")):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+ return None
+
+
+def _metadata_shape_error( # noqa: C901
+ payload: dict[str, Any], label: str, budget: _ResultBudget
+) -> ChartError | None:
+ if "success" in payload and type(dict.__getitem__(payload, "success")) is not bool:
+ return _invalid_metadata(label)
+ if "status" in payload and type(dict.__getitem__(payload, "status")) is not str:
+ return _invalid_metadata(label)
if (
- raw_status is None
- and "data" not in payload
- and "queries" not in payload
- and (message := _query_error_text(payload.get("message")))
+ "result_format" in payload
+ and dict.__getitem__(payload, "result_format") not in _RESULT_FORMAT_VALUES
):
- return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ return _invalid_metadata(label)
+ for count_key in ("rowcount", "sql_rowcount", "total_rows"):
+ if count_key in payload:
+ count = dict.__getitem__(payload, count_key)
+ if count is not None and not (
+ type(count) is int and 0 <= count <= MAX_QUERY_RESULT_ROWCOUNT
+ ):
+ return _invalid_metadata(label)
+ if "is_cached" in payload:
+ cached = dict.__getitem__(payload, "is_cached")
+ if cached is None:
+ if reason := _charge_json_bytes(
+ budget, 1, metadata=True
+ ): # ``false`` vs ``null``
+ return _invalid_result(reason)
+ dict.__setitem__(payload, "is_cached", False)
+ elif type(cached) is not bool:
+ return _invalid_metadata(label)
+ if "cache_timeout" in payload:
+ timeout = dict.__getitem__(payload, "cache_timeout")
+ if timeout is not None and not (
+ type(timeout) is int
+ and (
+ timeout == CACHE_DISABLED_TIMEOUT
+ or 0 <= timeout <= MAX_QUERY_RESULT_CACHE_TIMEOUT
+ )
+ ):
+ return _invalid_metadata(label)
+ if "cache_key" in payload:
+ cache_key = dict.__getitem__(payload, "cache_key")
+ if cache_key is not None and (type(cache_key) is not str or not cache_key):
+ return _invalid_metadata(label)
+ for timestamp_key in ("cached_dttm", "cache_dttm", "queried_dttm"):
+ if timestamp_key not in payload:
+ continue
+ timestamp = dict.__getitem__(payload, timestamp_key)
+ if timestamp is None:
+ continue
+ if (
+ type(timestamp) is not str
+ or not timestamp
+ or len(timestamp) > MAX_QUERY_RESULT_TIMESTAMP_LENGTH
+ ):
+ return _invalid_metadata(label)
+ normalized = f"{timestamp[:-1]}+00:00" if timestamp.endswith("Z") else timestamp
+ try:
+ parsed = datetime.fromisoformat(normalized)
+ except ValueError:
+ return _invalid_metadata(label)
+ if parsed.tzinfo is None:
+ parsed = parsed.replace(tzinfo=timezone.utc)
+ if parsed.utcoffset() != timedelta(0):
+ return _invalid_metadata(label)
+ canonical_timestamp = parsed.astimezone(timezone.utc).isoformat()
+ original_size = _json_string_size(timestamp, MAX_RESULT_STRING_LENGTH)
+ canonical_size = _json_string_size(
+ canonical_timestamp, MAX_RESULT_STRING_LENGTH
+ )
+ assert original_size is not None
+ assert canonical_size is not None
+ if canonical_size > original_size and (
+ reason := _charge_json_bytes(
+ budget, canonical_size - original_size, metadata=True
+ )
+ ):
+ return _invalid_result(reason)
+ dict.__setitem__(payload, timestamp_key, canonical_timestamp)
+ return None
+
+
+def _filter_metadata_is_valid(value: Any, *, rejected: bool) -> bool:
+ if type(value) is not list or list.__len__(value) > MAX_RESULT_VALUE_ITEMS:
+ return False
+ expected = {"column", "reason"} if rejected else {"column"}
+ for entry in value:
+ if type(entry) is not dict or set(dict.keys(entry)) != expected:
+ return False
+ if type(dict.__getitem__(entry, "column")) is not str:
+ return False
+ if rejected and type(dict.__getitem__(entry, "reason")) is not str:
+ return False
+ return True
+
+
+def _validate_query_metadata(query: dict[str, Any], index: int) -> ChartError | None:
+ label = f"Chart query {index}"
+ colnames_present = "colnames" in query
+ coltypes_present = "coltypes" in query
+ colnames = dict.get(query, "colnames", [])
+ coltypes = dict.get(query, "coltypes", [])
+ if (
+ type(colnames) is not list
+ or list.__len__(colnames) > MAX_QUERY_RESULT_COLUMNS
+ or any(type(column) is not str or not column for column in colnames)
+ ):
+ return _invalid_result(f"malformed column metadata for query {index}")
+ if (
+ type(coltypes) is not list
+ or list.__len__(coltypes) > MAX_QUERY_RESULT_COLUMNS
+ or any(
+ type(value) is not int or value not in _COLTYPE_VALUES for value in coltypes
+ )
+ ):
+ return _invalid_result(f"malformed column type metadata for query {index}")
+ if (colnames_present or coltypes_present) and (
+ not colnames_present
+ or not coltypes_present
+ or list.__len__(colnames) != list.__len__(coltypes)
+ ):
+ return _invalid_result(f"misaligned column metadata for query {index}")
+ if "applied_filters" in query and not _filter_metadata_is_valid(
+ dict.__getitem__(query, "applied_filters"), rejected=False
+ ):
+ return _invalid_metadata(label)
+ if "rejected_filters" in query and not _filter_metadata_is_valid(
+ dict.__getitem__(query, "rejected_filters"), rejected=True
+ ):
+ return _invalid_metadata(label)
+ if "rejected_filter_columns" in query:
+ rejected_columns = dict.__getitem__(query, "rejected_filter_columns")
+ if type(rejected_columns) is not list or any(
+ type(column) is not str for column in rejected_columns
+ ):
+ return _invalid_metadata(label)
+ return None
+
+
+def validate_query_result_envelope( # noqa: C901
+ result: Any, *, none_as_empty: bool = False
+) -> ChartError | None:
+ """Canonicalize and strictly validate one real command result in place.
+
+ ``ChartDataCommand.run`` adds a live ``query_context`` sidecar. It is
+ intentionally exempted without access or traversal: MCP consumers only use
+ the wire query payload, and inspecting the sidecar would cross arbitrary
+ datasource/query hooks. Every wire value is charged against one aggregate
+ row/node/UTF-8 budget shared by all queries.
+ """
+ if type(result) is not dict:
+ return _invalid_result("a malformed result envelope")
+
+ budget = _ResultBudget()
+ if reason := _charge_value(budget, metadata=True):
+ return _invalid_result(reason)
+ top_level_keys = list(dict.keys(result))
+ if any(type(key) is not str for key in top_level_keys):
+ return _invalid_metadata("Chart query")
+ has_query_context = any(key == "query_context" for key in top_level_keys)
+ wire_item_count = dict.__len__(result) - has_query_context
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(wire_item_count, mapping=True),
+ metadata=True,
+ ):
+ return _invalid_result(reason)
+ for key in top_level_keys:
+ if key == "query_context":
+ continue
+ if reason := _charge_text(key, budget, key=True, metadata=True):
+ return _invalid_result(reason)
+ if key == "queries":
+ continue
+ if reason := _normalize_metadata_value(result, key, budget):
+ return _invalid_metadata("Chart query")
+
+ if error := _metadata_shape_error(result, "Chart query", budget):
+ return error
+ if failure := _failure_for_payload(result, "Chart query"):
+ return failure
+ if not dict.__contains__(result, "queries"):
+ return _invalid_result("no query result envelope")
+ queries = dict.__getitem__(result, "queries")
+ if type(queries) is not list or not queries:
+ return _invalid_result("no query result envelope")
+ if list.__len__(queries) > MAX_QUERY_RESULTS:
+ return _invalid_result("too many query result envelopes")
+ if reason := _charge_value(budget):
+ return _invalid_result(reason)
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(list.__len__(queries), mapping=False),
+ ):
+ return _invalid_result(reason)
+
+ for offset in range(list.__len__(queries)):
+ index = offset + 1
+ query = list.__getitem__(queries, offset)
+ if type(query) is not dict:
+ return _invalid_result(f"a malformed query {index} result envelope")
+ if reason := _charge_value(budget):
+ return _invalid_result(reason)
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(dict.__len__(query), mapping=True),
+ metadata=True,
+ ):
+ return _invalid_result(reason)
+ for key in list(dict.keys(query)):
+ if type(key) is not str:
+ return _invalid_metadata(f"Chart query {index}")
+ if reason := _charge_text(key, budget, key=True, metadata=True):
+ return _invalid_result(reason)
+ if key in {"data", "indexnames"}:
+ continue
+ if reason := _normalize_metadata_value(query, key, budget):
+ return _invalid_metadata(f"Chart query {index}")
+
+ if error := _metadata_shape_error(query, f"Chart query {index}", budget):
+ return error
+ if failure := _failure_for_payload(query, f"Chart query {index}"):
+ return failure
+ if not dict.__contains__(query, "data"):
+ return _invalid_result(f"query {index} result data is not an array of rows")
+ data = dict.__getitem__(query, "data")
+ if data is None and none_as_empty:
+ data = []
+ dict.__setitem__(query, "data", data)
+ if type(data) is not list:
+ return _invalid_result(f"query {index} result data is not an array of rows")
+ data_length = list.__len__(data)
+ if data_length > MAX_QUERY_RESULT_ROWS_PER_QUERY:
+ return _invalid_result(f"too many rows for query {index}")
+ budget.rows += data_length
+ if budget.rows > MAX_QUERY_RESULT_ROWS:
+ return _invalid_result("too many aggregate rows")
+ if reason := _charge_value(budget):
+ return _invalid_result(reason)
+ if reason := _charge_json_bytes(
+ budget, _container_json_syntax_size(data_length, mapping=False)
+ ):
+ return _invalid_result(reason)
+
+ if error := _validate_query_metadata(query, index):
+ return error
+ declared_columns = (
+ dict.__getitem__(query, "colnames") if "colnames" in query else None
+ )
+ declared_row_columns = (
+ list(dict.fromkeys(declared_columns))
+ if declared_columns is not None
+ else None
+ )
+
+ # Check declared order in the same capped pass that normalizes cells.
+ # A width mismatch fails before comparing names, so empty rows cannot
+ # multiply work by the maximum declared column count.
+ for row_offset in range(data_length):
+ row = list.__getitem__(data, row_offset)
+ if type(row) is not dict or dict.__len__(row) > MAX_QUERY_RESULT_COLUMNS:
+ return _invalid_result(f"a malformed data row for query {index}")
+ if reason := _charge_value(budget):
+ return _invalid_result(reason)
+ row_items = list(dict.items(row))
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(list.__len__(row_items), mapping=True),
+ ):
+ return _invalid_result(reason)
+ if declared_row_columns is not None and list.__len__(
+ row_items
+ ) != list.__len__(declared_row_columns):
+ return ChartError(
+ error=(
+ f"Chart query {index} returned rows that do not match the "
+ "declared column order."
+ ),
+ error_type="InvalidQueryResult",
+ )
+ for column_offset, (column, value) in enumerate(row_items):
+ if type(column) is not str:
+ return _invalid_result(
+ f"hostile or oversized row data for query {index}"
+ )
+ if declared_row_columns is not None and column != list.__getitem__(
+ declared_row_columns, column_offset
+ ):
+ return ChartError(
+ error=(
+ f"Chart query {index} returned rows that do not match "
+ "the declared column order."
+ ),
+ error_type="InvalidQueryResult",
+ )
+ if reason := _charge_text(column, budget, key=True):
+ return _invalid_result(reason)
+ normalized, reason = _normalize_value(value, budget)
Review Comment:
A raw Table over an engine that returns tuple-valued cells (for example a ClickHouse `tuple(1.0, 2.0)` column) now fails the whole `get_chart_data`/`query_dataset`/`get_table` call with `InvalidQueryResult`, because the row value is run through `_normalize_value`, which only accepts exact `list`/`dict` containers and plain scalars. Those cells reach this point as tuples or numpy arrays (pyarrow materialises them as arrays), and before this change they serialized to JSON arrays; the docs added in this PR also say tuples are sanitized. Should sequence-like cells be converted to lists before this check instead of rejecting the entire result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4192908316
##########
superset/mcp_service/chart/plugins/table.py:
##########
@@ -44,6 +46,12 @@ class TableChartPlugin(BaseChartPlugin):
}
supports_column_append = True
+ def prepare_query_form_data(self, form_data: dict[str, Any]) -> None:
+ """Resolve inherited offsets before the filter merge removes their source."""
+ inherited = (form_data.get("extra_form_data") or {}).get("time_compare")
+ if inherited and inherited not in as_list(form_data.get("time_compare") or []):
Review Comment:
This replaces the whole selection with the inherited offset whenever it isn't literally in the raw `time_compare` list, but `_table_time_offsets` only compares it after resolving `custom`. A saved aggregate Table with `time_compare=["1 year ago", "custom"]`, `start_date_offset="2 weeks ago"` and `extra_form_data={"time_compare": "2 weeks ago"}` becomes `["2 weeks ago"]` here, so the year-over-year comparison column is dropped even though the frontend keeps both. Could the hook resolve custom offsets before the membership check, or leave the replacement to `_table_time_offsets`?
##########
superset/mcp_service/chart/plugins/xy.py:
##########
@@ -53,6 +58,17 @@ class XYChartPlugin(BaseChartPlugin):
"echarts_area": "Area Chart",
"echarts_timeseries_scatter": "Scatter Plot",
}
+ query_role_keys = BaseChartPlugin.query_role_keys | {"x_axis"}
+
+ def vega_lite_preview(
Review Comment:
This hook now claims saved XY previews, but the plugin still has `resizes_saved_preview=False`, so `_create_plugin_preview` skips the width/height/description framing that the saved-chart fallback applied before. A saved `echarts_timeseries_line` preview requested with `width=800, height=200` used to honor those dimensions and now comes back as `width="container", height=400`. Should `resizes_saved_preview` be `True` for XY, as Gantt does?
##########
superset/mcp_service/chart/preview_utils.py:
##########
@@ -1452,33 +1671,35 @@ def _generate_vega_lite_preview_from_data( # noqa: C901
mark = viz_to_mark.get(viz_type, "bar")
# Basic Vega-Lite spec
- spec = {
+ preview_data = _bounded_vega_data(data)
+ spec: dict[str, Any] = {
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
- "data": {"values": data},
+ "data": {"values": preview_data},
"mark": mark,
}
# Get x_axis and metrics from form_data
- x_axis = form_data.get("x_axis")
+ axis = form_data.get("x_axis")
+ x_axis = get_column_name(axis) if axis else "__timestamp"
Review Comment:
The new axis normalization has no renderer-level coverage: the added preview tests only use string `x_axis`. A legacy grouped timeseries with no `x_axis` and wide rows keyed by `__timestamp`, or an SQL adhoc `x_axis` with label `"Month"`, would lose the x encoding or raise an unhashable-dict error if this line were reverted, and nothing would fail. Can a test assert `encoding.x.field` for both cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4206380247
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -2158,6 +3048,56 @@ def map_mixed_timeseries_config(
_add_adhoc_filters(form_data, config.filters)
+ fields_set = config.model_fields_set
+ if "adhoc_filters_b" in fields_set:
+ form_data["adhoc_filters_b"] = config.adhoc_filters_b
+ elif "filters_secondary" in fields_set:
+ form_data["adhoc_filters_b"] = [
Review Comment:
`filters_secondary` is mapped straight into `adhoc_filters_b` here, but `MixedTimeseriesPlugin.extract_column_refs` only collects the columns from the primary `filters`. A misspelled column such as `{"column": "regoin", "op": "=", "value": "EMEA"}` in `filters_secondary` is never checked against the dataset, so the chart compiles and Query B silently runs unfiltered across every region. Should the secondary filter columns go through the same dataset validation and normalization as `filters`?
##########
superset/mcp_service/chart/chart_helpers.py:
##########
@@ -450,12 +582,200 @@ def _resolve_deck_gl_metrics(
if value:
metrics.append(value)
elif isinstance(prf, str) and _is_metric_ref(prf):
- # Legacy deck_scatter: point_radius_fixed as a bare non-numeric metric key
- logger.debug("Legacy point_radius_fixed string metric encountered: %s", prf)
metrics.append(prf)
return metrics
+def _deck_query_adapter( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any], viz_type: str
+) -> dict[str, Any]:
+ """Apply the native frontend builder for a single Deck.gl layer."""
+ base_columns = list(query.get("columns") or [])
+ base_metrics = list(query.get("metrics") or [])
+ filters = list(query.get("filters") or [])
+ tooltips = _deck_tooltip_columns(form_data.get("tooltip_contents"))
+
+ def add_null(column: str, *, value: Any = ...) -> None:
+ clause: dict[str, Any] = {"col": column, "op": "IS NOT NULL"}
+ if value is not ...:
+ clause["val"] = value
+ filters.append(clause)
+
+ if viz_type == "deck_geojson":
+ geometry = form_data.get("geojson")
+ if not isinstance(geometry, str) or not geometry:
+ raise ValueError("GeoJSON column is required for GeoJSON charts")
+ columns = _add_deck_columns(base_columns, [geometry] if geometry else [])
+ cross_filter = form_data.get("cross_filter_column")
+ if cross_filter:
+ columns = _add_deck_columns(columns, [cross_filter])
+ columns = _add_deck_columns(columns, tooltips)
+ if form_data.get("filter_nulls", True) and isinstance(geometry, str):
+ add_null(geometry)
+ query.update(
+ columns=columns,
+ metrics=[],
+ groupby=[],
+ filters=filters,
+ is_timeseries=False,
+ )
+ return query
+
+ if viz_type == "deck_polygon":
+ line_column = form_data.get("line_column")
+ if not isinstance(line_column, str) or not line_column:
+ raise ValueError("Polygon column is required for Polygon charts")
+ columns = _add_deck_columns(base_columns, [line_column] if line_column else [])
+ cross_filter = form_data.get("cross_filter_column")
+ if cross_filter:
+ columns = _add_deck_columns(columns, [cross_filter])
+ columns = _add_deck_columns(columns, tooltips)
+ metrics: list[Any] = []
+ if metric := form_data.get("metric"):
+ metrics.append(metric)
+ radius = form_data.get("point_radius_fixed")
+ if (
+ isinstance(radius, dict)
+ and radius.get("type") == "metric"
+ and radius.get("value") is not None
+ ):
+ metrics.append(radius["value"])
+ if form_data.get("filter_nulls", True) and isinstance(line_column, str):
+ add_null(line_column)
+ if metric:
+ add_null(_deck_metric_label(metric))
+ query.update(
+ columns=columns,
+ metrics=metrics,
+ filters=filters,
+ is_timeseries=False,
+ )
+ return query
+
+ if viz_type == "deck_path":
+ line_column = form_data.get("line_column")
+ if not isinstance(line_column, str) or not line_column:
+ raise ValueError("Line column is required for Path charts")
+ columns = list(base_columns)
+ metrics = [metric for metric in base_metrics if _is_deck_metric_value(metric)]
+ groupby = list(query.get("groupby") or [])
+ metric = form_data.get("metric")
+ if metrics or metric:
+ if metric and metric not in metrics:
+ metrics.append(metric)
+ if line_column and line_column not in groupby:
+ groupby.append(line_column)
+ elif line_column:
+ columns = _add_deck_columns(columns, [line_column])
+ if dimension := form_data.get("dimension"):
+ columns = _add_deck_columns(columns, [dimension])
+
+ line_width = form_data.get("line_width")
+ raw_width = (
+ line_width
+ if isinstance(line_width, str)
+ else line_width.get("value")
+ if isinstance(line_width, dict)
+ else None
+ )
+ width_metric = (
+ raw_width
+ if _is_deck_metric_value(line_width)
+ and raw_width is not None
+ and not isinstance(raw_width, (int, float))
+ else None
+ )
+ for extra_metric in (width_metric, form_data.get("breakpoint_metric")):
+ if extra_metric is None:
+ continue
+ labels = {_deck_metric_label(item) for item in metrics}
+ if _deck_metric_label(extra_metric) not in labels:
+ metrics.append(extra_metric)
+ if line_column and line_column not in groupby:
+ groupby.append(line_column)
+ columns = _add_deck_columns(columns, tooltips)
+ groupby = _add_deck_columns(groupby, tooltips)
+ if not any(
+ filter_.get("col") == line_column and filter_.get("op") == "IS NOT NULL"
+ for filter_ in filters
+ if isinstance(filter_, dict)
+ ):
+ add_null(line_column)
+ query.update(
+ columns=columns,
+ metrics=metrics,
+ groupby=groupby,
Review Comment:
For a saved `deck_path` chart without a stored `query_context`, `line_column` is placed in `groupby` while `columns` is also set. `QueryObjectFactory.create()` only applies a deprecated `groupby` through `kwargs.setdefault("columns", ...)`, so the existing `columns` wins and the path column is dropped. With `line_column="path"` and `metric="count"`, `get_chart_data` and `get_chart_sql` would return one dataset-wide aggregate instead of per-path rows. Could the line column go into canonical `columns` here (as the non-metric branch already does)?
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -2627,6 +3810,37 @@ class XYChartConfig(BaseChartConfig):
le=10000,
)
+ series_limit_metric: ColumnRef | None = Field(
+ None,
+ description="Metric used to rank limited series; null clears it",
+ )
+ timeseries_limit_metric: ColumnRef | None = Field(
+ None,
+ description="Native timeseries ranking metric; null clears it",
+ )
+
+ @field_validator("series_limit_metric", "timeseries_limit_metric", mode="before")
+ @classmethod
+ def coerce_series_ranking_metric(cls, value: Any) -> Any:
+ """Accept the same native metric references as the Y-axis."""
+ if isinstance(value, dict) and "expressionType" in value:
Review Comment:
This before-validator only coerces dict values that carry `expressionType`, so a saved-metric string passes through unchanged and then fails `ColumnRef` validation. For example, `{"viz_type": "echarts_timeseries_line", "x_axis": "ds", "metrics": ["count"], "timeseries_limit_metric": "count"}` normalizes `metrics` but is rejected on `timeseries_limit_metric`, even though `create_metric_object` itself emits the plain string for a saved ranking metric. Could string values be coerced the same way `metrics` entries are (e.g. via `_coerce_native_metric`)?
##########
docs/docs/using-superset/using-ai-with-superset.mdx:
##########
@@ -461,6 +472,24 @@ also have `data: null`. For each statement, `executed_sql` is `null` when it equ
- Restart your AI client if you recently changed the configuration
+### Sunburst chart data
+
+MCP chart tools support ordered Sunburst hierarchies with a primary metric and
+an optional secondary metric. When supplied, the secondary-to-primary ratio
+drives a sequential color scale (for example, profit divided by revenue shows
+margin); when omitted, arc colors are categorical. Use `chart_type: "sunburst"`
+in typed requests; the native chart uses `viz_type: "sunburst_v2"`.
+
+SQL NULL aggregate values are rendered as zero, matching the Sunburst frontend,
+including during compile checks, previews, and data exports. Missing result
+columns, numeric strings, booleans, and infinite metric values remain invalid.
Review Comment:
The claim that infinite metric values "remain invalid" doesn't match the implementation: non-finite floats are normalized to `null` and then converted to zero, so compile checks pass and previews and exports show `0` (`test_compile_accepts_nullable_and_finite_metric_boundaries` covers this). An analyst relying on the documented rejection could read an overflowing aggregate as a genuine zero. Should this sentence describe the normalization instead?
##########
tests/unit_tests/mcp_service/chart/test_preview_utils.py:
##########
@@ -154,11 +172,41 @@ def test_build_query_columns_empty_columns_key_keeps_groupby():
) == ["country"]
+def test_generate_preview_seeds_form_data_before_query_execution():
+ """Preview execution seeds the form data consumed by virtual-dataset Jinja."""
+ with (
+ patch(
+ "superset.charts.data.form_data.set_query_context_form_data"
+ ) as mock_set_form_data,
+ patch(
+ "superset.commands.chart.data.get_data_command.ChartDataCommand"
+ ) as mock_cmd_cls,
+ patch(
+ "superset.common.query_context_factory.QueryContextFactory"
+ ) as mock_factory,
+ patch("superset.extensions.db") as mock_db,
+ ):
+ mock_db.session.get.return_value = MagicMock(id=12)
+ query_context = MagicMock()
+ mock_factory.return_value.create.return_value = query_context
+ mock_cmd_cls.return_value.run.return_value = chart_data_command_result(
+ rows=[], columns=["value"]
+ )
+
+ preview_utils.generate_preview_from_form_data(
+ form_data={"metrics": [{"label": "count"}]},
+ dataset_id=12,
+ preview_format="table",
+ )
+
+ mock_set_form_data.assert_called_once_with(query_context, 12, "table")
Review Comment:
This regression test only asserts that the mocked seeder was called, so it would still pass if seeding moved after `command.run()`. An unsaved virtual-dataset preview using `filter_values("region")` or `url_param("tenant")` would then run with missing or stale inputs and the test would stay green. Could the test use the real seeding and assert those macro values are visible when the command is built, plus a successful preview result? The other macro tests cover the helper and saved previews, not this unsaved sequence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5943903933
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5744641303
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-6003095616
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4190901993
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -3867,6 +5070,133 @@ def _normalize_chart_request_input(data: Any) -> Any:
config.pop("viz_type", None)
elif config.get("chart_type") != "table":
config.pop("viz_type", None)
+
+ if config.get("chart_type") == "xy":
+ # Native Timeseries form data uses ``x_axis`` for the semantic
+ # query column, while the typed MCP contract uses the same spelling
+ # for AxisConfig presentation state. Disambiguate before the
+ # discriminated union runs: scalars and native query-column objects
+ # become ``x``; title/format/scale mappings remain ``x_axis``.
+ if "x_axis" in config and config["x_axis"] is not None:
+ native_x_axis = config["x_axis"]
+ is_native_column = not isinstance(native_x_axis, dict) or bool(
+ {
+ "column",
+ "column_name",
+ "columnName",
+ "expressionType",
+ "sqlExpression",
+ }
+ & set(native_x_axis)
+ )
+ if is_native_column:
+ if "x" in config:
+ raise ValueError(
+ "XY config cannot provide both semantic x and native "
+ "x_axis query columns"
+ )
+ if isinstance(native_x_axis, dict):
+ allowed_x_axis_keys = {
+ "column",
+ "column_name",
+ "columnName",
+ "columnType",
+ "expressionType",
+ "label",
+ "sqlExpression",
+ "timeGrain",
+ }
+ if unknown := set(native_x_axis) - allowed_x_axis_keys:
+ raise ValueError(
+ "Unknown XY native x_axis field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ expression_type = native_x_axis.get("expressionType")
+ if expression_type == "SIMPLE":
+ column = native_x_axis.get("column")
+ if isinstance(column, dict):
+ column = XYNativeMetricColumn.model_validate(
+ column
+ ).column_name
+ native_x_axis = {
+ "name": native_x_axis.get("column_name")
+ or native_x_axis.get("columnName")
+ or column,
+ }
+ elif (
+ expression_type == "SQL"
+ and native_x_axis.get("columnType") == "BASE_AXIS"
+ ):
+ native_x_axis = {
+ "name": native_x_axis.get("sqlExpression"),
+ "label": native_x_axis.get("label"),
+ }
+ elif expression_type == "SQL":
+ native_x_axis = {
+ "sql_expression": native_x_axis.get("sqlExpression"),
+ "label": native_x_axis.get("label"),
+ }
+ elif "column_name" in native_x_axis:
+ native_x_axis = {
+ **native_x_axis,
+ "name": native_x_axis.get("column_name"),
+ }
+ config["x"] = native_x_axis
+ config.pop("x_axis", None)
+
+ def collect_axis_config(
+ field_name: str,
+ native_keys: dict[str, str],
+ ) -> None:
+ axis = config.get(field_name)
+ axis_state = dict(axis) if isinstance(axis, dict) else {}
+ found = isinstance(axis, dict)
+ for native_key, typed_key in native_keys.items():
+ if native_key in config:
+ axis_state[typed_key] = config.pop(native_key)
+ found = True
+ if found:
+ config[field_name] = axis_state
+
+ collect_axis_config(
+ "x_axis",
+ {"x_axis_title": "title", "x_axis_format": "format"},
+ )
+ collect_axis_config(
+ "y_axis",
+ {
+ "y_axis_title": "title",
+ "y_axis_format": "format",
+ "y_axis_scale": "scale",
+ },
+ )
+ if "legendOrientation" in config:
+ config["legend_orientation"] = config.pop("legendOrientation")
+
+ # These native query/presentation values are deterministically
+ # reconstructed from semantic x plus dataset temporal metadata.
+ config.pop("granularity_sqla", None)
+ config.pop("x_axis_sort_series_type", None)
+ config.pop("x_axis_sort_series_ascending", None)
+
+ # Saved metrics are string references; native SIMPLE/SQL objects
+ # carry expressionType metadata. Typed ``y`` strings keep their
+ # existing column-with-default-SUM semantics.
+ if "y" not in config and "metrics" in config:
Review Comment:
Confirmed and fixed in 495180c88ee8fe6c0faf43161b7643c5a969446b. A typed `{"chart_type":"xy","metrics":["revenue"]}` was rewritten to `saved_metric=True`. The rewrite is now limited to native Explore payloads, detected by the presence of `viz_type` (Explore form data always carries it; typed requests use `chart_type`). The typed `metrics` alias feeds `y` unchanged again and maps to `SUM(revenue)`. New tests: `test_typed_xy_metrics_alias_keeps_column_semantics` (for chart_type `xy`, `bar`, `echarts_timeseries_bar`; failed before, pass after) and `test_native_xy_metrics_strings_remain_saved_metric_references`.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,775 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ if getattr(config, "filters", None) == []:
Review Comment:
Confirmed and fixed in 495180c88ee8fe6c0faf43161b7643c5a969446b. Any non-None `filters` (an explicit clear or a replacement) now drops every saved predicate source (`filters`, `extra_filters`, `where`, `having`, `adhoc_filters`) that the new form data does not set itself, so the generated `adhoc_filters` are the only predicates left. New test `test_replacement_filters_drop_legacy_predicates_on_cross_viz_update` converts a Table saved with EMEA legacy filters/where/having to Sunburst with an APAC filter. It asserts the legacy keys are gone and the built query has APAC with no EMEA. It failed before the fix and passes after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4190030428
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1416 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
Review Comment:
Fixed in 0ba0030f39 (test follow-up e0c5b4badb). `_normalize_scalar` now accepts exact `bytes`, `bytearray` and `memoryview`. It renders them the way the MCP serializer does: UTF-8 text when the bytes decode, otherwise `base64:`-prefixed text (`b"\xff"` becomes `"base64:/w=="`). The raw size is bounded before copying, and the rendered string must fit the per-cell text budget, so only oversized binary is still rejected. Tests: `test_binary_cells_use_the_response_serializer_encoding`, plus the rejection cases updated to oversized binary.
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1416 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = list.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
Review Comment:
Fixed in b0d5676e2d. `indexnames` no longer goes through the metadata normalizer. `_normalize_index_names` bounds it by `MAX_QUERY_RESULT_ROWS_PER_QUERY` and charges each label to the shared value and byte budgets like a data cell. MultiIndex tuple labels are normalized as arrays. Coverage uses a real FULL payload: the new `full_producer_command_result` fixture runs a DataFrame through `query_actions._materialize_full_payload` and `ChartDataCommand.run`. Tests: `test_full_payload_index_metadata_uses_the_row_budget` (4,097 rows) and `test_full_payload_index_metadata_stays_bounded_per_query`.
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -90,6 +102,70 @@ class _ChartFacts(NamedTuple):
datasource_type: str | None = None
+def _is_expected_json_load_error(exc: Exception) -> bool:
+ """Recognize only the exact exceptions emitted for ordinary JSON failures."""
+ return type(exc) in {TypeError, ValueError, JSONDecodeError}
+
+
+def _requested_filter_columns(extra_form_data: dict[str, Any] | None) -> set[str]:
+ """Return simple column names explicitly requested through extra form data."""
+ if not extra_form_data:
+ return set()
+
+ columns: set[str] = set()
+ for filter_ in extra_form_data.get("filters", []):
Review Comment:
Fixed in da9e5ba876. `get_chart_data` had a stale local copy of the rejected-filter helpers, and that copy iterated a null `filters` list. The tool now uses the shared `rejected_requested_filter_columns`, as master does, which treats null lists as absent. `adhoc_filters: null` also crashed earlier, in the core extra-form-data merge, so `merge_extra_form_data` now drops null filter lists before that merge. Regression test: `test_null_extra_form_data_filter_lists_return_data`, parametrized over `filters` and `adhoc_filters` through the real tool.
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1416 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
Review Comment:
Fixed in ce7c1f1274. The "Response too large" troubleshooting section now separates the two limits. The configurable `ResponseSizeGuardMiddleware` (`max_bytes` / `enabled`) is one. The other is the mandatory chart query result limits, reported as `InvalidQueryResult`: rows per query and in total, columns, values, JSON bytes and per-cell text. The section states that `MCP_RESPONSE_SIZE_CONFIG` does not affect them and how to resolve them. `test_troubleshooting_guide_separates_mandatory_result_limits` keeps the documented numbers in sync with the constants.
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -1634,6 +1652,1045 @@ def reject_metric_style_groupby(self) -> "TreemapChartUpdateConfig":
return self
+class SunburstStandardizedControls(UnknownFieldCheckMixin):
+ """Bounded shared controls retained by Explore across viz changes."""
+
+ model_config = ConfigDict(extra="ignore")
+
+ metrics: list[JsonValue] = Field(default_factory=list)
+ columns: list[JsonValue] = Field(default_factory=list)
+
+
+class SunburstStandardizedFormData(UnknownFieldCheckMixin):
+ """Validated shape of Explore's cross-plugin UI memory."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ controls: SunburstStandardizedControls
+ memorized_form_data: list[tuple[str, dict[str, JsonValue]]] = Field(
+ default_factory=list,
+ validation_alias=AliasChoices("memorized_form_data", "memorizedFormData"),
+ serialization_alias="memorizedFormData",
+ )
+
+
+_NATIVE_COLUMN_META_MAX_DEPTH = 8
+_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS = 128
+_NATIVE_COLUMN_META_MAX_TOTAL_VALUES = 512
+_NATIVE_COLUMN_META_MAX_KEY_BYTES = 1_024
+_NATIVE_COLUMN_META_MAX_STRING_BYTES = 16 * 1_024
+_NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES = 64 * 1_024
+_NATIVE_COLUMN_META_MAX_INT_BITS = 64
+_NATIVE_METRIC_ALLOWED_KEYS = frozenset(
+ {
+ "aggregate",
+ "column",
+ "datasourceWarning",
+ "expressionType",
+ "hasCustomLabel",
+ "label",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_EXPRESSION_TYPES = frozenset({"SIMPLE", "SQL"})
+_NATIVE_METRIC_DISCRIMINATOR_KEYS = frozenset(
+ {
+ "column",
+ "datasourceWarning",
+ "hasCustomLabel",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_MISSING = object()
+_NATIVE_METRIC_INPUT_KEYS = frozenset(
+ {"metric", "metrics", "secondary_metric", "secondaryMetric"}
+)
+_NATIVE_METRIC_INVALID_VALUE = b"invalid native metric value"
+_NATIVE_METRIC_INVALID_KEY = "__invalid_native_metric_key__"
+_NATIVE_METRIC_SANITIZE_MAX_DEPTH = 16
+_NATIVE_METRIC_SANITIZE_MAX_VALUES = 1_024
+_NATIVE_METRIC_ERROR_STRING_LENGTH = 128
+
+
+def _native_column_meta_string_bytes(value: str, *, key: bool = False) -> int:
+ """Return exact UTF-8 size for a trusted built-in string."""
+ kind = "key" if key else "string"
+ try:
+ size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(
+ f"Sunburst native metric column metadata {kind} is invalid"
+ ) from ex
+ limit = (
+ _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ if key
+ else _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ )
+ if size > limit:
+ raise ValueError(f"Sunburst native metric column metadata {kind} is too long")
+ return size
+
+
+def _sanitize_native_metric_dict(
+ value: dict[Any, Any],
+ depth: int,
+ remaining: list[int],
+ ancestors: set[int],
+) -> tuple[Any, bool]:
+ """Project one exact dictionary through built-in operations only."""
+ value_id = id(value)
+ if value_id in ancestors:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ return {
+ f"invalid_{index}": _NATIVE_METRIC_INVALID_VALUE
+ for index in range(_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS + 1)
+ }, True
+
+ ancestors.add(value_id)
+ projected: dict[str, Any] = {}
+ requires_rejection = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ projected[_NATIVE_METRIC_INVALID_KEY] = _NATIVE_METRIC_INVALID_VALUE
+ requires_rejection = True
+ continue
+ try:
+ key_size = len(str.encode(key, "utf-8"))
+ except UnicodeEncodeError:
+ key_size = _NATIVE_COLUMN_META_MAX_KEY_BYTES + 1
+ if (
+ str.__len__(key) > _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ or key_size > _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ ):
+ projected["x" * (_NATIVE_COLUMN_META_MAX_KEY_BYTES + 1)] = (
+ _NATIVE_METRIC_INVALID_VALUE
+ )
+ requires_rejection = True
+ continue
+ projected_item, item_requires_rejection = _sanitize_native_metric_value(
+ item,
+ depth=depth + 1,
+ remaining=remaining,
+ ancestors=ancestors,
+ )
+ projected[key] = projected_item
+ requires_rejection = requires_rejection or item_requires_rejection
+ ancestors.remove(value_id)
+ return projected, requires_rejection
+
+
+def _sanitize_native_metric_list(
+ value: list[Any],
+ depth: int,
+ remaining: list[int],
+ ancestors: set[int],
+) -> tuple[Any, bool]:
+ """Project one exact list through built-in operations only."""
+ value_id = id(value)
+ if value_id in ancestors:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ return [_NATIVE_METRIC_INVALID_VALUE] * (
+ _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS + 1
+ ), True
+
+ ancestors.add(value_id)
+ projected_items: list[Any] = []
+ requires_rejection = False
+ for index in range(list.__len__(value)):
+ projected_item, item_requires_rejection = _sanitize_native_metric_value(
+ list.__getitem__(value, index),
+ depth=depth + 1,
+ remaining=remaining,
+ ancestors=ancestors,
+ )
+ projected_items.append(projected_item)
+ requires_rejection = requires_rejection or item_requires_rejection
+ ancestors.remove(value_id)
+ return projected_items, requires_rejection
+
+
+def _sanitize_native_metric_scalar(value: Any) -> tuple[Any, bool] | None:
+ """Project an exact scalar, or signal that ``value`` is a container."""
+ value_type = type(value)
+ if value_type is str:
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if (
+ str.__len__(value) > _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ or encoded_size > _NATIVE_COLUMN_META_MAX_STRING_BYTES * 4
+ ):
+ return "x" * (_NATIVE_COLUMN_META_MAX_STRING_BYTES + 1), True
+ return (
+ str.__getitem__(value, slice(0, _NATIVE_METRIC_ERROR_STRING_LENGTH)),
+ False,
+ )
+ if value_type is int and int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ return 1 << (_NATIVE_COLUMN_META_MAX_INT_BITS + 1), True
+ if value_type in (int, float, bool, type(None), ColumnRef):
+ return value, False
+ return None
+
+
+def _sanitize_native_metric_value(
+ value: Any,
+ *,
+ depth: int = 0,
+ remaining: list[int] | None = None,
+ ancestors: set[int] | None = None,
+) -> tuple[Any, bool]:
+ """Build a bounded exact-type projection without dispatching object hooks."""
+ if remaining is None:
+ remaining = [_NATIVE_METRIC_SANITIZE_MAX_VALUES]
+ if ancestors is None:
+ ancestors = set()
+ remaining[0] -= 1
+ if remaining[0] < 0 or depth > _NATIVE_METRIC_SANITIZE_MAX_DEPTH:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+
+ if (scalar := _sanitize_native_metric_scalar(value)) is not None:
+ return scalar
+ value_type = type(value)
+ if value_type is dict:
+ return _sanitize_native_metric_dict(value, depth, remaining, ancestors)
+ if value_type is list:
+ return _sanitize_native_metric_list(value, depth, remaining, ancestors)
+ return _NATIVE_METRIC_INVALID_VALUE, True
+
+
+def _metric_paths_in_dict_require_safe_rejection(
+ value: dict[Any, Any], depth: int, remaining: list[int], seen: set[int]
+) -> bool:
+ """Inspect metric-bearing paths of one exact dictionary."""
+ value_id = id(value)
+ if value_id in seen:
+ return False
+ seen.add(value_id)
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ return True
+ if key in _NATIVE_METRIC_INPUT_KEYS:
+ if _sanitize_native_metric_value(nested)[1]:
+ return True
+ elif _native_metric_paths_require_safe_rejection(
+ nested, depth=depth + 1, remaining=remaining, seen=seen
+ ):
+ return True
+ return False
+
+
+def _metric_paths_in_list_require_safe_rejection(
+ value: list[Any], depth: int, remaining: list[int], seen: set[int]
+) -> bool:
+ """Inspect metric-bearing paths of one exact list."""
+ value_id = id(value)
+ if value_id in seen:
+ return False
+ seen.add(value_id)
+ return any(
+ _native_metric_paths_require_safe_rejection(
+ list.__getitem__(value, index),
+ depth=depth + 1,
+ remaining=remaining,
+ seen=seen,
+ )
+ for index in range(list.__len__(value))
+ )
+
+
+def _native_metric_paths_require_safe_rejection(
+ value: Any,
+ *,
+ depth: int = 0,
+ remaining: list[int] | None = None,
+ seen: set[int] | None = None,
+) -> bool:
+ """Find unsafe metric subtrees without consuming non-exact containers."""
+ if remaining is None:
+ remaining = [_NATIVE_METRIC_SANITIZE_MAX_VALUES]
+ if seen is None:
+ seen = set()
+ remaining[0] -= 1
+ if remaining[0] < 0 or depth > _NATIVE_METRIC_SANITIZE_MAX_DEPTH:
+ return False
+ if type(value) is dict:
+ return _metric_paths_in_dict_require_safe_rejection(
+ value, depth, remaining, seen
+ )
+ if type(value) is list:
+ return _metric_paths_in_list_require_safe_rejection(
+ value, depth, remaining, seen
+ )
+ return False
+
+
+def _replace_exact_dict_contents(
+ value: dict[Any, Any], replacement: dict[str, Any]
+) -> None:
+ """Replace exact-dict input through built-in operations only."""
+ dict.clear(value)
+ for key, item in dict.items(replacement):
+ dict.__setitem__(value, key, item)
+
+
+def _sanitize_rejected_native_metric_config(
+ data: dict[str, Any], original_data: dict[str, Any] | None
+) -> dict[str, Any]:
+ """Project a rejected config and replace Pydantic's retained input."""
+ projected, _ = _sanitize_native_metric_value(data)
+ if type(projected) is not dict:
+ projected = {_NATIVE_METRIC_INVALID_KEY: _NATIVE_METRIC_INVALID_VALUE}
+ if original_data is not None:
+ _replace_exact_dict_contents(original_data, projected)
+ return projected
+
+
+def _validate_native_metric_string(
+ value: Any, field: str, max_length: int
+) -> str | None:
+ """Validate one optional exact string in the closed native metric wrapper."""
+ if value is None:
+ return None
+ if type(value) is not str:
+ raise ValueError(f"Sunburst native metric {field} must be a string")
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(f"Sunburst native metric {field} is invalid") from ex
+ if str.__len__(value) > max_length or encoded_size > max_length * 4:
+ raise ValueError(f"Sunburst native metric {field} is too long")
+ return value
+
+
+def _inspect_native_metric_expression_type(value: Any) -> tuple[object, bool]:
+ """Inspect an exact wrapper without hashing or comparing hostile objects."""
+ if type(value) is not dict:
+ return _NATIVE_METRIC_MISSING, False
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+ expression_type: object = _NATIVE_METRIC_MISSING
+ has_native_discriminator = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key in _NATIVE_METRIC_DISCRIMINATOR_KEYS:
+ has_native_discriminator = True
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError(
+ "Sunburst native metric expressionType must be a string"
+ )
+ if item not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError(
+ "Sunburst native metric expressionType must be SIMPLE or SQL"
+ )
+ expression_type = item
+ return expression_type, has_native_discriminator
+
+
+def _validate_native_metric_column_value(value: Any) -> None:
+ """Validate a wrapper column without dispatching container or scalar hooks."""
+ if type(value) is str:
+ _validate_native_metric_string(value, "column", 255)
+ elif type(value) is dict:
+ _validate_bounded_native_column_meta(value)
+ elif value is not None:
+ raise ValueError("Sunburst native metric column must be a string or object")
+
+
+def _validate_native_metric_item(key: str, item: Any) -> object:
+ """Validate one value after its wrapper key is known to be an exact string."""
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError("Sunburst native metric expressionType must be a string")
+ _validate_native_metric_string(item, "expressionType", 10)
+ return item
+ if key in ("label", "optionName"):
+ _validate_native_metric_string(item, key, 500)
+ elif key == "sqlExpression":
+ _validate_native_metric_string(item, key, 2_000)
+ elif key == "aggregate":
+ _validate_native_metric_string(item, key, 32)
+ elif key in ("datasourceWarning", "hasCustomLabel"):
+ if item is not None and type(item) is not bool:
+ raise ValueError(f"Sunburst native metric {key} must be a boolean")
+ elif key == "column":
+ _validate_native_metric_column_value(item)
+ return _NATIVE_METRIC_MISSING
+
+
+def _validate_native_metric_wrapper(value: Any) -> dict[str, Any]:
+ """Validate and copy the closed native metric wrapper without calling hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric must be a string or object")
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+
+ validated: dict[str, Any] = {}
+ expression_type: object = _NATIVE_METRIC_MISSING
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key not in _NATIVE_METRIC_ALLOWED_KEYS:
+ raise ValueError(f"Unknown Sunburst native metric field: {key}")
+ item_expression_type = _validate_native_metric_item(key, item)
+ if item_expression_type is not _NATIVE_METRIC_MISSING:
+ expression_type = item_expression_type
+ validated[key] = item
+
+ if expression_type is _NATIVE_METRIC_MISSING:
+ raise ValueError("Sunburst native metric requires expressionType")
+ if expression_type not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError("Sunburst native metric expressionType must be SIMPLE or SQL")
+ return validated
+
+
+def _validate_non_dict_native_metric(value: Any) -> ColumnRef:
+ """Accept an existing model while rejecting hook-bearing dict subclasses."""
+ if type(value) is ColumnRef:
+ return value
+ raise ValueError("Sunburst native metric must be a string or object")
+
+
+def _native_column_meta_dict_children(
+ value: dict[Any, Any], depth: int
+) -> tuple[list[tuple[Any, int]], int]:
+ """Read exact-dict children while validating bounded primitive keys."""
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata object is too large")
+ children: list[tuple[Any, int]] = []
+ string_bytes = 0
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ raise ValueError(
+ "Sunburst native metric column metadata keys must be strings"
+ )
+ string_bytes += _native_column_meta_string_bytes(key, key=True)
+ children.append((nested, depth + 1))
+ return children, string_bytes
+
+
+def _native_column_meta_list_children(
+ value: list[Any], depth: int
+) -> list[tuple[Any, int]]:
+ """Read exact-list children without dispatching subclass hooks."""
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata array is too large")
+ return [
+ (list.__getitem__(value, index), depth + 1)
+ for index in range(list.__len__(value))
+ ]
+
+
+def _validate_native_column_meta_scalar(value: Any) -> int:
+ """Validate one exact JSON scalar and return its string-byte contribution."""
+ value_type = type(value)
+ if value_type is str:
+ return _native_column_meta_string_bytes(value)
+ if value_type is int:
+ if int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ raise ValueError(
+ "Sunburst native metric column metadata integer is too large"
+ )
+ return 0
+ if value_type is float:
+ if not math.isfinite(value):
+ raise ValueError(
+ "Sunburst native metric column metadata number must be finite"
+ )
+ return 0
+ if value_type not in (bool, type(None)):
+ raise ValueError(
+ "Sunburst native metric column metadata contains an invalid value"
+ )
+ return 0
+
+
+def _validate_bounded_native_column_meta(value: Any) -> Any:
+ """Validate an extensible frontend ColumnMeta object without invoking hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric column must be an object")
+
+ total_values = 0
+ total_string_bytes = 0
+ stack: list[tuple[Any, int]] = [(value, 0)]
+ while stack:
+ item, depth = stack.pop()
+ total_values += 1
+ if total_values > _NATIVE_COLUMN_META_MAX_TOTAL_VALUES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ if depth > _NATIVE_COLUMN_META_MAX_DEPTH:
+ raise ValueError(
+ "Sunburst native metric column metadata is too deeply nested"
+ )
+
+ item_type = type(item)
+ if item_type is dict:
+ children, key_bytes = _native_column_meta_dict_children(item, depth)
+ total_string_bytes += key_bytes
+ stack.extend(children)
+ elif item_type is list:
+ stack.extend(_native_column_meta_list_children(item, depth))
+ else:
+ total_string_bytes += _validate_native_column_meta_scalar(item)
+
+ if total_string_bytes > _NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ return value
+
+
+class SunburstNativeMetricColumn(BaseModel):
+ """Owned projection of the frontend's intentionally extensible ColumnMeta."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ type: StrictStr | None = Field(None, max_length=255)
+
+ @model_validator(mode="before")
+ @classmethod
+ def validate_bounded_column_meta(cls, value: Any) -> Any:
+ """Bound the full open object before projecting the fields we own."""
+ projected, requires_safe_rejection = _sanitize_native_metric_value(value)
+ if requires_safe_rejection:
+ if type(value) is dict:
+ if type(projected) is not dict:
+ projected = {
+ _NATIVE_METRIC_INVALID_KEY: _NATIVE_METRIC_INVALID_VALUE
+ }
+ _replace_exact_dict_contents(value, projected)
+ value = projected
+ else:
+ # Let Pydantic reject the safe scalar so the original subclass
+ # is not retained as the input of an in-validator exception.
+ return projected
+ value = _validate_bounded_native_column_meta(value)
+ snake_name = dict.get(value, "column_name", _NATIVE_METRIC_MISSING)
+ camel_name = dict.get(value, "columnName", _NATIVE_METRIC_MISSING)
+ if (
+ snake_name is not _NATIVE_METRIC_MISSING
+ and camel_name is not _NATIVE_METRIC_MISSING
+ ):
+ if type(snake_name) is not str or type(camel_name) is not str:
+ raise ValueError(
+ "column_name and columnName must both be exact strings"
+ )
+ if snake_name != camel_name:
+ raise ValueError(
+ "column_name and columnName must match when both are provided"
+ )
+ return value
+
+
+class XYNativeMetricColumn(UnknownFieldCheckMixin):
+ """Closed legacy ColumnMeta projection retained for native XY axes."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ id: StrictInt | None = Field(None, ge=0)
+ type: StrictStr | None = Field(None, max_length=255)
+ type_generic: StrictInt | None = Field(None, ge=0, le=10)
+ groupby: StrictBool | None = None
+ is_dttm: StrictBool | None = None
+ filterable: StrictBool | None = None
+ verbose_name: StrictStr | None = Field(None, max_length=500)
+ description: StrictStr | None = Field(None, max_length=2000)
+ expression: StrictStr | None = Field(None, max_length=2000)
+ database_expression: StrictStr | None = Field(None, max_length=2000)
+ python_date_format: StrictStr | None = Field(None, max_length=255)
+ option_name: StrictStr | None = Field(
+ None,
+ max_length=500,
+ validation_alias=AliasChoices("option_name", "optionName"),
+ )
+ filter_by: StrictStr | None = Field(
+ None,
+ max_length=255,
+ validation_alias=AliasChoices("filter_by", "filterBy"),
+ )
+ value: StrictStr | None = Field(None, max_length=500)
+ advanced_data_type: StrictStr | None = Field(None, max_length=255)
+ uuid: StrictStr | None = Field(None, max_length=64)
+
+
+class SunburstLegacySavedFields(BaseModel):
+ """Known obsolete controls tolerated only while reading saved Sunbursts."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ country_fieldtype: StrictStr | None = Field(None, max_length=255)
+ entity: StrictStr | None = Field(None, max_length=255)
+ granularity: StrictStr | None = Field(None, max_length=255)
+ limit: StrictInt | StrictStr | None = None
+ markup_type: Literal["markdown", "html"] | None = None
+ show_bubbles: StrictBool | None = None
+
+ @field_validator("limit")
+ @classmethod
+ def validate_legacy_limit(cls, value: int | str | None) -> int | str | None:
+ """Bound the ignored legacy row limit and reject non-numeric strings."""
+ if value is None:
+ return None
+ if isinstance(value, str):
+ if not value.isascii() or not value.isdigit():
+ raise ValueError("legacy Sunburst limit must be an integer")
+ numeric_value = int(value)
+ else:
+ numeric_value = value
+ if not 1 <= numeric_value <= 50000:
+ raise ValueError("legacy Sunburst limit must be between 1 and 50000")
+ return value
+
+
+_NATIVE_FORM_DATA_MARKER = "_mcp_native_form_data"
+_SUNBURST_IGNORED_LEGACY_FIELDS = frozenset(
+ {
+ "country_fieldtype",
+ "entity",
+ "granularity",
+ "limit",
+ "markup_type",
+ "show_bubbles",
+ }
+)
+
+
+class SunburstChartConfig(BaseChartConfig):
+ """Config for the ECharts Sunburst plugin (viz_type ``sunburst_v2``)."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ chart_type: Literal["sunburst"] = "sunburst"
+ viz_type: Literal["sunburst_v2"] = Field(
+ "sunburst_v2",
+ description="Exact Superset frontend visualization tag",
+ )
+ hierarchy: List[ColumnRef] = Field(
+ ...,
+ min_length=1,
+ description=(
+ "Hierarchy dimensions in ring order, from the innermost/root level "
+ "to the outermost/leaf level"
+ ),
+ validation_alias=AliasChoices("hierarchy", "columns", "groupby"),
+ )
+ metric: ColumnRef = Field(
+ ...,
+ description=(
+ "Primary metric used to size arcs. Use aggregate for a SIMPLE "
+ "adhoc metric, saved_metric=True for a dataset metric, or "
+ "sql_expression plus label for a SQL metric."
+ ),
+ )
+ secondary_metric: ColumnRef | None = Field(
+ None,
+ description=(
+ "Optional metric whose ratio to the primary metric drives the "
+ "sequential color scale (for example, profit/revenue represents "
+ "margin). When omitted, arc colors are categorical."
+ ),
+ validation_alias=AliasChoices("secondary_metric", "secondaryMetric"),
+ )
+ filters: List[FilterConfig] | None = Field(
+ None,
+ description=(
+ "Structured WHERE filters (column/op/value/clause). An omitted list is "
+ "preserved on updates; an explicit [] clears saved filters."
+ ),
+ )
+ time_range: str | None = Field(
+ None,
+ min_length=1,
+ max_length=1000,
+ description=(
+ "Superset time range, for example 'Last year', "
+ "'2025-01-01 : 2025-12-31', or 'No filter'"
+ ),
+ )
+ time_grain: TimeGrain | None = Field(
+ None,
+ description="Optional bucket for temporal hierarchy columns",
+ validation_alias=AliasChoices("time_grain", "time_grain_sqla"),
+ )
+ sort_by_metric: bool = Field(
+ False,
+ description=(
+ "Order hierarchy rows by the primary metric descending before "
+ "applying row_limit, matching the frontend buildQuery transform"
+ ),
+ )
+ row_limit: int = Field(10000, description="Maximum hierarchy rows", ge=1, le=50000)
+ color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Categorical scheme used when secondary_metric is omitted",
+ )
+ linear_color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Sequential scheme used when secondary_metric is present",
+ )
+ show_labels: bool = False
+ show_labels_threshold: float = Field(
+ 5,
+ ge=0,
+ le=100,
+ description="Minimum arc size in percentage points for showing a label",
+ )
+ show_total: bool = False
+ show_null_values: bool = Field(
+ True,
+ description="Keep null-valued hierarchy nodes in the rendered tree",
+ )
+ label_type: Literal["key", "value", "key_value"] = "key"
+ number_format: str = Field("SMART_NUMBER", min_length=1, max_length=50)
+ date_format: str = Field("smart_date", min_length=1, max_length=50)
+ currency_format: CurrencyFormat | None = None
+ # Bounded native Explore envelope/UI state. These keys occur in real saved
+ # form_data and are safe to round-trip, but remain typed so native mode does
+ # not become an escape hatch for arbitrary plugin keys or malformed nesting.
+ annotation_layers: list[dict[str, JsonValue]] | None = None
+ dashboard_id: int | None = Field(
+ None,
+ validation_alias=AliasChoices("dashboard_id", "dashboardId"),
+ serialization_alias="dashboardId",
+ )
+ dashboards: list[int] | None = None
+ datasource: str | None = Field(None, min_length=1, max_length=255)
+ extra_form_data: dict[str, JsonValue] | None = None
+ slice_id: int | None = None
+ url_params: dict[str, JsonValue] | None = None
+ standardized_form_data: SunburstStandardizedFormData | None = Field(
+ None,
+ validation_alias=AliasChoices("standardized_form_data", "standardizedFormData"),
+ serialization_alias="standardizedFormData",
+ )
+ since: str | None = Field(None, max_length=1000)
+ until: str | None = Field(None, max_length=1000)
+ time_compare: str | list[str] | None = None
+ compare_lag: int | str | None = None
+ compare_suffix: str | None = Field(None, max_length=255)
+
+ @staticmethod
+ def _looks_like_native_form_data(data: Any) -> bool:
+ """Identify saved Explore payloads without weakening typed typo checks."""
+ if type(data) is not dict:
+ return False
+ native_marker = dict.get(data, _NATIVE_FORM_DATA_MARKER)
+ viz_type = dict.get(data, "viz_type")
+ if not (
+ (type(native_marker) is bool and native_marker)
+ or (type(viz_type) is str and viz_type == "sunburst_v2")
+ ):
+ return False
+ metric = dict.get(data, "metric")
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(metric)
+ )
+ return (
+ any(
+ key in data
+ for key in (
+ "adhoc_filters",
+ "annotation_layers",
+ "datasource",
+ "extra_form_data",
+ "since",
+ "slice_id",
+ "standardizedFormData",
+ "until",
+ )
+ )
+ or type(metric) is str
+ or expression_type is not _NATIVE_METRIC_MISSING
+ or has_native_discriminator
+ )
+
+ @staticmethod
+ def _coerce_native_metric(
+ value: Any, *, allow_extensible_column_meta: bool = True
+ ) -> Any:
+ """Accept saved metric names and native SIMPLE/SQL metric objects."""
+ if type(value) is str:
+ return {"name": value, "saved_metric": True}
+ if type(value) is dict:
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(value)
+ )
+ if expression_type is _NATIVE_METRIC_MISSING:
+ if has_native_discriminator:
+ raise ValueError("Sunburst native metric requires expressionType")
+ return value
+ value = _validate_native_metric_wrapper(value)
+ else:
+ return _validate_non_dict_native_metric(value)
+
+ expression_type = value.get("expressionType")
+ label = value.get("label")
+ has_custom_label = value.get("hasCustomLabel")
+ if expression_type == "SQL":
+ sql_expression = value.get("sqlExpression")
+ return {
+ "sql_expression": sql_expression,
+ # The frontend still assigns a result label when the label is
+ # not custom. Preserve a nonempty native label first, then use
+ # the SQL expression as the same effective fallback.
+ "label": label or sql_expression,
+ "has_custom_label": has_custom_label,
+ }
+ if expression_type == "SIMPLE":
+ column = value.get("column")
+ if type(column) is dict:
+ column_model = (
+ SunburstNativeMetricColumn
+ if allow_extensible_column_meta
+ else XYNativeMetricColumn
+ )
+ column_metadata = column_model.model_validate(column)
+ name = column_metadata.column_name
+ dtype = column_metadata.type
+ elif type(column) is str:
+ name = column
+ dtype = None
+ else:
+ raise ValueError(
+ "Sunburst SIMPLE metric column must be a column name or object"
+ )
+ return {
+ "name": name,
+ "aggregate": value.get("aggregate"),
+ "label": label,
+ "has_custom_label": has_custom_label,
+ "dtype": dtype,
+ }
+ return value
+
+ @staticmethod
+ def _coerce_native_hierarchy(data: dict[str, Any]) -> None:
+ """Normalize native ``columns``/``groupby`` dimension shortcuts."""
+ hierarchy_key = next(
+ (key for key in ("hierarchy", "columns", "groupby") if key in data),
+ None,
+ )
+ if hierarchy_key is None:
+ return
+ hierarchy = data[hierarchy_key]
+ if isinstance(hierarchy, (str, dict)):
+ hierarchy = [hierarchy]
+ if isinstance(hierarchy, list):
+ data[hierarchy_key] = [
+ {"name": item} if isinstance(item, str) else item for item in hierarchy
+ ]
+ if hierarchy_key != "groupby":
+ # Saved Sunburst payloads sometimes retain an empty groupby from a
+ # previous plugin. The explicit columns/hierarchy control wins.
+ data.pop("groupby", None)
+
+ @staticmethod
+ def _coerce_native_filter(
+ native_filter: Any, data: dict[str, Any]
+ ) -> dict[str, Any] | None:
+ """Convert one native SIMPLE filter, extracting temporal controls."""
+ if not isinstance(native_filter, dict):
+ raise ValueError("Each Sunburst native adhoc_filter must be an object")
+ allowed_keys = {
+ "clause",
+ "comparator",
+ "datasourceWarning",
+ "expressionType",
+ "filterOptionName",
+ "isExtra",
+ "operator",
+ "sqlExpression",
+ "subject",
+ }
+ if unknown := set(native_filter) - allowed_keys:
+ raise ValueError(
+ "Unknown Sunburst native adhoc_filter field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ if native_filter.get("expressionType") != "SIMPLE":
+ raise ValueError(
+ "Sunburst native adhoc_filters round-trip supports SIMPLE filters "
+ "only; express filters with the typed 'filters' field"
+ )
+ operator = native_filter.get("operator")
+ clause = native_filter.get("clause", "WHERE")
+ if not isinstance(clause, str) or clause != "WHERE":
+ raise ValueError(
+ "Sunburst native SIMPLE filter clause must be 'WHERE'; "
+ "SIMPLE HAVING is unsupported because the shared query mapper "
+ "cannot preserve HAVING semantics"
+ )
+ if operator == "TEMPORAL_RANGE":
+ if "temporal_column" not in data and "granularity_sqla" not in data:
+ data["temporal_column"] = native_filter.get("subject")
+ data.setdefault("time_range", native_filter.get("comparator"))
+ return None
Review Comment:
Fixed in 6cf22d8bb1. Native TEMPORAL_RANGE predicates are now collected and promoted together. Neutral ("No filter") predicates restrict nothing. Duplicate identical ranges collapse into one. More than one distinct restricting range is rejected, since the typed config holds a single `temporal_column`/`time_range`. A range that conflicts with an explicit saved time subject or range is also rejected. These now raise instead of losing the ship-date restriction. Tests: `test_native_multiple_temporal_ranges_are_rejected_not_dropped`, `test_native_restricting_temporal_range_wins_over_neutral_binding`, `test_native_temporal_range_on_another_bound_column_is_rejected`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4190740401
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -3867,6 +5070,133 @@ def _normalize_chart_request_input(data: Any) -> Any:
config.pop("viz_type", None)
elif config.get("chart_type") != "table":
config.pop("viz_type", None)
+
+ if config.get("chart_type") == "xy":
+ # Native Timeseries form data uses ``x_axis`` for the semantic
+ # query column, while the typed MCP contract uses the same spelling
+ # for AxisConfig presentation state. Disambiguate before the
+ # discriminated union runs: scalars and native query-column objects
+ # become ``x``; title/format/scale mappings remain ``x_axis``.
+ if "x_axis" in config and config["x_axis"] is not None:
+ native_x_axis = config["x_axis"]
+ is_native_column = not isinstance(native_x_axis, dict) or bool(
+ {
+ "column",
+ "column_name",
+ "columnName",
+ "expressionType",
+ "sqlExpression",
+ }
+ & set(native_x_axis)
+ )
+ if is_native_column:
+ if "x" in config:
+ raise ValueError(
+ "XY config cannot provide both semantic x and native "
+ "x_axis query columns"
+ )
+ if isinstance(native_x_axis, dict):
+ allowed_x_axis_keys = {
+ "column",
+ "column_name",
+ "columnName",
+ "columnType",
+ "expressionType",
+ "label",
+ "sqlExpression",
+ "timeGrain",
+ }
+ if unknown := set(native_x_axis) - allowed_x_axis_keys:
+ raise ValueError(
+ "Unknown XY native x_axis field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ expression_type = native_x_axis.get("expressionType")
+ if expression_type == "SIMPLE":
+ column = native_x_axis.get("column")
+ if isinstance(column, dict):
+ column = XYNativeMetricColumn.model_validate(
+ column
+ ).column_name
+ native_x_axis = {
+ "name": native_x_axis.get("column_name")
+ or native_x_axis.get("columnName")
+ or column,
+ }
+ elif (
+ expression_type == "SQL"
+ and native_x_axis.get("columnType") == "BASE_AXIS"
+ ):
+ native_x_axis = {
+ "name": native_x_axis.get("sqlExpression"),
+ "label": native_x_axis.get("label"),
+ }
+ elif expression_type == "SQL":
+ native_x_axis = {
+ "sql_expression": native_x_axis.get("sqlExpression"),
+ "label": native_x_axis.get("label"),
+ }
+ elif "column_name" in native_x_axis:
+ native_x_axis = {
+ **native_x_axis,
+ "name": native_x_axis.get("column_name"),
+ }
+ config["x"] = native_x_axis
+ config.pop("x_axis", None)
+
+ def collect_axis_config(
+ field_name: str,
+ native_keys: dict[str, str],
+ ) -> None:
+ axis = config.get(field_name)
+ axis_state = dict(axis) if isinstance(axis, dict) else {}
+ found = isinstance(axis, dict)
+ for native_key, typed_key in native_keys.items():
+ if native_key in config:
+ axis_state[typed_key] = config.pop(native_key)
+ found = True
+ if found:
+ config[field_name] = axis_state
+
+ collect_axis_config(
+ "x_axis",
+ {"x_axis_title": "title", "x_axis_format": "format"},
+ )
+ collect_axis_config(
+ "y_axis",
+ {
+ "y_axis_title": "title",
+ "y_axis_format": "format",
+ "y_axis_scale": "scale",
+ },
+ )
+ if "legendOrientation" in config:
+ config["legend_orientation"] = config.pop("legendOrientation")
+
+ # These native query/presentation values are deterministically
+ # reconstructed from semantic x plus dataset temporal metadata.
+ config.pop("granularity_sqla", None)
+ config.pop("x_axis_sort_series_type", None)
+ config.pop("x_axis_sort_series_ascending", None)
+
+ # Saved metrics are string references; native SIMPLE/SQL objects
+ # carry expressionType metadata. Typed ``y`` strings keep their
+ # existing column-with-default-SUM semantics.
+ if "y" not in config and "metrics" in config:
Review Comment:
A typed `xy` request that still uses the `metrics` alias, e.g. `{"chart_type":"xy","x":"region","metrics":["revenue"]}`, is now rewritten into saved-metric references. On master that alias fed `y`, where a string is a physical column with the default `SUM`. So a request against a numeric `revenue` column with no saved metric of that name now fails saved-metric validation, and if both exist it silently queries the saved metric instead. Should this rewrite be limited to native Explore payloads (the `_NATIVE_FORM_DATA_MARKER` path) so the typed `metrics` alias keeps its current meaning?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,775 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ if getattr(config, "filters", None) == []:
Review Comment:
On an update that supplies non-empty `filters`, the `== []` branch is skipped and the `is None` branch is skipped too, so a saved legacy `filters` key (kept through `_FORM_DATA_UPDATE_PRESERVE_KEYS` on a cross-viz update) stays next to the newly generated `adhoc_filters`. E.g. converting a Table saved with `filters=[{"col":"region","op":"==","val":"EMEA"}]` to Sunburst with a typed filter selecting APAC combines both predicates at query time and returns no rows instead of the requested replacement. Should the legacy `filters`/`extra_filters`/`where`/`having` keys also be dropped when new filters replace the saved ones?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4191582413
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1272 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": form_data.get("time_grain_sqla"),
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ query.setdefault("metrics", []).append(sort_metric)
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ )
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
+ extras = query.get("extras")
+ if isinstance(extras, dict):
+ extras.pop("time_grain_sqla", None)
+ if not extras:
+ query.pop("extras", None)
+ elif not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
query["post_processing"] = post_processing
- # ``granularity`` does two jobs downstream: it names the temporal column the
- # time range filters on, and it is the column ``time_grain_sqla`` buckets
- # (``models/helpers.py`` swaps a selected column for its timestamp expression
- # when that column equals ``granularity``). Only the first job depends on
- # there being an active range, so set it whenever form data carries one โ
- # matching ``extractExtras.ts``, which sets it unconditionally. Gating it on
- # ``time_range`` dropped the bucketing, so an ordinary "all-time totals by
- # month" chart exported one row per raw timestamp instead of one per month.
- if granularity := form_data.get("granularity") or form_data.get("granularity_sqla"):
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
Review Comment:
Fixed in b6e34104724dd7a1873981c0f84d397810a813d0. Every MCP rebuild now sets the requested result_format (Excel maps to xlsx), result_type=results, and the effective row_limit. Tables use the non-paginated download path, so the saved page size cannot override the request and no rowcount query is added. test_rebuilt_table_data_does_not_use_saved_server_pagination covers unsaved, cached-update, and saved-fallback paths: CSV/Excel export 100 rows despite a 10-row page, and limit=1 queries one row despite a 1,000-row page. These cases failed before the fix and pass after.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1272 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": form_data.get("time_grain_sqla"),
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ query.setdefault("metrics", []).append(sort_metric)
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ )
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
+ extras = query.get("extras")
+ if isinstance(extras, dict):
+ extras.pop("time_grain_sqla", None)
+ if not extras:
+ query.pop("extras", None)
+ elif not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
query["post_processing"] = post_processing
- # ``granularity`` does two jobs downstream: it names the temporal column the
- # time range filters on, and it is the column ``time_grain_sqla`` buckets
- # (``models/helpers.py`` swaps a selected column for its timestamp expression
- # when that column equals ``granularity``). Only the first job depends on
- # there being an active range, so set it whenever form data carries one โ
- # matching ``extractExtras.ts``, which sets it unconditionally. Gating it on
- # ``time_range`` dropped the bucketing, so an ordinary "all-time totals by
- # month" chart exported one row per raw timestamp instead of one per month.
- if granularity := form_data.get("granularity") or form_data.get("granularity_sqla"):
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
+ page_size = int(form_data.get("server_page_length") or 0)
+ configured_limit = int(form_data.get("row_limit") or 0)
+ query["row_limit"] = (
+ min(page_size, configured_limit) if configured_limit else page_size
+ )
+ query["row_offset"] = 0
+
+ queries = [query]
+ if form_data.get("percent_metric_calculation") == "all_records" and percent_metrics:
+ all_records = dict(query)
+ all_records.update(
+ {
+ "columns": [],
+ "metrics": percent_metrics,
+ "post_processing": [],
+ "row_limit": 0,
+ "row_offset": 0,
+ "orderby": [],
+ "is_timeseries": False,
+ }
+ )
+ queries.append(all_records)
+ if form_data.get("show_totals") and metrics:
+ totals_aggregate = form_data.get("totals_aggregate")
+ if totals_aggregate not in {"SUM", "AVG"}:
+ totals_aggregate = None
+ totals_metrics = [
+ {
+ **metric,
+ "aggregate": totals_aggregate,
+ }
+ if totals_aggregate
+ and isinstance(metric, dict)
+ and metric.get("expressionType") == "SIMPLE"
+ else metric
+ for metric in metrics
+ ]
+ totals = dict(query)
+ totals.update(
+ {
+ "columns": [],
+ "metrics": totals_metrics,
+ "row_limit": 0,
+ "row_offset": 0,
+ "post_processing": [contribution] if contribution else [],
+ }
+ )
+ totals.pop("order_desc", None)
+ totals.pop("orderby", None)
+ queries.append(totals)
+ if form_data.get("server_pagination") and not is_download:
+ rowcount = dict(query)
+ rowcount.update(
+ {
+ "time_offsets": [],
+ "row_limit": int(form_data.get("row_limit") or 0),
+ "row_offset": 0,
+ "post_processing": [],
+ "is_rowcount": True,
+ }
+ )
+ queries.insert(1, rowcount)
+ return queries
+
+
+def _treemap_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ """Apply Treemap's bounded metric and hierarchy ordering contract."""
+ hierarchy = _as_list(form_data.get("groupby"))
+ metric = form_data.get("metric")
+ query["columns"] = hierarchy
+ query["metrics"] = [metric] if metric is not None else []
+ granularity = form_data.get("granularity", form_data.get("granularity_sqla"))
+ if granularity is not None:
query["granularity"] = granularity
- if form_data.get("row_limit"):
- query["row_limit"] = form_data["row_limit"]
+ ordering = [[metric, False]] if form_data.get("sort_by_metric") and metric else []
+ ordering.extend(
+ [column, True] for column in hierarchy if isinstance(column, str) and column
+ )
+ try:
+ bounded = float(query.get("row_limit") or 0) != 0
+ except (ValueError, TypeError):
+ bounded = True
+ query.pop("orderby", None)
+ if bounded and ordering:
+ query["orderby"] = ordering
+
+
+def _bubble_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ """Preserve Bubble's entity/series roles and explicit sort-metric direction."""
+ query["columns"] = _deduplicate_fields(
+ [*_as_list(form_data.get("entity")), *_as_list(form_data.get("series"))]
+ )
+ query.pop("orderby", None)
+ if sort_metric := next(iter(_as_list(form_data.get("orderby"))), None):
+ query["orderby"] = [[sort_metric, not query.get("order_desc", True)]]
+
+
+def _entity_map_query(
+ form_data: dict[str, Any], query: dict[str, Any], *, world_map: bool
+) -> None:
+ """Group World/Country Map queries by their ``entity`` role.
+
+ Mirrors the World Map and Country Map ``buildQuery`` transforms: the
+ ``entity`` control is the only grouping column, Country Map selects one
+ ``metric``, and World Map adds a distinct ``secondary_metric`` and orders by
+ the primary metric only when ``sort_by_metric`` is enabled.
+ """
+ entity = form_data.get("entity")
+ query["columns"] = [entity] if entity else []
+ metric = form_data.get("metric")
+ metrics = [metric] if metric else []
+ if world_map:
+ secondary_metric = form_data.get("secondary_metric")
+ if secondary_metric and (
+ not metric
+ or _label(secondary_metric, metric=True) != _label(metric, metric=True)
+ ):
+ metrics.append(secondary_metric)
+ query.pop("orderby", None)
+ if form_data.get("sort_by_metric") and metric:
+ query["orderby"] = [[metric, False]]
+ query["metrics"] = metrics
+
+
+def build_query_objects_from_form_data( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ viz_type: str | None = None,
+ row_limit: int | None = None,
+ order_desc: bool | None = None,
+ filters_prepared: bool = False,
+ secondary_form_data: dict[str, Any] | None = None,
+) -> list[dict[str, Any]]:
+ """Build frontend-equivalent QueryObject dictionaries from form data.
+
+ Every real MCP product path and the legacy common fallback share this
+ extractor and the chart adapters below. This prevents compile/SQL/preview
+ and saved/fallback data paths from silently selecting different fields.
+ """
+ effective_viz = viz_type or str(form_data.get("viz_type") or "")
+ extraction_form_data = form_data
+ if is_raw_query_mode(form_data):
+ # ``extractQueryFields`` is strict, while the legacy server fallback has
+ # always kept old saved tables executable by dropping malformed order
+ # entries. Normalize that bounded raw-table input before invoking the
+ # shared extractor; valid entries still use the centralized contract.
+ extraction_form_data = {
+ **form_data,
+ "query_mode": "raw",
+ "order_by_cols": orderby_from_form_data(form_data, [], effective_viz),
+ }
+ query = _base_query_object(
+ extraction_form_data,
+ row_limit=row_limit,
+ order_desc=order_desc,
+ filters_prepared=filters_prepared,
+ )
+
+ if effective_viz in {"table", "ag-grid-table"}:
+ return _finalize_query_objects(form_data, _table_queries(form_data, query))
+ if effective_viz == "histogram_v2":
+ _histogram_query(form_data, query)
+ elif effective_viz == "box_plot":
+ _box_plot_query(form_data, query)
+ elif effective_viz == "pivot_table_v2":
+ _pivot_query(form_data, query)
+ elif effective_viz == "waterfall":
+ _waterfall_query(form_data, query)
+ elif effective_viz in {"gantt", "gantt_chart"}:
+ _gantt_query(form_data, query)
+ elif effective_viz.startswith("echarts_timeseries") or effective_viz in {
+ "echarts_area",
+ "mixed_timeseries",
+ }:
+ _timeseries_query(form_data, query)
+ elif effective_viz == "big_number":
+ return _finalize_query_objects(form_data, _big_number_queries(form_data, query))
+ elif effective_viz == "big_number_total":
+ query["columns"] = []
+ elif effective_viz == "handlebars":
+ _normalize_query_orderby(query)
+ elif effective_viz == "pie":
+ if form_data.get("sort_by_metric") and form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ query["post_processing"] = _pie_contribution_post_processing(
+ list(query.get("metrics") or [])
+ )
+ elif effective_viz == "gauge_chart":
+ if form_data.get("sort_by_metric") and form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ elif effective_viz == "treemap_v2":
+ _treemap_query(form_data, query)
+ elif effective_viz == "sunburst_v2":
+ if not form_data.get("sort_by_metric"):
+ query["orderby"] = []
+ elif form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ elif effective_viz in {"bubble", "bubble_v2"}:
+ _bubble_query(form_data, query)
+ elif effective_viz in {"world_map", "country_map"}:
+ _entity_map_query(form_data, query, world_map=effective_viz == "world_map")
+ elif effective_viz == "ag-grid-pivot-table":
+ query["columns"] = _temporalized_columns(
+ form_data, _as_list(form_data.get("groupby"))
+ )
+
+ if not effective_viz and not query.get("orderby"):
Review Comment:
Fixed in b6e34104724dd7a1873981c0f84d397810a813d0. The metric-ordering fallback runs in the unhandled-adapter branch, including nonempty viz types. Handled adapters can still deliberately leave ordering empty. test_unadapted_viz_keeps_metric_ordering covers Funnel, an unregistered visualization, and an empty visualization through both the common query-context builder and MCP query builder. The nonempty cases failed before and pass after, with ORDER BY revenue descending before LIMIT 2.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,777 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ requested_filters = getattr(config, "filters", None)
+ if requested_filters is not None:
+ # Legacy ``filters``/``where``/``having`` are reconstructed into adhoc
+ # filters at query time, so an explicit clear or replacement removes
+ # every saved source the new form data does not set itself.
+ for key in _SAVED_PREDICATE_FORM_DATA_KEYS - new_form_data.keys():
+ merged.pop(key, None)
+ else:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool(
+ {"temporal_column", "time_grain", "time_range"} & fields_set
+ ),
+ )
+ if filters is not None:
+ merged["adhoc_filters"] = filters
+ return merged
+
+
+def retain_mixed_timeseries_secondary_update_state(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Return saved query-B controls a same-viz Mixed Timeseries update keeps.
+
+ Query B inherits unsuffixed controls only when the suffixed key is absent.
+ Preserve explicit native clears for controls the typed mapper did not
+ replace, so []/None never turns into accidental inheritance from query A.
+ Valid comparison state is also retained; malformed/stale dataset roles
+ remain fail-closed and are dropped.
+ """
+ from superset.common.form_data_query_context import (
+ MIXED_TIMESERIES_SECONDARY_QUERY_KEYS,
+ )
+
+ retained: Dict[str, Any] = {}
+ for key in MIXED_TIMESERIES_SECONDARY_QUERY_KEYS:
Review Comment:
Fixed in b6e34104724dd7a1873981c0f84d397810a813d0. Omitted modeled query-B controls are validated against their schema and restored, including time_range_b, time_grain_sqla_b, row limits/offsets, series limits, and rolling/resample controls. Newly supplied controls and explicit clears take precedence. test_mixed_update_retains_omitted_secondary_query_controls failed before and passes after; it compares the rebuilt secondary query before/after a presentation-only update and also checks explicit replacements. The stale-role fixture uses malformed secondary controls rather than valid string-valued native settings.
##########
superset/mcp_service/utils/serialization.py:
##########
@@ -170,24 +179,30 @@ def _sanitize_other(value: Any, depth: int) -> Any:
# FastMCP's ``fallback=str``, which the structured-content path
# (``to_jsonable_python``) does not apply.
return _sanitize_str(str(value))
- return sanitize_json_value(converted, depth + 1)
+ return sanitize_json_value(
+ converted, depth + 1, preserve_decimals=preserve_decimals
+ )
-def sanitize_json_value(value: Any, depth: int = 0) -> Any:
+def sanitize_json_value(
+ value: Any, depth: int = 0, *, preserve_decimals: bool = False
+) -> Any:
"""Return ``value`` with anything ``pydantic_core.to_json`` cannot encode
replaced by a JSON-safe equivalent."""
+ if preserve_decimals and type(value) is Decimal:
Review Comment:
Fixed in b6e34104724dd7a1873981c0f84d397810a813d0. Finite Decimals keep the existing numeric JSON wire representation. JSON-only serializers project ExactJsonSafeMapping/Values to numbers, while Python values remain exact for validation, statistics, and CSV export. test_chart_decimal_wire_values_remain_numbers covers Table, Pie, and Sunburst data, query_results, and column samples under schema_version 2.0; it failed before and passes after. Existing FastMCP Sunburst tests also check numeric JSON values while retaining precise CSV output. The projection is documented.
##########
tests/unit_tests/mcp_service/chart/test_native_form_data_and_dataset_rebind_contracts.py:
##########
@@ -0,0 +1,776 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from copy import deepcopy
+from unittest.mock import MagicMock, patch
+
+import pytest
+from pydantic import ValidationError
+
+from superset.mcp_service.chart.chart_helpers import (
+ build_query_context_from_form_data,
+ build_query_dicts_from_form_data,
+)
+from superset.mcp_service.chart.chart_utils import (
+ map_config_to_form_data,
+ merge_form_data_for_update,
+ scrub_dataset_bound_form_data,
+)
+from superset.mcp_service.chart.schemas import (
+ GenerateChartRequest,
+ SunburstChartConfig,
+ TableChartConfig,
+ UpdateChartPreviewRequest,
+ UpdateChartRequest,
+)
+
+
+@pytest.mark.parametrize(
+ "form_data,expected_columns,expected_metrics,expected_queries",
+ [
+ (
+ {
+ "viz_type": "histogram_v2",
+ "column": "revenue",
+ "groupby": ["region"],
+ "bins": 8,
+ },
+ ["region", "revenue"],
+ [],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "pivot_table_v2",
+ "groupbyRows": ["region"],
+ "groupbyColumns": ["product"],
+ "metrics": ["sum_revenue"],
+ },
+ ["region", "product"],
+ ["sum_revenue"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "waterfall",
+ "x_axis": "month",
+ "groupby": ["region"],
+ "metric": "sum_revenue",
+ },
+ ["month", "region"],
+ ["sum_revenue"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "gantt_chart",
+ "start_time": "started_at",
+ "end_time": "ended_at",
+ "y_axis": "task",
+ "series": ["team"],
+ "tooltip_columns": ["owner"],
+ "tooltip_metrics": ["duration"],
+ "order_by_cols": ['["started_at", true]'],
+ },
+ ["started_at", "ended_at", "task", "team", "owner"],
+ ["duration"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "mixed_timeseries",
+ "x_axis": "ds",
+ "groupby": ["region"],
+ "metrics": ["revenue"],
+ "groupby_b": ["product"],
+ "metrics_b": ["profit"],
+ },
+ ["ds", "region"],
+ ["revenue"],
+ 2,
+ ),
+ (
+ {
+ "viz_type": "table",
+ "query_mode": "raw",
+ "all_columns": ["region", "revenue"],
+ "order_by_cols": ['["revenue", false]'],
+ },
+ ["region", "revenue"],
+ [],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "echarts_timeseries_line",
+ "x_axis": "ds",
+ "groupby": ["region"],
+ "metrics": ["revenue"],
+ "series_limit": 5,
+ "series_limit_metric": "revenue",
+ "order_desc": False,
+ },
+ ["ds", "region"],
+ ["revenue"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "box_plot",
+ "columns": ["revenue"],
+ "groupby": ["region"],
+ "metrics": ["avg_revenue"],
+ "whiskerOptions": "Tukey",
+ },
+ ["revenue", "region"],
+ ["avg_revenue"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "ag-grid-pivot-table",
+ "groupby": ["region", "product"],
+ "metrics": ["revenue"],
+ },
+ ["region", "product"],
+ ["revenue"],
+ 1,
+ ),
+ ],
+ ids=[
+ "histogram",
+ "pivot",
+ "waterfall",
+ "gantt",
+ "mixed",
+ "raw-table",
+ "xy",
+ "box-plot",
+ "interactive-pivot",
+ ],
+)
+def test_product_query_context_uses_registered_frontend_adapters(
+ form_data: dict[str, object],
+ expected_columns: list[object],
+ expected_metrics: list[object],
+ expected_queries: int,
+) -> None:
+ """Exercise production QueryObject dictionaries at the factory boundary."""
+ factory = MagicMock()
+ factory.create.return_value = object()
+ query_form_data = {"datasource": "7__table", **deepcopy(form_data)}
+ with (
+ patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ),
+ patch(
+ "superset.common.query_context_factory.QueryContextFactory",
+ return_value=factory,
+ ),
+ ):
+ build_query_context_from_form_data(query_form_data)
+
+ queries = factory.create.call_args.kwargs["queries"]
+ assert len(queries) == expected_queries
+ if (
+ isinstance(form_data.get("x_axis"), str)
+ and form_data["x_axis"] in expected_columns
+ ):
+ expected_columns = [
+ {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": form_data["x_axis"],
+ "label": form_data["x_axis"],
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ if column == form_data["x_axis"]
+ else column
+ for column in expected_columns
+ ]
+ assert queries[0]["columns"] == expected_columns
+ assert queries[0]["metrics"] == expected_metrics
+
+ if form_data["viz_type"] == "histogram_v2":
+ assert queries[0]["post_processing"][0]["operation"] == "histogram"
+ if form_data["viz_type"] == "box_plot":
+ assert queries[0]["post_processing"][0]["operation"] == "boxplot"
+ if form_data["viz_type"] == "echarts_timeseries_line":
+ assert queries[0]["series_columns"] == ["region"]
+ assert queries[0]["orderby"] == [["revenue", True]]
+ assert [rule["operation"] for rule in queries[0]["post_processing"]] == [
+ "pivot",
+ "flatten",
+ ]
+ if form_data["viz_type"] == "waterfall":
+ assert queries[0]["orderby"] == [["month", True], ["region", True]]
+ if form_data["viz_type"] == "gantt_chart":
+ assert queries[0]["series_columns"] == ["team"]
+ assert queries[0]["orderby"] == [["started_at", True]]
+ if form_data["viz_type"] == "mixed_timeseries":
+ assert queries[1]["columns"] == [
+ {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": "ds",
+ "label": "ds",
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ },
+ "product",
+ ]
+ assert queries[1]["metrics"] == ["profit"]
+
+
+@pytest.mark.parametrize("secondary_key", ["adhoc_filters_b", "filters_b"])
+def test_mixed_secondary_explicit_filter_clear_never_inherits_primary(
+ secondary_key: str,
+) -> None:
+ primary_filter = {
+ "clause": "WHERE",
+ "expressionType": "SIMPLE",
+ "subject": "region",
+ "operator": "==",
+ "comparator": "EMEA",
+ }
+ form_data = {
+ "viz_type": "mixed_timeseries",
+ "x_axis": "ds",
+ "metrics": ["revenue"],
+ "metrics_b": ["profit"],
+ "adhoc_filters": [primary_filter],
+ secondary_key: [],
+ "time_range": "Last year",
+ "time_range_b": None,
+ }
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ primary, secondary = build_query_dicts_from_form_data(form_data, 7, "table")
+
+ assert primary["filters"] == [{"col": "region", "op": "==", "val": "EMEA"}]
+ assert secondary["filters"] == []
+ assert secondary.get("time_range") is None
+
+
+@pytest.mark.parametrize(
+ "viz_type,roles",
+ [
+ ("table", {"all_columns": ["old"], "order_by_cols": ['["old", true]']}),
+ ("pivot_table_v2", {"groupbyRows": ["old"], "groupbyColumns": ["old2"]}),
+ ("histogram_v2", {"column": "old", "groupby": ["old2"]}),
+ ("waterfall", {"x_axis": "old", "groupby": ["old2"]}),
+ ("echarts_timeseries_line", {"x_axis": "old", "series_columns": ["old2"]}),
+ ("mixed_timeseries", {"metrics_b": ["old"], "adhoc_filters_b": []}),
+ ("gantt_chart", {"start_time": "old", "tooltip_columns": ["old2"]}),
+ ("world_map", {"entity": "old", "metric": "old_metric"}),
+ ("deck_scatter", {"spatial": {"type": "latlong", "lonCol": "old"}}),
+ ],
+)
+def test_dataset_rebind_scrubs_complete_query_role_contracts(
+ viz_type: str, roles: dict[str, object]
+) -> None:
+ scrubbed = scrub_dataset_bound_form_data(
+ {"viz_type": viz_type, **roles, "show_legend": True}
+ )
+ assert set(roles).isdisjoint(scrubbed)
+ assert scrubbed == {"viz_type": viz_type, "show_legend": True}
+
+
+def test_dataset_rebind_fails_closed_without_a_complete_viz_contract() -> None:
+ with pytest.raises(ValueError, match="no complete dataset role contract"):
+ scrub_dataset_bound_form_data(
+ {"viz_type": "third_party_unknown", "mystery_column": "old"}
+ )
+
+
+_NATIVE_METRICS = [
+ "saved_revenue",
+ {
+ "expressionType": "SIMPLE",
+ "column": {"column_name": "revenue"},
+ "aggregate": "SUM",
+ "label": "SUM(revenue)",
+ "hasCustomLabel": False,
+ "optionName": "metric_revenue",
+ "datasourceWarning": False,
+ "sqlExpression": None,
+ },
+ {
+ "expressionType": "SQL",
+ "column": None,
+ "aggregate": None,
+ "sqlExpression": "COUNT(*)",
+ "label": "Count",
+ "hasCustomLabel": True,
+ "optionName": "metric_count",
+ "datasourceWarning": False,
+ },
+]
+
+
+@pytest.mark.parametrize("request_type", ["generate", "update", "preview"])
+@pytest.mark.parametrize("metric", _NATIVE_METRICS, ids=["saved", "simple", "sql"])
+def test_native_xy_form_data_round_trips_through_every_request_model(
+ request_type: str, metric: object
+) -> None:
+ config = {
+ "viz_type": "echarts_timeseries_bar",
+ "x_axis": "ds",
+ "x_axis_title": "Date",
+ "x_axis_format": "smart_date",
+ "y_axis_title": "Revenue",
+ "y_axis_scale": "log",
+ "metrics": [metric],
+ "groupby": ["region"],
+ "row_limit": 123,
+ }
+ payload: dict[str, object]
+ request_model: type[
+ GenerateChartRequest | UpdateChartRequest | UpdateChartPreviewRequest
+ ]
+ if request_type == "generate":
+ request_model = GenerateChartRequest
+ payload = {"dataset_id": 7, "config": config}
+ elif request_type == "update":
+ request_model = UpdateChartRequest
+ payload = {"identifier": 19, "config": config}
+ else:
+ request_model = UpdateChartPreviewRequest
+ payload = {"dataset_id": 7, "config": config}
+
+ request = request_model.model_validate(payload)
+ xy = request.config
+ assert xy is not None
+ assert xy.x is not None
+ assert xy.x.name == "ds"
+ assert xy.x_axis is not None
+ assert xy.x_axis.title == "Date"
+ assert xy.y_axis is not None
+ assert xy.y_axis.scale == "log"
+
+ with patch(
+ "superset.mcp_service.chart.chart_utils.is_column_truly_temporal",
+ return_value=True,
+ ):
+ mapped = map_config_to_form_data(xy, dataset_id=7)
+ assert mapped["x_axis"] == "ds"
+ assert mapped["x_axis_title"] == "Date"
+ assert mapped["x_axis_format"] == "smart_date"
+ assert mapped["metrics"][0]
+
+
+def test_native_xy_object_axis_config_remains_presentation_state() -> None:
+ request = GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "viz_type": "echarts_timeseries_line",
+ "x": {"name": "ds"},
+ "x_axis": {"title": "Date", "format": "smart_date"},
+ "metrics": ["saved_revenue"],
+ },
+ }
+ )
+ assert request.config.x is not None
+ assert request.config.x.name == "ds"
+ assert request.config.x_axis is not None
+ assert request.config.x_axis.title == "Date"
+
+
+@pytest.mark.parametrize(
+ "x_axis",
+ [
+ {"title": "Date", "formt": "smart_date"},
+ {"column_name": "ds", "column_nmae": "typo"},
+ {
+ "expressionType": "SIMPLE",
+ "column": {"column_name": "ds", "column_nmae": "typo"},
+ },
+ ],
+)
+def test_native_xy_rejects_malformed_nested_axis_state(x_axis: object) -> None:
+ with pytest.raises(ValidationError, match="Unknown"):
+ GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "viz_type": "echarts_timeseries_line",
+ "x_axis": x_axis,
+ "metrics": ["saved_revenue"],
+ },
+ }
+ )
+
+
+def test_native_xy_explicit_axis_reset_survives_sparse_merge() -> None:
+ request = UpdateChartRequest.model_validate(
+ {
+ "identifier": 19,
+ "config": {
+ "viz_type": "echarts_timeseries_line",
+ "x_axis": "ds",
+ "x_axis_title": None,
+ "metrics": ["saved_revenue"],
+ },
+ }
+ )
+ assert request.config is not None
+ with patch(
+ "superset.mcp_service.chart.chart_utils.is_column_truly_temporal",
+ return_value=True,
+ ):
+ mapped = map_config_to_form_data(request.config, dataset_id=7)
+ merged = merge_form_data_for_update(
+ {
+ "viz_type": "echarts_timeseries_line",
+ "x_axis": "old_ds",
+ "metrics": ["old_metric"],
+ "x_axis_title": "Old title",
+ },
+ mapped,
+ request.config,
+ )
+ assert "x_axis_title" not in merged
+
+
+def test_mixed_typed_secondary_nulls_remain_explicit_query_clears() -> None:
+ request = GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "chart_type": "mixed_timeseries",
+ "x": {"name": "ds"},
+ "y": [{"name": "revenue", "aggregate": "SUM"}],
+ "y_secondary": [{"name": "profit", "aggregate": "SUM"}],
+ "adhoc_filters_b": None,
+ "time_range_b": None,
+ "annotation_layers_b": [],
+ },
+ }
+ )
+ with patch(
+ "superset.mcp_service.chart.chart_utils.is_column_truly_temporal",
+ return_value=True,
+ ):
+ form_data = map_config_to_form_data(request.config, dataset_id=7)
+ form_data.update(
+ {
+ "adhoc_filters": [
+ {
+ "clause": "WHERE",
+ "expressionType": "SIMPLE",
+ "subject": "region",
+ "operator": "==",
+ "comparator": "EMEA",
+ }
+ ],
+ "time_range": "Last year",
+ }
+ )
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ _, secondary = build_query_dicts_from_form_data(form_data, 7, "table")
+ assert secondary["filters"] == []
+ assert "time_range" not in secondary
+ assert secondary["annotation_layers"] == []
+
+
+def test_mixed_secondary_receives_dashboard_extra_form_data() -> None:
+ form_data = {
+ "viz_type": "mixed_timeseries",
+ "x_axis": "ds",
+ "metrics": ["revenue"],
+ "metrics_b": ["profit"],
+ }
+ extra_form_data = {"filters": [{"col": "region", "op": "IN", "val": ["EMEA"]}]}
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ primary, secondary = build_query_dicts_from_form_data(
+ form_data, 7, "table", extra_form_data=extra_form_data
+ )
+ assert primary["filters"] == secondary["filters"]
+ assert secondary["filters"] == extra_form_data["filters"]
+
+
+def test_gantt_adapter_accepts_native_adhoc_axis_objects() -> None:
+ y_axis = {
+ "label": "Task",
+ "sqlExpression": "task_name",
+ "expressionType": "SQL",
+ }
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ query = build_query_dicts_from_form_data(
+ {
+ "viz_type": "gantt_chart",
+ "start_time": "started_at",
+ "end_time": "ended_at",
+ "y_axis": y_axis,
+ "series": "team",
+ "tooltip_metrics": ["duration"],
+ "order_by_cols": ['["started_at", true]'],
+ },
+ 7,
+ "table",
+ )[0]
+ assert query["columns"] == ["started_at", "ended_at", y_axis, "team"]
+ assert query["series_columns"] == ["team"]
+
+
+def test_pivot_non_additive_metrics_preserve_grouping_sets_contract() -> None:
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ query = build_query_dicts_from_form_data(
+ {
+ "viz_type": "pivot_table_v2",
+ "groupbyRows": ["region"],
+ "groupbyColumns": ["product"],
+ "metrics": ["saved_revenue"],
+ "rowTotals": True,
+ "colTotals": True,
+ },
+ 7,
+ "table",
+ )[0]
+ assert query["grouping_sets"] == [
+ [],
+ ["product"],
+ ["region"],
+ ["region", "product"],
+ ]
+
+
+def test_big_number_raw_aggregation_preserves_two_query_contract() -> None:
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ trend, overall = build_query_dicts_from_form_data(
+ {
+ "viz_type": "big_number",
+ "metric": "saved_ratio",
+ "granularity_sqla": "ds",
+ "aggregation": "raw",
+ },
+ 7,
+ "table",
+ )
+ assert trend["columns"] == []
+ assert trend["is_timeseries"] is True
+ assert trend["post_processing"][0]["options"]["index"] == ["__timestamp"]
+ assert [rule["operation"] for rule in trend["post_processing"]] == [
+ "pivot",
+ "flatten",
+ ]
+ assert overall["columns"] == []
+ assert overall["is_timeseries"] is False
+ assert overall["post_processing"] == []
+
+
+def _sunburst_replacement_config() -> SunburstChartConfig:
+ return SunburstChartConfig(
+ hierarchy=[{"name": "region"}, {"name": "country"}],
+ metric={"name": "sales", "aggregate": "SUM"},
+ )
+
+
+def test_cached_table_rebind_drops_legacy_sql_predicates() -> None:
+ """Top-level ``where``/``having`` reference the previous dataset."""
+ cached = {
+ "viz_type": "table",
+ "datasource": "10__table",
+ "query_mode": "aggregate",
+ "groupby": ["region"],
+ "metrics": ["count"],
+ "where": "region = 'EMEA'",
+ "having": "COUNT(*) > 5",
+ "show_cell_bars": True,
+ }
+ config = TableChartConfig(columns=[{"name": "region"}])
+ new_form_data = map_config_to_form_data(config)
+
+ merged = merge_form_data_for_update(
+ cached, new_form_data, config, dataset_rebind=True
+ )
+
+ assert "where" not in merged
+ assert "having" not in merged
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ (query,) = build_query_dicts_from_form_data(merged, 99, "table")
+ assert "region = 'EMEA'" not in str(query)
+ assert "COUNT(*) > 5" not in str(query)
+
+
+def test_cross_viz_rebind_from_unregistered_viz_uses_mapped_target() -> None:
+ """A complete config replaces a source viz that has no role contract."""
+ saved = {
+ "viz_type": "word_cloud",
+ "datasource": "10__table",
+ "series": "old_word",
+ "metric": "old_count",
+ "rotation": "square",
+ "adhoc_filters": [{"subject": "old_word"}],
+ "color_scheme": "supersetColors",
+ }
+ config = _sunburst_replacement_config()
+ new_form_data = map_config_to_form_data(config)
+
+ merged = merge_form_data_for_update(
+ saved, new_form_data, config, dataset_rebind=True
+ )
+
+ assert merged["viz_type"] == "sunburst_v2"
+ assert merged["columns"] == ["region", "country"]
+ assert merged["color_scheme"] == "supersetColors"
+ assert {"series", "rotation", "adhoc_filters"}.isdisjoint(merged)
+ assert "old_word" not in str(merged)
+ assert "old_count" not in str(merged)
+
+
+def test_cross_viz_dataset_only_scrub_still_requires_role_contract() -> None:
+ """Without a replacement viz, an unknown source still fails closed."""
+ with pytest.raises(ValueError, match="no complete dataset role contract"):
+ scrub_dataset_bound_form_data({"viz_type": "word_cloud", "series": "old"})
+
+
+def test_explicit_empty_filters_clear_legacy_predicates_on_cross_viz_update() -> None:
+ """``filters=[]`` removes adhoc, legacy structured, and free-form SQL state."""
+ saved = {
+ "viz_type": "table",
+ "datasource": "10__table",
+ "query_mode": "aggregate",
+ "groupby": ["region"],
+ "metrics": ["count"],
+ "adhoc_filters": [
+ {
+ "clause": "WHERE",
+ "expressionType": "SIMPLE",
+ "subject": "region",
+ "operator": "==",
+ "comparator": "EMEA",
+ }
+ ],
+ "filters": [{"col": "region", "op": "==", "val": "EMEA"}],
+ "where": "region = 'EMEA'",
+ "having": "COUNT(*) > 5",
+ }
+ config = SunburstChartConfig(
+ hierarchy=[{"name": "region"}, {"name": "country"}],
+ metric={"name": "sales", "aggregate": "SUM"},
+ filters=[],
+ )
+ new_form_data = map_config_to_form_data(config)
+
+ merged = merge_form_data_for_update(saved, new_form_data, config)
+
+ assert {"adhoc_filters", "filters", "where", "having"}.isdisjoint(merged)
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ (query,) = build_query_dicts_from_form_data(merged, 10, "table")
+ assert query["filters"] == []
+ assert "EMEA" not in str(query)
+
+
+@pytest.mark.parametrize("chart_type", ["xy", "bar", "echarts_timeseries_bar"])
+def test_typed_xy_metrics_alias_keeps_column_semantics(chart_type: str) -> None:
+ """Only native Explore payloads read ``metrics`` strings as saved metrics."""
+ request = GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "chart_type": chart_type,
+ "x": {"name": "region"},
+ "metrics": ["revenue"],
+ },
+ }
+ )
+
+ assert request.config is not None
+ (metric,) = request.config.y
+ assert metric.name == "revenue"
+ assert metric.saved_metric is False
+ mapped = map_config_to_form_data(request.config, dataset_id=7)
+ (mapped_metric,) = mapped["metrics"]
+ assert mapped_metric["expressionType"] == "SIMPLE"
+ assert mapped_metric["aggregate"] == "SUM"
+ assert mapped_metric["column"]["column_name"] == "revenue"
+
+
+def test_native_xy_metrics_strings_remain_saved_metric_references() -> None:
+ request = GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "viz_type": "echarts_timeseries_bar",
+ "x_axis": "region",
+ "metrics": ["revenue"],
+ },
+ }
+ )
+
+ assert request.config is not None
+ (metric,) = request.config.y
+ assert metric.name == "revenue"
+ assert metric.saved_metric is True
+
+
+def test_replacement_filters_drop_legacy_predicates_on_cross_viz_update() -> None:
+ """Typed filters replace every saved predicate source, not only adhoc ones."""
+ saved = {
+ "viz_type": "table",
+ "datasource": "10__table",
+ "query_mode": "aggregate",
+ "groupby": ["region"],
+ "metrics": ["count"],
+ "filters": [{"col": "region", "op": "==", "val": "EMEA"}],
+ "extra_filters": [{"col": "region", "op": "in", "val": ["EMEA"]}],
+ "where": "region = 'EMEA'",
+ "having": "COUNT(*) > 5",
+ }
+ config = SunburstChartConfig(
+ hierarchy=[{"name": "region"}, {"name": "country"}],
+ metric={"name": "sales", "aggregate": "SUM"},
+ filters=[{"column": "region", "op": "=", "value": "APAC"}],
+ )
+ new_form_data = map_config_to_form_data(config)
+
+ merged = merge_form_data_for_update(saved, new_form_data, config)
+
+ assert {"filters", "extra_filters", "where", "having"}.isdisjoint(merged)
Review Comment:
Fixed in b6e34104724dd7a1873981c0f84d397810a813d0. Added test_replacement_filters_drop_legacy_predicates_on_same_viz_update: an aggregate Table on the same dataset replaces saved EMEA filters/extra_filters/where/having with APAC. It checks merge_form_data_for_update and the real query builder, asserting only APAC remains and neither SQL clause survives in the query or extras. The test passes with the existing cleanup and fails when where/having are removed from _SAVED_PREDICATE_FORM_DATA_KEYS, so it specifically covers the gap you identified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5513770973
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5561926353
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi merged PR #43771:
URL: https://github.com/apache/superset/pull/43771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3922270702
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -1045,6 +1063,518 @@ def reject_sql_expression_on_dimensions(self) -> "PieChartConfig":
return self
+class SunburstStandardizedControls(UnknownFieldCheckMixin):
+ """Bounded shared controls retained by Explore across viz changes."""
+
+ model_config = ConfigDict(extra="ignore")
+
+ metrics: list[JsonValue] = Field(default_factory=list)
+ columns: list[JsonValue] = Field(default_factory=list)
+
+
+class SunburstStandardizedFormData(UnknownFieldCheckMixin):
+ """Validated shape of Explore's cross-plugin UI memory."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ controls: SunburstStandardizedControls
+ memorized_form_data: list[tuple[str, dict[str, JsonValue]]] = Field(
+ default_factory=list,
+ validation_alias=AliasChoices("memorized_form_data", "memorizedFormData"),
+ serialization_alias="memorizedFormData",
+ )
+
+
+class SunburstNativeMetricColumn(UnknownFieldCheckMixin):
Review Comment:
Saved Sunburst charts with an adhoc SIMPLE metric can include normal frontend column metadata such as `certified_by`; this validator rejects it before the mapper extracts `column_name`, so generate/update cannot round-trip those charts. Could the nested `column` model tolerate the full frontend `ColumnMeta` shape while still validating the fields it consumes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3971689298
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -1115,56 +1224,22 @@ async def _query_from_form_data( # noqa: C901
error_type="NoData",
)
- columns = []
- for col_name in raw_columns:
- sample_values = [
- row.get(col_name) for row in data[:3] if row.get(col_name) is not None
- ]
- data_type = "string"
- if sample_values and all(
- isinstance(v, (int, float)) for v in sample_values
- ):
- data_type = "numeric"
- columns.append(
- DataColumn(
- name=col_name,
- display_name=col_name.replace("_", " ").title(),
- data_type=data_type,
- sample_values=sample_values[:3],
- null_count=sum(1 for row in data if row.get(col_name) is None),
- unique_count=len({str(row.get(col_name)) for row in data}),
- )
- )
+ columns = _build_data_columns(
+ data, raw_columns, query_result.get("coltypes", [])
+ )
cache_status = get_cache_status_from_result(
query_result, force_refresh=request.force_refresh
)
chart_name = form_data.get("slice_name", "Unsaved chart")
- if request.format in {"csv", "excel"}:
- from superset.models.slice import Slice
-
- # A transient chart supplies export metadata without saving anything.
- chart = Slice(id=0, slice_name=chart_name, viz_type=viz_type)
- export = (
- _export_data_as_csv
- if request.format == "csv"
- else _export_data_as_excel
- )
- return export(
- chart,
- data[: request.limit] if request.limit else data,
- raw_columns,
- cache_status,
- PerformanceMetadata(query_duration_ms=0, cache_status="fresh_query"),
- )
summary = (
f"Unsaved chart ({viz_type}). "
f"Contains {len(data)} rows across {len(raw_columns)} columns."
)
await ctx.report_progress(4, 4, "Building response")
- return ChartData(
+ response = ChartData(
Review Comment:
The unsaved `form_data_key` path now always builds `ChartData`, so `format="csv"` or `"excel"` silently returns JSON with no export payload while the saved-chart path still exports. Can this preserve the export branch here and restore coverage for unsaved exports?
##########
superset/mcp_service/app.py:
##########
@@ -395,8 +395,8 @@ def get_default_instructions(
- chart_type="table": Data table for detailed views
- chart_type="table", viz_type="ag-grid-table": Interactive AG Grid table
- chart_type="pie": Pie chart for proportional data (set donut=True for donut)
-- chart_type="gauge": Gauge/dial for one numeric metric, optionally grouped
- into up to 10 dials (native viz_type is "gauge_chart")
+- chart_type="sunburst": Hierarchical part-to-whole chart (hierarchy + metric
Review Comment:
Replacing the Gauge entry with Sunburst removes Gauge from the MCP capability catalog even though `GaugeChartPlugin` remains registered and available. Clients following these instructions will not discover a supported chart type; can the Gauge entry and display-name mapping remain alongside Sunburst?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,82 +15,1521 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
+from decimal import Decimal
+from enum import Enum
from typing import Any
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = list.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (list.__getitem__(item, index), item, index, depth + 1, False)
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ children: list[tuple[Any, dict[str, Any], str, int, bool]] = []
+ for key, child in dict.items(item):
+ if type(key) is not str:
+ return None, "a non-string object key"
+ if reason := _charge_text(key, budget, key=True, metadata=metadata):
+ return None, reason
+ children.append((child, item, key, depth + 1, False))
+ stack.append((item, None, None, depth, True))
+ stack.extend(reversed(children))
+ continue
+
+ source_item = item
+ if type(item) in enum_types:
+ normalized = _safe_enum_value(item, enum_types)
+ if normalized is None:
+ return None, "an unsupported enum"
+ item = normalized
+ elif any(base is Enum for base in _type_mro(type(item))):
+ return None, "an enum outside its expected metadata slot"
+
+ normalized, reason = _normalize_scalar(item)
+ if reason is not None:
+ return None, reason
+ max_string_bytes = (
+ MAX_QUERY_RESULT_METADATA_BYTES if metadata else MAX_RESULT_STRING_LENGTH
+ )
+ if type(normalized) is str:
+ scalar_size = _json_string_size(normalized, max_string_bytes)
+ if scalar_size is None:
+ return None, "invalid text data"
+ else:
+ scalar_size = _normalized_scalar_json_size(normalized)
+ if reason := _charge_json_bytes(
+ budget,
+ scalar_size,
+ metadata=metadata,
+ ):
+ return None, reason
+ if parent is None:
+ root = normalized
+ elif normalized is not source_item:
+ if type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized)
+ return root, None
+
+
+def _normalize_metadata_value(
+ payload: dict[str, Any], key: str, budget: _ResultBudget
+) -> str | None:
+ enum_slots: dict[str, frozenset[type[Any]]] = {
+ "status": frozenset({QueryStatus}),
+ "result_format": frozenset({ChartDataResultFormat}),
+ "coltypes": frozenset({GenericDataType}),
+ "applied_filters": frozenset({ExtraFiltersTimeColumnType}),
+ "rejected_filters": frozenset(
+ {ExtraFiltersReasonType, ExtraFiltersTimeColumnType}
+ ),
+ }
+ value = dict.__getitem__(payload, key)
+ normalized, reason = _normalize_value(
+ value,
+ budget,
+ enum_types=enum_slots.get(key, frozenset()),
+ metadata=True,
+ )
+ if reason is None and normalized is not value:
+ dict.__setitem__(payload, key, normalized)
+ return reason
+
+
+def _error_text(value: Any) -> str | None:
+ """Extract a bounded error from an already validated primitive tree."""
+ stack: list[Any] = [value]
+ parts: list[str] = []
+ used = 0
+ while stack and len(parts) < 3 and used < _MAX_ERROR_TEXT_BYTES:
+ item = stack.pop()
+ if type(item) is dict:
+ stack.extend(
+ reversed(
+ [dict.__getitem__(item, key) for key in _ERROR_KEYS if key in item]
+ )
+ )
+ continue
+ if type(item) is list:
+ stack.extend(
+ list.__getitem__(item, index)
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if item is None or item is False:
+ continue
+ if type(item) is str:
+ remaining = _MAX_ERROR_TEXT_BYTES - used - (2 if parts else 0)
+ encoded = str.encode(item, "utf-8")[:remaining]
+ text = bytes.decode(encoded, "utf-8", errors="ignore")
+ if text:
+ parts.append(text)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ return "; ".join(parts) or None
+
+
+def _failure_for_payload(payload: dict[str, Any], label: str) -> ChartError | None:
for key in ("error", "errors", "error_message"):
- if message := _query_error_text(payload.get(key)):
+ if key in payload and (message := _error_text(dict.__getitem__(payload, key))):
return ChartError(
error=f"{label} failed: {message}", error_type="QueryError"
)
- raw_status = payload.get("status")
- status = str(getattr(raw_status, "value", raw_status) or "")
+ raw_status = dict.get(payload, "status")
+ status = raw_status if type(raw_status) is str else ""
normalized_status = status.strip().casefold().replace("-", "_").replace(" ", "_")
if normalized_status in FAILED_QUERY_STATUSES:
message = (
- _query_error_text(payload.get("message"))
- or _query_error_text(payload.get("error_message"))
+ _error_text(dict.get(payload, "message"))
+ or _error_text(dict.get(payload, "error_message"))
or normalized_status
)
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
- if payload.get("success") is False:
- message = _query_error_text(payload.get("message")) or "request failed"
+ if dict.get(payload, "success") is False:
+ message = _error_text(dict.get(payload, "message")) or "request failed"
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if raw_status is None and "data" not in payload and "queries" not in payload:
+ if message := _error_text(dict.get(payload, "message")):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+ return None
+
+
+def _metadata_shape_error( # noqa: C901
+ payload: dict[str, Any], label: str, budget: _ResultBudget
+) -> ChartError | None:
+ if "success" in payload and type(dict.__getitem__(payload, "success")) is not bool:
+ return _invalid_metadata(label)
+ if "status" in payload and type(dict.__getitem__(payload, "status")) is not str:
+ return _invalid_metadata(label)
if (
- raw_status is None
- and "data" not in payload
- and "queries" not in payload
- and (message := _query_error_text(payload.get("message")))
+ "result_format" in payload
+ and dict.__getitem__(payload, "result_format") not in _RESULT_FORMAT_VALUES
):
- return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ return _invalid_metadata(label)
+ for count_key in ("rowcount", "sql_rowcount", "total_rows"):
+ if count_key in payload:
+ count = dict.__getitem__(payload, count_key)
+ if count is not None and not (
+ type(count) is int and 0 <= count <= MAX_QUERY_RESULT_ROWCOUNT
+ ):
+ return _invalid_metadata(label)
+ if "is_cached" in payload:
+ cached = dict.__getitem__(payload, "is_cached")
+ if cached is None:
+ if reason := _charge_json_bytes(
+ budget, 1, metadata=True
+ ): # ``false`` vs ``null``
+ return _invalid_result(reason)
+ dict.__setitem__(payload, "is_cached", False)
+ elif type(cached) is not bool:
+ return _invalid_metadata(label)
+ if "cache_timeout" in payload:
+ timeout = dict.__getitem__(payload, "cache_timeout")
+ if timeout is not None and not (
+ type(timeout) is int
+ and (
+ timeout == CACHE_DISABLED_TIMEOUT
+ or 0 <= timeout <= MAX_QUERY_RESULT_CACHE_TIMEOUT
+ )
+ ):
+ return _invalid_metadata(label)
+ if "cache_key" in payload:
+ cache_key = dict.__getitem__(payload, "cache_key")
+ if cache_key is not None and (type(cache_key) is not str or not cache_key):
+ return _invalid_metadata(label)
+ for timestamp_key in ("cached_dttm", "cache_dttm", "queried_dttm"):
+ if timestamp_key not in payload:
+ continue
+ timestamp = dict.__getitem__(payload, timestamp_key)
+ if timestamp is None:
+ continue
+ if (
+ type(timestamp) is not str
+ or not timestamp
+ or len(timestamp) > MAX_QUERY_RESULT_TIMESTAMP_LENGTH
+ ):
+ return _invalid_metadata(label)
+ normalized = f"{timestamp[:-1]}+00:00" if timestamp.endswith("Z") else timestamp
+ try:
+ parsed = datetime.fromisoformat(normalized)
+ except ValueError:
+ return _invalid_metadata(label)
+ if parsed.tzinfo is None or parsed.utcoffset() != timedelta(0):
Review Comment:
A normal cache hit supplies `cached_dttm` as Superset's naive UTC timestamp, but this now rejects every naive timestamp as hostile. That makes the MCP data and preview paths fail on their second, warm-cache request; can this normalize the cache timestamp as UTC instead of rejecting it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3971732087
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1249,6 +1336,659 @@ def map_gauge_config(config: GaugeChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def preserve_previous_adhoc_filters(
+ new_form_data: Dict[str, Any], previous_form_data: Mapping[str, Any]
+) -> None:
+ """Compatibility entry point backed by the shared filter merge."""
+ filters = _merge_preserved_adhoc_filters(
+ previous_form_data,
+ new_form_data,
+ drop_existing_temporal=False,
+ )
+ if filters is not None:
+ new_form_data["adhoc_filters"] = filters
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update( # noqa: C901
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+ """
+ if dataset_rebind:
+ existing_form_data = scrub_dataset_bound_form_data(existing_form_data)
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ if new_form_data.get("viz_type") == "mixed_timeseries" and not dataset_rebind:
+ from superset.common.form_data_query_context import (
+ MIXED_TIMESERIES_SECONDARY_QUERY_KEYS,
+ )
+
+ # Query B inherits unsuffixed controls only when the suffixed key is
+ # absent. Preserve explicit native clears for controls the typed
+ # mapper did not replace, so []/None never turns into accidental
+ # inheritance from query A. Valid comparison state is also retained;
+ # malformed/stale dataset roles remain fail-closed and are dropped.
+ for key in MIXED_TIMESERIES_SECONDARY_QUERY_KEYS:
+ if key in new_form_data or key not in existing_form_data:
+ continue
+ value = existing_form_data[key]
+ is_explicit_clear = value is None or value in ([], {}, "")
+ is_valid_comparison = key == "comparison_type_b" and value in {
+ "values",
+ "difference",
+ "percentage",
+ "ratio",
+ }
+ is_valid_list_state = key in {
+ "adhoc_filters_b",
+ "annotation_layers_b",
+ "time_compare_b",
+ } and isinstance(value, list)
+ if is_explicit_clear or is_valid_comparison or is_valid_list_state:
+ merged[key] = value
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ if getattr(config, "filters", None) == []:
+ merged.pop("adhoc_filters", None)
+ elif getattr(config, "filters", None) is None:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool(
+ {"temporal_column", "time_grain", "time_range"} & fields_set
+ ),
+ )
+ if filters is not None:
+ merged["adhoc_filters"] = filters
+
+ if not isinstance(config, SunburstChartConfig):
+ return merged
+
+ temporal_fields = {"time_grain", "temporal_column"}
+ for field_name, form_key in _SUNBURST_UPDATE_FIELD_KEYS.items():
+ if field_name in temporal_fields:
+ continue
+ if field_name not in fields_set:
+ if same_viz:
+ if form_key in existing_form_data:
+ merged[form_key] = existing_form_data[form_key]
+ else:
+ merged.pop(form_key, None)
+ continue
+
+ value = getattr(config, field_name)
+ if value is None:
+ merged.pop(form_key, None)
+ if field_name == "time_range":
+ # The generic query-context mapper reconstructs time_range from
+ # these legacy keys. A clear must remove all three sources.
+ merged.pop("since", None)
+ merged.pop("until", None)
+
+ # A null temporal control is an atomic clear. Apply it after every merge so
+ # cached/native aliases and extra-form-data overrides cannot recreate time
+ # state in QueryContextFactory.
+ explicit_temporal_clear = any(
+ field_name in fields_set and getattr(config, field_name) is None
+ for field_name in ("temporal_column", "time_grain", "time_range")
+ )
+ if explicit_temporal_clear:
+ merged = _scrub_temporal_form_data(merged)
Review Comment:
This scrubs a newly supplied `time_range` whenever the same partial update explicitly clears `time_grain`, and only restores column/grain afterward. An update that sets `time_range="Last month"` while clearing a saved grain succeeds with no range; can this reapply an explicitly supplied range after the atomic clear and cover that mixed update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5628583845
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5637725320
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5939670733
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5768935853
This one's huge, 77 files. Per your own landing-order note, neither #44218 nor #44219 has merged yet, so we'd be pulling in duplicated code that's meant to come from #44219 later. I'd rather hold until those two land and let this rebase down before diving into a full review. Ping me once they're in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4226289921
##########
superset/mcp_service/chart/tool/get_chart_sql.py:
##########
@@ -377,35 +450,74 @@ def _extract_sql_from_result(
errors: list[str] = []
language = "sql"
- for idx, query_result in enumerate(queries, start=1):
- if not isinstance(query_result, dict):
- continue
- if "language" in query_result and isinstance(query_result["language"], str):
- language = query_result["language"]
- query_sql = query_result.get("query", "")
- query_error = query_result.get("error")
+ source_bytes = 0
+ for offset in range(query_count):
+ idx = offset + 1
+ query_result = list.__getitem__(queries, offset)
+ if type(query_result) is not dict:
+ return ChartError(
+ error=f"Malformed chart SQL result: query {idx} must be an object.",
+ error_type="MalformedQueryResult",
+ )
+ query_language = dict.get(query_result, "language")
+ if query_language is not None:
+ if (
+ type(query_language) is not str
+ or _bounded_utf8_length(query_language, 256) is None
+ ):
+ return ChartError(
+ error=(
+ f"Malformed chart SQL result: query {idx} language must be "
+ "a bounded exact string."
+ ),
+ error_type="MalformedQueryResult",
+ )
+ language = query_language
+ query_sql = dict.get(query_result, "query", "")
+ query_error = dict.get(query_result, "error")
+ for label, value in (("query", query_sql), ("error", query_error)):
+ if value is None:
+ continue
+ if type(value) is not str:
Review Comment:
Fixed in 72ddec085278cc42c8af8e81a5da6de0d0832254. SQL extraction resolves concrete Flask-Babel LazyString errors before the exact-string check, preserving the original QueryGenerationFailed message. The resolved text still goes through the aggregate byte budget; arbitrary conversion hooks remain rejected. Regression tests cover error-only and SQL-plus-error results, oversized translations, and failed translation resolution.
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1628 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+import base64
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
+from importlib import import_module
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.utils.serialization import BINARY_PREFIX
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+def _postgres_range_types() -> tuple[type[Any], ...]:
+ """Discover concrete range scalars only for installed PostgreSQL drivers."""
+ range_types: list[type[Any]] = []
+ for module_name, names in (
+ (
+ "psycopg2.extras",
+ ("Range", "NumericRange", "DateRange", "DateTimeRange", "DateTimeTZRange"),
+ ),
+ (
+ "psycopg.types.range",
+ (
+ "Range",
+ "Int4Range",
+ "Int8Range",
+ "NumericRange",
+ "DateRange",
+ "TimestampRange",
+ "TimestamptzRange",
+ ),
+ ),
+ ):
+ try:
+ module = import_module(module_name)
+ except ImportError:
+ continue
+ range_types.extend(getattr(module, name) for name in names)
+ return tuple(range_types)
+
+
+_POSTGRES_RANGE_TYPES = _postgres_range_types()
+
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _canonical_binary(
+ value: bytes | bytearray | memoryview,
+) -> tuple[str | None, str | None]:
+ """Render exact binary cells as UTF-8 text or ``base64:``-prefixed text.
+
+ Matches the MCP response serializer: valid UTF-8 is returned as text and
+ anything else is losslessly base64-encoded. The raw size is bounded before
+ copying so the rendered string stays within the per-cell text budget.
+ """
+ size = value.nbytes if type(value) is memoryview else len(value)
+ if size > MAX_RESULT_STRING_LENGTH:
+ return None, "an oversized binary value"
+ try:
+ raw = bytes(value)
+ except (BufferError, TypeError, ValueError):
+ return None, "an invalid binary value"
+ try:
+ text = raw.decode("utf-8")
+ except UnicodeDecodeError:
+ text = BINARY_PREFIX + base64.b64encode(raw).decode("ascii")
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized binary value"
+ return text, None
+
+
+def _canonical_postgres_range(value: Any) -> tuple[str | None, str | None]:
+ """Render exact driver ranges after bounding and validating their endpoints.
+
+ Never invoke a range's string hook: even a concrete range can contain a
+ hostile endpoint. Only exact numeric and temporal bounds are supported.
+ """
+ try:
+ bounds = object.__getattribute__(value, "_bounds")
+ lower = object.__getattribute__(value, "_lower")
+ upper = object.__getattribute__(value, "_upper")
+ except AttributeError:
+ return None, "an invalid PostgreSQL range"
+ if bounds is None or (type(bounds) is str and bounds == ""):
+ return "empty", None
+ if type(bounds) is not str or bounds not in {"[)", "(]", "()", "[]"}:
+ return None, "an invalid PostgreSQL range"
+ endpoints: list[str] = []
+ for endpoint in (lower, upper):
+ if endpoint is None:
+ endpoints.append("None")
+ continue
+ if not any(
+ type(endpoint) is trusted
+ for trusted in (int, float, Decimal, date, datetime)
+ ):
+ return None, "an unsupported PostgreSQL range endpoint"
+ normalized, reason = _normalize_scalar(endpoint)
+ if reason or normalized is None:
+ return None, reason or "an invalid PostgreSQL range endpoint"
+ # Canonical temporal text has already validated timezone state; ranges
+ # use the driver's space separator instead of datetime's ISO 'T'.
+ text = (
+ normalized.replace("T", " ", 1)
+ if type(endpoint) is datetime
+ else str(normalized)
+ )
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized PostgreSQL range"
+ endpoints.append(text)
+ text = f"{bounds[0]}{endpoints[0]}, {endpoints[1]}{bounds[1]}"
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized PostgreSQL range"
+ return text, None
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ return (value, None) if math.isfinite(value) else (None, None)
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+ if value_type is bytes or value_type is bytearray or value_type is memoryview:
+ return _canonical_binary(value)
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, None)
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ if any(value_type is trusted for trusted in _POSTGRES_RANGE_TYPES):
+ return _canonical_postgres_range(value)
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list or type(item) is tuple or type(item) is np.ndarray:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ if type(item) is np.ndarray:
+ if item.ndim == 0:
+ return None, "a non-sequence array"
+ width = item.shape[0]
+ elif type(item) is tuple:
+ width = tuple.__len__(item)
+ else:
+ width = list.__len__(cast(list[Any], item))
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ # Convert only exact producer sequences, after checking their size.
+ # Traverse children through the same depth/node/byte budgets, and
+ # track the original container so object-array cycles stay bounded.
+ if type(item) is tuple:
+ normalized_array = [
+ tuple.__getitem__(item, index) for index in range(width)
+ ]
+ elif type(item) is np.ndarray:
+ normalized_array = [
+ np.ndarray.__getitem__(item, index) for index in range(width)
+ ]
+ else:
+ normalized_array = cast(list[Any], item)
+ if normalized_array is not item:
+ if parent is None:
+ root = normalized_array
+ elif type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized_array)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized_array)
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (
+ list.__getitem__(normalized_array, index),
+ normalized_array,
+ index,
+ depth + 1,
+ False,
+ )
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ normalized_object: dict[str, Any] = {}
+ for key, child in dict.items(item):
+ # Only exact built-in keys may be stringified: arbitrary objects
+ # and subclasses can run conversion hooks. Bound integers before
+ # rendering and charge each rendered key against the text budget.
+ if metadata and type(key) is not str:
+ return None, "a non-string metadata key"
+ if type(key) is int:
+ if reason := _integer_failure(key):
+ return None, reason
+ elif type(key) not in {str, bool, float, type(None)}:
+ return None, "an unsupported object key"
+ normalized_key = key if type(key) is str else str(key)
+ if reason := _charge_text(
+ normalized_key, budget, key=True, metadata=metadata
+ ):
+ return None, reason
+ normalized_object[normalized_key] = child
+ if parent is None:
+ root = normalized_object
+ elif type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized_object)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized_object)
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (child, normalized_object, key, depth + 1, False)
+ for key, child in reversed(list(dict.items(normalized_object)))
+ )
+ continue
+
+ source_item = item
+ if type(item) in enum_types:
+ normalized = _safe_enum_value(item, enum_types)
+ if normalized is None:
+ return None, "an unsupported enum"
+ item = normalized
+ elif any(base is Enum for base in _type_mro(type(item))):
+ return None, "an enum outside its expected metadata slot"
+
+ # Metadata retains its strict wire contract (for example a rowcount
+ # cannot become null merely because the producer supplied infinity).
+ if metadata and (type(item) is float or type(item) in _NUMPY_FLOAT_TYPES):
+ number = float(item)
+ if not math.isfinite(number) and not math.isnan(number):
+ return None, "a non-finite metadata number"
+ normalized, reason = _normalize_scalar(item)
+ if reason is not None:
+ return None, reason
+ max_string_bytes = (
+ MAX_QUERY_RESULT_METADATA_BYTES if metadata else MAX_RESULT_STRING_LENGTH
+ )
+ if type(normalized) is str:
+ scalar_size = _json_string_size(normalized, max_string_bytes)
+ if scalar_size is None:
+ return None, "invalid text data"
+ else:
+ scalar_size = _normalized_scalar_json_size(normalized)
+ if reason := _charge_json_bytes(
+ budget,
+ scalar_size,
+ metadata=metadata,
+ ):
+ return None, reason
+ if parent is None:
+ root = normalized
+ elif normalized is not source_item:
+ if type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized)
+ return root, None
+
+
+def _normalize_metadata_value(
+ payload: dict[str, Any], key: str, budget: _ResultBudget
+) -> str | None:
+ enum_slots: dict[str, frozenset[type[Any]]] = {
+ "status": frozenset({QueryStatus}),
+ "result_format": frozenset({ChartDataResultFormat}),
+ "coltypes": frozenset({GenericDataType}),
+ "applied_filters": frozenset({ExtraFiltersTimeColumnType}),
+ "rejected_filters": frozenset(
+ {ExtraFiltersReasonType, ExtraFiltersTimeColumnType}
+ ),
+ }
+ value = dict.__getitem__(payload, key)
+ normalized, reason = _normalize_value(
+ value,
+ budget,
+ enum_types=enum_slots.get(key, frozenset()),
+ metadata=True,
+ )
+ if reason is None and normalized is not value:
+ dict.__setitem__(payload, key, normalized)
+ return reason
+
+
+def _error_text(value: Any) -> str | None:
+ """Extract a bounded error from an already validated primitive tree."""
+ stack: list[Any] = [value]
+ parts: list[str] = []
+ used = 0
+ while stack and len(parts) < 3 and used < _MAX_ERROR_TEXT_BYTES:
+ item = stack.pop()
+ if type(item) is dict:
+ stack.extend(
+ reversed(
+ [dict.__getitem__(item, key) for key in _ERROR_KEYS if key in item]
+ )
+ )
+ continue
+ if type(item) is list:
+ stack.extend(
+ list.__getitem__(item, index)
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if item is None or item is False:
+ continue
+ if type(item) is str:
+ remaining = _MAX_ERROR_TEXT_BYTES - used - (2 if parts else 0)
+ encoded = str.encode(item, "utf-8")[:remaining]
+ text = bytes.decode(encoded, "utf-8", errors="ignore")
+ if text:
+ parts.append(text)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ return "; ".join(parts) or None
+
+
+def _failure_for_payload(payload: dict[str, Any], label: str) -> ChartError | None:
for key in ("error", "errors", "error_message"):
- if message := _query_error_text(payload.get(key)):
+ if key in payload and (message := _error_text(dict.__getitem__(payload, key))):
return ChartError(
error=f"{label} failed: {message}", error_type="QueryError"
)
- raw_status = payload.get("status")
- status = str(getattr(raw_status, "value", raw_status) or "")
+ raw_status = dict.get(payload, "status")
+ status = raw_status if type(raw_status) is str else ""
normalized_status = status.strip().casefold().replace("-", "_").replace(" ", "_")
if normalized_status in FAILED_QUERY_STATUSES:
message = (
- _query_error_text(payload.get("message"))
- or _query_error_text(payload.get("error_message"))
+ _error_text(dict.get(payload, "message"))
+ or _error_text(dict.get(payload, "error_message"))
or normalized_status
)
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
- if payload.get("success") is False:
- message = _query_error_text(payload.get("message")) or "request failed"
+ if dict.get(payload, "success") is False:
+ message = _error_text(dict.get(payload, "message")) or "request failed"
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if raw_status is None and "data" not in payload and "queries" not in payload:
+ if message := _error_text(dict.get(payload, "message")):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+ return None
+
+
+def _metadata_shape_error( # noqa: C901
+ payload: dict[str, Any], label: str, budget: _ResultBudget
+) -> ChartError | None:
+ if "success" in payload and type(dict.__getitem__(payload, "success")) is not bool:
+ return _invalid_metadata(label)
+ if "status" in payload and type(dict.__getitem__(payload, "status")) is not str:
+ return _invalid_metadata(label)
if (
- raw_status is None
- and "data" not in payload
- and "queries" not in payload
- and (message := _query_error_text(payload.get("message")))
+ "result_format" in payload
+ and dict.__getitem__(payload, "result_format") not in _RESULT_FORMAT_VALUES
):
- return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ return _invalid_metadata(label)
+ for count_key in ("rowcount", "sql_rowcount", "total_rows"):
+ if count_key in payload:
+ count = dict.__getitem__(payload, count_key)
+ if count is not None and not (
+ type(count) is int and 0 <= count <= MAX_QUERY_RESULT_ROWCOUNT
+ ):
+ return _invalid_metadata(label)
+ if "is_cached" in payload:
+ cached = dict.__getitem__(payload, "is_cached")
+ if cached is None:
+ if reason := _charge_json_bytes(
+ budget, 1, metadata=True
+ ): # ``false`` vs ``null``
+ return _invalid_result(reason)
+ dict.__setitem__(payload, "is_cached", False)
+ elif type(cached) is not bool:
+ return _invalid_metadata(label)
+ if "cache_timeout" in payload:
+ timeout = dict.__getitem__(payload, "cache_timeout")
+ if timeout is not None and not (
+ type(timeout) is int
+ and (
+ timeout == CACHE_DISABLED_TIMEOUT
+ or 0 <= timeout <= MAX_QUERY_RESULT_CACHE_TIMEOUT
+ )
+ ):
+ return _invalid_metadata(label)
+ if "cache_key" in payload:
+ cache_key = dict.__getitem__(payload, "cache_key")
+ if cache_key is not None and (type(cache_key) is not str or not cache_key):
+ return _invalid_metadata(label)
+ for timestamp_key in ("cached_dttm", "cache_dttm", "queried_dttm"):
+ if timestamp_key not in payload:
+ continue
+ timestamp = dict.__getitem__(payload, timestamp_key)
+ if timestamp is None:
+ continue
+ if (
+ type(timestamp) is not str
+ or not timestamp
+ or len(timestamp) > MAX_QUERY_RESULT_TIMESTAMP_LENGTH
+ ):
+ return _invalid_metadata(label)
+ normalized = f"{timestamp[:-1]}+00:00" if timestamp.endswith("Z") else timestamp
+ try:
+ parsed = datetime.fromisoformat(normalized)
+ except ValueError:
+ return _invalid_metadata(label)
+ if parsed.tzinfo is None:
+ parsed = parsed.replace(tzinfo=timezone.utc)
+ if parsed.utcoffset() != timedelta(0):
+ return _invalid_metadata(label)
+ canonical_timestamp = parsed.astimezone(timezone.utc).isoformat()
+ original_size = _json_string_size(timestamp, MAX_RESULT_STRING_LENGTH)
+ canonical_size = _json_string_size(
+ canonical_timestamp, MAX_RESULT_STRING_LENGTH
+ )
+ assert original_size is not None
+ assert canonical_size is not None
+ if canonical_size > original_size and (
+ reason := _charge_json_bytes(
+ budget, canonical_size - original_size, metadata=True
+ )
+ ):
+ return _invalid_result(reason)
+ dict.__setitem__(payload, timestamp_key, canonical_timestamp)
+ return None
+
+
+def _filter_metadata_is_valid(value: Any, *, rejected: bool) -> bool:
+ if type(value) is not list or list.__len__(value) > MAX_RESULT_VALUE_ITEMS:
+ return False
+ expected = {"column", "reason"} if rejected else {"column"}
+ for entry in value:
+ if type(entry) is not dict or set(dict.keys(entry)) != expected:
+ return False
+ if type(dict.__getitem__(entry, "column")) is not str:
+ return False
+ if rejected and type(dict.__getitem__(entry, "reason")) is not str:
+ return False
+ return True
+
+
+def _validate_query_metadata(query: dict[str, Any], index: int) -> ChartError | None:
+ label = f"Chart query {index}"
+ colnames_present = "colnames" in query
+ coltypes_present = "coltypes" in query
+ colnames = dict.get(query, "colnames", [])
+ coltypes = dict.get(query, "coltypes", [])
+ if (
+ type(colnames) is not list
+ or list.__len__(colnames) > MAX_QUERY_RESULT_COLUMNS
+ or any(type(column) is not str or not column for column in colnames)
+ ):
+ return _invalid_result(f"malformed column metadata for query {index}")
+ if (
+ type(coltypes) is not list
+ or list.__len__(coltypes) > MAX_QUERY_RESULT_COLUMNS
+ or any(
+ type(value) is not int or value not in _COLTYPE_VALUES for value in coltypes
+ )
+ ):
+ return _invalid_result(f"malformed column type metadata for query {index}")
+ if (colnames_present or coltypes_present) and (
+ not colnames_present
+ or not coltypes_present
+ or list.__len__(colnames) != list.__len__(coltypes)
+ ):
+ return _invalid_result(f"misaligned column metadata for query {index}")
+ if "applied_filters" in query and not _filter_metadata_is_valid(
+ dict.__getitem__(query, "applied_filters"), rejected=False
+ ):
+ return _invalid_metadata(label)
+ if "rejected_filters" in query and not _filter_metadata_is_valid(
+ dict.__getitem__(query, "rejected_filters"), rejected=True
+ ):
+ return _invalid_metadata(label)
+ if "rejected_filter_columns" in query:
+ rejected_columns = dict.__getitem__(query, "rejected_filter_columns")
+ if type(rejected_columns) is not list or any(
Review Comment:
Verified against the live producer path: the adhoc dict is appended in models/helpers.py, but it does not reach this validator. FULL results remove the raw rejection list in _materialize_full_payload and rebuild it through _filter_status, which applies get_column_name to each entry. QUERY results use the same helper. get_column_name extracts the adhoc label or sqlExpression, so rejected_filter_columns is a string list at the MCP boundary.
Evidence: [FULL materialization](https://github.com/apache/superset/blob/72ddec085278cc42c8af8e81a5da6de0d0832254/superset/common/query_actions.py#L242-L253), [shared normalization](https://github.com/apache/superset/blob/72ddec085278cc42c8af8e81a5da6de0d0832254/superset/common/query_actions.py#L81-L113), and [adhoc name extraction](https://github.com/apache/superset/blob/72ddec085278cc42c8af8e81a5da6de0d0832254/superset/utils/core.py#L1403-L1423).
No production change is needed. Commit 72ddec085278cc42c8af8e81a5da6de0d0832254 adds test_adhoc_rejected_filters_are_normalized_before_mcp_validation, exercising the real FULL/QUERY producers and MCP data/SQL consumers with labeled and unlabeled rejected adhoc columns. Rows and SQL are returned successfully, with string-valued rejected-filter metadata. The strict wire-shape check is retained.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1784,804 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "BubbleChartConfig": {
+ "series": (("series",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "series_limit_metric": (("series_limit_metric",), ("timeseries_limit_metric",)),
+ "timeseries_limit_metric": (
+ ("series_limit_metric",),
+ ("timeseries_limit_metric",),
+ ),
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "groupby": (("group_by",),),
+ "groupby_b": (("group_by_secondary",),),
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
Review Comment:
Fixed in 72ddec085278cc42c8af8e81a5da6de0d0832254. The Table plugin restores omitted native percent_metrics for same-visualization aggregate updates (both table and ag-grid-table). Raw-mode updates, visualization changes, and dataset rebinds still drop those saved roles. Regression tests verify the contribution operator and preservation across cached, immediate-save, and preview update products.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1784,804 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "BubbleChartConfig": {
+ "series": (("series",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "series_limit_metric": (("series_limit_metric",), ("timeseries_limit_metric",)),
+ "timeseries_limit_metric": (
+ ("series_limit_metric",),
+ ("timeseries_limit_metric",),
+ ),
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "groupby": (("group_by",),),
+ "groupby_b": (("group_by_secondary",),),
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ requested_filters = getattr(config, "filters", None)
+ if requested_filters is not None:
+ # Legacy ``filters``/``where``/``having`` are reconstructed into adhoc
+ # filters at query time, so an explicit clear or replacement removes
+ # every saved source the new form data does not set itself.
+ for key in _SAVED_PREDICATE_FORM_DATA_KEYS - new_form_data.keys():
+ merged.pop(key, None)
+ else:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool({"temporal_column", "time_range"} & fields_set),
+ )
+ if filters is not None:
+ merged["adhoc_filters"] = filters
+ return merged
+
+
+def retain_mixed_timeseries_secondary_update_state(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Return saved query-B controls a same-viz Mixed Timeseries update keeps.
+
+ Query B inherits unsuffixed controls only when the suffixed key is absent.
+ Preserve explicit native clears for controls the typed mapper did not
+ replace, so []/None never turns into accidental inheritance from query A.
+ Valid modeled secondary controls and comparison state are also retained;
+ malformed controls and stale replacement roles remain fail-closed.
+ """
+ from superset.common.form_data_query_context import (
+ MIXED_TIMESERIES_SECONDARY_QUERY_KEYS,
+ )
+
+ retained: Dict[str, Any] = {}
+ for key in MIXED_TIMESERIES_SECONDARY_QUERY_KEYS:
+ if key in new_form_data or key not in existing_form_data:
+ continue
+ value = existing_form_data[key]
+ is_explicit_clear = value is None or value in ([], {}, "")
+ is_valid_comparison = key == "comparison_type_b" and value in {
+ "values",
+ "difference",
+ "percentage",
+ "ratio",
+ }
+ is_valid_list_state = key in {
+ "adhoc_filters_b",
+ "annotation_layers_b",
+ "time_compare_b",
+ } and isinstance(value, list)
+ is_valid_modeled_control = False
+ if field := MixedTimeseriesChartConfig.model_fields.get(
+ {"groupby_b": "group_by_secondary"}.get(key, key)
+ ):
+ try:
+ TypeAdapter(field.rebuild_annotation()).validate_python(value)
Review Comment:
Fixed in 72ddec085278cc42c8af8e81a5da6de0d0832254. Retained series_limit_metric_b and timeseries_limit_metric_b are validated using the native metric coercion used for XY ranking, while the original saved string or adhoc object is preserved in form data. Regression tests cover both controls with saved metric names, SIMPLE metrics, and SQL metrics, and assert query B retains the ranking metric and series limit during a show_legend-only update.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +427,1295 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
Review Comment:
Fixed in 72ddec085278cc42c8af8e81a5da6de0d0832254. The typed Mixed Timeseries mapper emits groupby_b: [] by default, matching Explore's independent secondary grouping default without changing native unsuffixed-control inheritance. Regression tests rebuild both queries with omitted, empty, and explicit secondary grouping and verify primary region grouping does not leak into B. Same-viz update omission still preserves saved grouping.
##########
docs/admin_docs/configuration/mcp-server.mdx:
##########
@@ -1447,6 +1454,10 @@ For Generic Chart Axes charts, both MCP overrides and dashboard time-grain
filters update the `BASE_AXIS` column's `timeGrain`, regardless of its position
in the query's `columns` list. Other columns are left untouched. Queries without
a `BASE_AXIS` column continue to use `extras.time_grain_sqla`.
+Same-visualization updates that supply only `time_grain` preserve saved temporal
Review Comment:
Corrected in 72ddec085278cc42c8af8e81a5da6de0d0832254. The documentation says that a non-null time_grain-only update preserves saved temporal range filters, while explicitly setting time_grain to null clears saved temporal state, including time_range and temporal range filters. This matches the existing atomic-clear behavior in the overlay and Sunburst finalizer; no code behavior was changed for this thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-6054774056
@michael-s-molina @msyavuz this is approved by @sadpandajoe and waiting on a code owner stamp. Could one of you take a look when you get a chance? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5531939527
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5927438764
Fixed in 0dced25899: Histogram and Gantt retain temporal granularity, restoring date WHERE predicates and respecting overrides/clears.
Added SQL regressions; 3,725 chart unit tests passed (3 skipped), and pre-commit passed. The same bug reproduces on master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5516190355
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5595567962
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4203639829
##########
superset/mcp_service/chart/preview_utils.py:
##########
@@ -1543,3 +1764,52 @@ def _generate_vega_lite_preview_from_data( # noqa: C901
data_url=None,
supports_streaming=False,
)
+
+
+def generate_xy_vega_lite_preview(
Review Comment:
The shared query builder now pivots grouped timeseries into flattened wide columns (`revenue, East`, `revenue, West`), and this adapter folds them for XY. `MixedTimeseriesChartPlugin` has no `vega_lite_preview` hook, so a Mixed Timeseries Vega-Lite preview with `group_by=["region"]` goes through the generic renderer, which plots only the first numeric column and has no `region` field left to color by. The preview reports success while showing one region. Before this PR Mixed returned long-form rows and kept the color encoding. Should Mixed get the same wide-result adapter, or reject this preview?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,792 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
Review Comment:
This strips every `query_role_keys` entry from the saved form data before overlaying the mapper output, but only XY has modeled controls that restore omitted optional roles. For a saved Mixed Timeseries chart with `groupby=["region"]` and `groupby_b=["product"]`, an update that resends `x`/`y`/`y_secondary` plus a presentation change and omits grouping saves successfully with both groupings gone, because `MixedTimeseriesChartConfig` has no `group_by` entry in `_MODELED_UPDATE_CONTROL_PATHS` (and `retain_mixed_timeseries_secondary_update_state` looks up `groupby_b` in `model_fields`, where the field is `group_by_secondary`). The same strip drops `timeseries_limit_metric`/`series_limit_metric` on XY: a line chart with `series_limit=1` ranked by `profit` keeps the limit but falls back to ranking by the first metric after an unrelated update. The base shallow merge kept these. Should omitted Mixed grouping and the XY series-limit metric be restored the way XY `group_by` is, keeping ex
plicit `[]`/`None` as clears?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +427,1311 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ extra_metric = sort_metric
+ else:
+ extra_metric = None
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ sort_metric=extra_metric,
+ )
+ if extra_metric is not None:
+ query.setdefault("metrics", []).append(extra_metric)
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = (
+ _label(explicit_x_axis)
+ if explicit_x_axis
+ else _x_axis_label(form_data, frontend_strict=True)
+ )
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
Review Comment:
A saved Big Number whose `x_axis` is a `{"column_name": "ds"}` mapping (with `granularity_sqla: "ds"`, `time_grain_sqla: "P1M"`, and a `time_range`) selects plain `ds`, then this drops both `granularity` and the grain. `normalize_time_column` only rewrites axes accepted by `_frontend_x_axis_column`, so nothing re-buckets or re-binds the time range. The fallback data/preview query groups by raw timestamps and ignores the requested range, so two January points that should sum to one monthly value stay separate and out-of-range rows can appear. The base builder kept both fields for this payload. Should the mapping axis keep `granularity`/grain (or be converted to the buckets `normalize_time_column` applies) instead of popping them?
##########
UPDATING.md:
##########
@@ -24,6 +24,10 @@ assists people when migrating to a new version.
## Next
+- MCP `update_chart` requires a complete `config` when changing `dataset_id`
Review Comment:
The new fixed result caps in `query_result.py` are enforced unconditionally (`get_chart_data.py` calls the validator before any export), but this upgrade note only mentions dataset rebinding. With `SQL_MAX_ROW >= 60000` and the response guard off, a saved Table that exported 60,000 rows via `get_chart_data(format="csv")` now returns `InvalidQueryResult`, and raising or disabling `MCP_RESPONSE_SIZE_CONFIG` cannot restore it. Should this entry also call out the mandatory limits so operators know they must lower row limits or aggregate?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1523 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+import base64
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.utils.serialization import BINARY_PREFIX
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _canonical_binary(
+ value: bytes | bytearray | memoryview,
+) -> tuple[str | None, str | None]:
+ """Render exact binary cells as UTF-8 text or ``base64:``-prefixed text.
+
+ Matches the MCP response serializer: valid UTF-8 is returned as text and
+ anything else is losslessly base64-encoded. The raw size is bounded before
+ copying so the rendered string stays within the per-cell text budget.
+ """
+ size = value.nbytes if type(value) is memoryview else len(value)
+ if size > MAX_RESULT_STRING_LENGTH:
+ return None, "an oversized binary value"
+ try:
+ raw = bytes(value)
+ except (BufferError, TypeError, ValueError):
+ return None, "an invalid binary value"
+ try:
+ text = raw.decode("utf-8")
+ except UnicodeDecodeError:
+ text = BINARY_PREFIX + base64.b64encode(raw).decode("ascii")
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized binary value"
+ return text, None
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+ if value_type is bytes or value_type is bytearray or value_type is memoryview:
+ return _canonical_binary(value)
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list or type(item) is tuple or type(item) is np.ndarray:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ if type(item) is np.ndarray:
+ if item.ndim == 0:
+ return None, "a non-sequence array"
+ width = item.shape[0]
+ elif type(item) is tuple:
+ width = tuple.__len__(item)
+ else:
+ width = list.__len__(cast(list[Any], item))
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ # Convert only exact producer sequences, after checking their size.
+ # Traverse children through the same depth/node/byte budgets, and
+ # track the original container so object-array cycles stay bounded.
+ if type(item) is tuple:
+ normalized_array = [
+ tuple.__getitem__(item, index) for index in range(width)
+ ]
+ elif type(item) is np.ndarray:
+ normalized_array = [
+ np.ndarray.__getitem__(item, index) for index in range(width)
+ ]
+ else:
+ normalized_array = cast(list[Any], item)
+ if normalized_array is not item:
+ if parent is None:
+ root = normalized_array
+ elif type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized_array)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized_array)
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (
+ list.__getitem__(normalized_array, index),
+ normalized_array,
+ index,
+ depth + 1,
+ False,
+ )
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ children: list[tuple[Any, dict[str, Any], str, int, bool]] = []
+ for key, child in dict.items(item):
+ if type(key) is not str:
Review Comment:
This validator rejects any non-`str` mapping key, and `_normalize_scalar` rejects a non-finite float, instead of coercing them the way the previous serializer did. A raw Table over ClickHouse `Map(UInt64, String)` (row `{"payload": {1: "a"}}`) used to return `{"payload": {"1": "a"}}` and a `Array(Float64)` cell `[1.0, inf]` used to return `[1.0, null]` (still what the serialization docs promise); both now fail the whole `get_chart_data`/`query_dataset`/`get_table` call with `InvalidQueryResult`. `df_to_records` only null-normalizes top-level cells, so nested values reach this check unchanged. Should these be bounded-normalized (stringify built-in key types, null non-finite floats) instead of failing the entire result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4204241814
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +427,1311 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ extra_metric = sort_metric
+ else:
+ extra_metric = None
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ sort_metric=extra_metric,
+ )
+ if extra_metric is not None:
+ query.setdefault("metrics", []).append(extra_metric)
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = (
+ _label(explicit_x_axis)
+ if explicit_x_axis
+ else _x_axis_label(form_data, frontend_strict=True)
+ )
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
Review Comment:
Fixed in 0fc667fbcf76f893be11a254f0ec6ad3c73b81c1.
Saved physical `column_name` references are normalized before the time-column mutator, and Big Number no longer removes the granularity/grain binding. The saved mapping selects a BASE_AXIS `ds` with `timeGrain=P1M` while retaining `granularity=ds` and the requested time range, matching the string-axis contract.
`test_big_number_saved_axis_keeps_monthly_range_binding` covers the reported mapping and a SQL adhoc axis; the mapping case failed before and passes after. The native-axis pivot and actual preview query-contract tests also pass.
Validation: 1,931 passed, 3 skipped across the 14 touched pytest modules; pre-commit passed on all 83 branch-changed files, including mypy.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,792 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
Review Comment:
Fixed in 0fc667fbcf76f893be11a254f0ec6ad3c73b81c1.
Both Mixed grouping roles participate in omitted-control restoration, and the secondary-state lookup uses `group_by_secondary` for `groupby_b`. Explicit secondary grouping clears emit `[]`, preventing accidental inheritance from query A. XY preserves omitted ranking metrics; supplying either ranking-metric spelling replaces the saved aliases, and explicit null clears them. Cross-viz updates do not inherit these roles.
`test_mixed_update_preserves_only_omitted_grouping` and `test_xy_update_keeps_omitted_series_ranking_metric` check the resulting queries, including omission, independent clears, replacement, both ranking aliases, and cross-viz behavior. The reported omission cases failed before and pass after. Explicit-ranking/default-precedence and metric-validation tests also pass.
Validation: 1,931 passed, 3 skipped across the 14 touched pytest modules; pre-commit passed on all 83 branch-changed files, including mypy.
##########
superset/mcp_service/chart/preview_utils.py:
##########
@@ -1543,3 +1764,52 @@ def _generate_vega_lite_preview_from_data( # noqa: C901
data_url=None,
supports_streaming=False,
)
+
+
+def generate_xy_vega_lite_preview(
Review Comment:
Fixed in 0fc667fbcf76f893be11a254f0ec6ad3c73b81c1.
Mixed Timeseries has a Vega-Lite preview hook using the wide-result adapter, so the renderer folds all flattened grouped series instead of selecting one numeric column.
`test_grouped_timeseries_preview_keeps_all_pivoted_series` executes the real Mixed query post-processing pipeline and checks the rendered fold/color/value encodings for one or two grouping dimensions, with metric truncation enabled and disabled. All four Mixed wide-result cases failed before and pass after. Long-form results retain their original grouping/color encoding.
Validation: 1,931 passed, 3 skipped across the 14 touched pytest modules; pre-commit passed on all 83 branch-changed files, including mypy.
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1523 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+import base64
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.utils.serialization import BINARY_PREFIX
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _canonical_binary(
+ value: bytes | bytearray | memoryview,
+) -> tuple[str | None, str | None]:
+ """Render exact binary cells as UTF-8 text or ``base64:``-prefixed text.
+
+ Matches the MCP response serializer: valid UTF-8 is returned as text and
+ anything else is losslessly base64-encoded. The raw size is bounded before
+ copying so the rendered string stays within the per-cell text budget.
+ """
+ size = value.nbytes if type(value) is memoryview else len(value)
+ if size > MAX_RESULT_STRING_LENGTH:
+ return None, "an oversized binary value"
+ try:
+ raw = bytes(value)
+ except (BufferError, TypeError, ValueError):
+ return None, "an invalid binary value"
+ try:
+ text = raw.decode("utf-8")
+ except UnicodeDecodeError:
+ text = BINARY_PREFIX + base64.b64encode(raw).decode("ascii")
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized binary value"
+ return text, None
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+ if value_type is bytes or value_type is bytearray or value_type is memoryview:
+ return _canonical_binary(value)
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list or type(item) is tuple or type(item) is np.ndarray:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ if type(item) is np.ndarray:
+ if item.ndim == 0:
+ return None, "a non-sequence array"
+ width = item.shape[0]
+ elif type(item) is tuple:
+ width = tuple.__len__(item)
+ else:
+ width = list.__len__(cast(list[Any], item))
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ # Convert only exact producer sequences, after checking their size.
+ # Traverse children through the same depth/node/byte budgets, and
+ # track the original container so object-array cycles stay bounded.
+ if type(item) is tuple:
+ normalized_array = [
+ tuple.__getitem__(item, index) for index in range(width)
+ ]
+ elif type(item) is np.ndarray:
+ normalized_array = [
+ np.ndarray.__getitem__(item, index) for index in range(width)
+ ]
+ else:
+ normalized_array = cast(list[Any], item)
+ if normalized_array is not item:
+ if parent is None:
+ root = normalized_array
+ elif type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized_array)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized_array)
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (
+ list.__getitem__(normalized_array, index),
+ normalized_array,
+ index,
+ depth + 1,
+ False,
+ )
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ children: list[tuple[Any, dict[str, Any], str, int, bool]] = []
+ for key, child in dict.items(item):
+ if type(key) is not str:
Review Comment:
Fixed in 0fc667fbcf76f893be11a254f0ec6ad3c73b81c1.
Nested row maps stringify exact built-in keys within the existing key/text budgets, and non-finite Python/NumPy floating-point values become null recursively. Integer bounds, container/depth/node/byte limits, cycle checks, and rejection of arbitrary objects/subclasses remain enforced. Metadata retains its strict wire contract.
`test_nested_warehouse_values_keep_serializer_coercions` covers integer keys, nested maps/arrays, key collisions, and non-finite floats; its six cases failed before and pass after. Additional tests verify oversized/hostile keys are still rejected and exercise the reported map/array cells through the actual `get_chart_data`, `query_dataset`, and built-in `get_table` entry points.
Validation: 1,931 passed, 3 skipped across the 14 touched pytest modules; pre-commit passed on all 83 branch-changed files, including mypy.
##########
UPDATING.md:
##########
@@ -24,6 +24,10 @@ assists people when migrating to a new version.
## Next
+- MCP `update_chart` requires a complete `config` when changing `dataset_id`
Review Comment:
Fixed in 0fc667fbcf76f893be11a254f0ec6ad3c73b81c1.
UPDATING.md explicitly documents the mandatory result caps before response serialization and CSV/XLSX export, including the 50,000-row per-query limit and aggregate/container/text budgets. It states that raising `SQL_MAX_ROW` or raising/disabling `MCP_RESPONSE_SIZE_CONFIG` cannot change these fixed limits, and explains that oversized saved Table exports return `InvalidQueryResult` and require lower limits, filtering, narrower selection, or aggregation.
This is a documentation-only correction to the cap behavior: the unconditional validator calls before export and the constants in `query_result.py` were verified, and the existing per-query/aggregate row-cap tests pass. Runtime caps were not changed.
Validation: 1,931 passed, 3 skipped across the 14 touched pytest modules; pre-commit passed on all 83 branch-changed files, including mypy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4226114682
##########
superset/mcp_service/chart/tool/get_chart_sql.py:
##########
@@ -377,35 +450,74 @@ def _extract_sql_from_result(
errors: list[str] = []
language = "sql"
- for idx, query_result in enumerate(queries, start=1):
- if not isinstance(query_result, dict):
- continue
- if "language" in query_result and isinstance(query_result["language"], str):
- language = query_result["language"]
- query_sql = query_result.get("query", "")
- query_error = query_result.get("error")
+ source_bytes = 0
+ for offset in range(query_count):
+ idx = offset + 1
+ query_result = list.__getitem__(queries, offset)
+ if type(query_result) is not dict:
+ return ChartError(
+ error=f"Malformed chart SQL result: query {idx} must be an object.",
+ error_type="MalformedQueryResult",
+ )
+ query_language = dict.get(query_result, "language")
+ if query_language is not None:
+ if (
+ type(query_language) is not str
+ or _bounded_utf8_length(query_language, 256) is None
+ ):
+ return ChartError(
+ error=(
+ f"Malformed chart SQL result: query {idx} language must be "
+ "a bounded exact string."
+ ),
+ error_type="MalformedQueryResult",
+ )
+ language = query_language
+ query_sql = dict.get(query_result, "query", "")
+ query_error = dict.get(query_result, "error")
+ for label, value in (("query", query_sql), ("error", query_error)):
+ if value is None:
+ continue
+ if type(value) is not str:
Review Comment:
`QueryObjectValidationError` messages are built with `lazy_gettext` (for example "Empty query?" in `models/helpers.py`), so `query["error"]` is a lazy string rather than an exact `str`. This check turns it into `MalformedQueryResult: query 1 error must be an exact string` instead of the previous `QueryGenerationFailed` carrying the real message, e.g. `get_chart_sql` on a saved chart that has no metrics or columns. `get_chart_data` resolves lazy strings in `response_preflight`; should this path resolve them the same way before the type check?
##########
docs/admin_docs/configuration/mcp-server.mdx:
##########
@@ -1447,6 +1454,10 @@ For Generic Chart Axes charts, both MCP overrides and dashboard time-grain
filters update the `BASE_AXIS` column's `timeGrain`, regardless of its position
in the query's `columns` list. Other columns are left untouched. Queries without
a `BASE_AXIS` column continue to use `extras.time_grain_sqla`.
+Same-visualization updates that supply only `time_grain` preserve saved temporal
Review Comment:
This says a `time_grain`-only update keeps saved temporal ranges even when the grain is explicitly cleared, but the update overlay calls `_scrub_temporal_form_data` for any explicit null `time_grain` (the Sunburst finalizer does the same), which removes `time_range` and the `TEMPORAL_RANGE` filters. Someone clearing the grain on a chart saved with "Last month" ends up with no date restriction. Is the intent to keep the range (a code change) or should this sentence say a null grain clears it?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1784,804 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "BubbleChartConfig": {
+ "series": (("series",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "series_limit_metric": (("series_limit_metric",), ("timeseries_limit_metric",)),
+ "timeseries_limit_metric": (
+ ("series_limit_metric",),
+ ("timeseries_limit_metric",),
+ ),
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "groupby": (("group_by",),),
+ "groupby_b": (("group_by_secondary",),),
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ requested_filters = getattr(config, "filters", None)
+ if requested_filters is not None:
+ # Legacy ``filters``/``where``/``having`` are reconstructed into adhoc
+ # filters at query time, so an explicit clear or replacement removes
+ # every saved source the new form data does not set itself.
+ for key in _SAVED_PREDICATE_FORM_DATA_KEYS - new_form_data.keys():
+ merged.pop(key, None)
+ else:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool({"temporal_column", "time_range"} & fields_set),
+ )
+ if filters is not None:
+ merged["adhoc_filters"] = filters
+ return merged
+
+
+def retain_mixed_timeseries_secondary_update_state(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Return saved query-B controls a same-viz Mixed Timeseries update keeps.
+
+ Query B inherits unsuffixed controls only when the suffixed key is absent.
+ Preserve explicit native clears for controls the typed mapper did not
+ replace, so []/None never turns into accidental inheritance from query A.
+ Valid modeled secondary controls and comparison state are also retained;
+ malformed controls and stale replacement roles remain fail-closed.
+ """
+ from superset.common.form_data_query_context import (
+ MIXED_TIMESERIES_SECONDARY_QUERY_KEYS,
+ )
+
+ retained: Dict[str, Any] = {}
+ for key in MIXED_TIMESERIES_SECONDARY_QUERY_KEYS:
+ if key in new_form_data or key not in existing_form_data:
+ continue
+ value = existing_form_data[key]
+ is_explicit_clear = value is None or value in ([], {}, "")
+ is_valid_comparison = key == "comparison_type_b" and value in {
+ "values",
+ "difference",
+ "percentage",
+ "ratio",
+ }
+ is_valid_list_state = key in {
+ "adhoc_filters_b",
+ "annotation_layers_b",
+ "time_compare_b",
+ } and isinstance(value, list)
+ is_valid_modeled_control = False
+ if field := MixedTimeseriesChartConfig.model_fields.get(
+ {"groupby_b": "group_by_secondary"}.get(key, key)
+ ):
+ try:
+ TypeAdapter(field.rebuild_annotation()).validate_python(value)
Review Comment:
Retained `*_b` controls are validated against the typed `ColumnRef` annotation, but Explore saves a ranking metric as a plain saved-metric string (or an adhoc metric dict). A saved `series_limit_metric_b="revenue"` fails that validation, so a presentation-only update of a Mixed Timeseries (say just `show_legend=false`) drops the ranking metric while `series_limit_b` survives, and query B can then rank by a different metric. Should string and adhoc-metric values be accepted here?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +427,1295 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
Review Comment:
Query B now inherits every unsuffixed control it doesn't set, including `groupby`. The Mixed Timeseries mapper only writes `groupby_b` when `group_by_secondary` is supplied, so a `generate_chart` payload with `group_by=["region"]` and no secondary grouping used to give an ungrouped query B. Now `get_chart_data` on the returned key splits the secondary series by region, while Explore (which defaults `groupby_b` to empty) shows it ungrouped. Should the mapper emit `groupby_b: []` when secondary grouping is omitted?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1784,804 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "BubbleChartConfig": {
+ "series": (("series",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "series_limit_metric": (("series_limit_metric",), ("timeseries_limit_metric",)),
+ "timeseries_limit_metric": (
+ ("series_limit_metric",),
+ ("timeseries_limit_metric",),
+ ),
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "groupby": (("group_by",),),
+ "groupby_b": (("group_by_secondary",),),
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
Review Comment:
On a same-viz Table update every key in the plugin's `query_role_keys` is dropped before the overlay, and that set now includes `percent_metrics`. `TableChartConfig` has no field for it and nothing in the modeled-update path restores it, so an Explore-built aggregate table with a percent-metric column updated via `update_chart(config={chart_type: "table", columns: [<same>], row_limit: 50})` silently loses that column; the previous shallow merge kept it. Should `percent_metrics` be carried over when the update doesn't restate it?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1628 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+import base64
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
+from importlib import import_module
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.utils.serialization import BINARY_PREFIX
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+def _postgres_range_types() -> tuple[type[Any], ...]:
+ """Discover concrete range scalars only for installed PostgreSQL drivers."""
+ range_types: list[type[Any]] = []
+ for module_name, names in (
+ (
+ "psycopg2.extras",
+ ("Range", "NumericRange", "DateRange", "DateTimeRange", "DateTimeTZRange"),
+ ),
+ (
+ "psycopg.types.range",
+ (
+ "Range",
+ "Int4Range",
+ "Int8Range",
+ "NumericRange",
+ "DateRange",
+ "TimestampRange",
+ "TimestamptzRange",
+ ),
+ ),
+ ):
+ try:
+ module = import_module(module_name)
+ except ImportError:
+ continue
+ range_types.extend(getattr(module, name) for name in names)
+ return tuple(range_types)
+
+
+_POSTGRES_RANGE_TYPES = _postgres_range_types()
+
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _canonical_binary(
+ value: bytes | bytearray | memoryview,
+) -> tuple[str | None, str | None]:
+ """Render exact binary cells as UTF-8 text or ``base64:``-prefixed text.
+
+ Matches the MCP response serializer: valid UTF-8 is returned as text and
+ anything else is losslessly base64-encoded. The raw size is bounded before
+ copying so the rendered string stays within the per-cell text budget.
+ """
+ size = value.nbytes if type(value) is memoryview else len(value)
+ if size > MAX_RESULT_STRING_LENGTH:
+ return None, "an oversized binary value"
+ try:
+ raw = bytes(value)
+ except (BufferError, TypeError, ValueError):
+ return None, "an invalid binary value"
+ try:
+ text = raw.decode("utf-8")
+ except UnicodeDecodeError:
+ text = BINARY_PREFIX + base64.b64encode(raw).decode("ascii")
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized binary value"
+ return text, None
+
+
+def _canonical_postgres_range(value: Any) -> tuple[str | None, str | None]:
+ """Render exact driver ranges after bounding and validating their endpoints.
+
+ Never invoke a range's string hook: even a concrete range can contain a
+ hostile endpoint. Only exact numeric and temporal bounds are supported.
+ """
+ try:
+ bounds = object.__getattribute__(value, "_bounds")
+ lower = object.__getattribute__(value, "_lower")
+ upper = object.__getattribute__(value, "_upper")
+ except AttributeError:
+ return None, "an invalid PostgreSQL range"
+ if bounds is None or (type(bounds) is str and bounds == ""):
+ return "empty", None
+ if type(bounds) is not str or bounds not in {"[)", "(]", "()", "[]"}:
+ return None, "an invalid PostgreSQL range"
+ endpoints: list[str] = []
+ for endpoint in (lower, upper):
+ if endpoint is None:
+ endpoints.append("None")
+ continue
+ if not any(
+ type(endpoint) is trusted
+ for trusted in (int, float, Decimal, date, datetime)
+ ):
+ return None, "an unsupported PostgreSQL range endpoint"
+ normalized, reason = _normalize_scalar(endpoint)
+ if reason or normalized is None:
+ return None, reason or "an invalid PostgreSQL range endpoint"
+ # Canonical temporal text has already validated timezone state; ranges
+ # use the driver's space separator instead of datetime's ISO 'T'.
+ text = (
+ normalized.replace("T", " ", 1)
+ if type(endpoint) is datetime
+ else str(normalized)
+ )
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized PostgreSQL range"
+ endpoints.append(text)
+ text = f"{bounds[0]}{endpoints[0]}, {endpoints[1]}{bounds[1]}"
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized PostgreSQL range"
+ return text, None
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ return (value, None) if math.isfinite(value) else (None, None)
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+ if value_type is bytes or value_type is bytearray or value_type is memoryview:
+ return _canonical_binary(value)
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, None)
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ if any(value_type is trusted for trusted in _POSTGRES_RANGE_TYPES):
+ return _canonical_postgres_range(value)
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list or type(item) is tuple or type(item) is np.ndarray:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ if type(item) is np.ndarray:
+ if item.ndim == 0:
+ return None, "a non-sequence array"
+ width = item.shape[0]
+ elif type(item) is tuple:
+ width = tuple.__len__(item)
+ else:
+ width = list.__len__(cast(list[Any], item))
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ # Convert only exact producer sequences, after checking their size.
+ # Traverse children through the same depth/node/byte budgets, and
+ # track the original container so object-array cycles stay bounded.
+ if type(item) is tuple:
+ normalized_array = [
+ tuple.__getitem__(item, index) for index in range(width)
+ ]
+ elif type(item) is np.ndarray:
+ normalized_array = [
+ np.ndarray.__getitem__(item, index) for index in range(width)
+ ]
+ else:
+ normalized_array = cast(list[Any], item)
+ if normalized_array is not item:
+ if parent is None:
+ root = normalized_array
+ elif type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized_array)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized_array)
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (
+ list.__getitem__(normalized_array, index),
+ normalized_array,
+ index,
+ depth + 1,
+ False,
+ )
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ normalized_object: dict[str, Any] = {}
+ for key, child in dict.items(item):
+ # Only exact built-in keys may be stringified: arbitrary objects
+ # and subclasses can run conversion hooks. Bound integers before
+ # rendering and charge each rendered key against the text budget.
+ if metadata and type(key) is not str:
+ return None, "a non-string metadata key"
+ if type(key) is int:
+ if reason := _integer_failure(key):
+ return None, reason
+ elif type(key) not in {str, bool, float, type(None)}:
+ return None, "an unsupported object key"
+ normalized_key = key if type(key) is str else str(key)
+ if reason := _charge_text(
+ normalized_key, budget, key=True, metadata=metadata
+ ):
+ return None, reason
+ normalized_object[normalized_key] = child
+ if parent is None:
+ root = normalized_object
+ elif type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized_object)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized_object)
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (child, normalized_object, key, depth + 1, False)
+ for key, child in reversed(list(dict.items(normalized_object)))
+ )
+ continue
+
+ source_item = item
+ if type(item) in enum_types:
+ normalized = _safe_enum_value(item, enum_types)
+ if normalized is None:
+ return None, "an unsupported enum"
+ item = normalized
+ elif any(base is Enum for base in _type_mro(type(item))):
+ return None, "an enum outside its expected metadata slot"
+
+ # Metadata retains its strict wire contract (for example a rowcount
+ # cannot become null merely because the producer supplied infinity).
+ if metadata and (type(item) is float or type(item) in _NUMPY_FLOAT_TYPES):
+ number = float(item)
+ if not math.isfinite(number) and not math.isnan(number):
+ return None, "a non-finite metadata number"
+ normalized, reason = _normalize_scalar(item)
+ if reason is not None:
+ return None, reason
+ max_string_bytes = (
+ MAX_QUERY_RESULT_METADATA_BYTES if metadata else MAX_RESULT_STRING_LENGTH
+ )
+ if type(normalized) is str:
+ scalar_size = _json_string_size(normalized, max_string_bytes)
+ if scalar_size is None:
+ return None, "invalid text data"
+ else:
+ scalar_size = _normalized_scalar_json_size(normalized)
+ if reason := _charge_json_bytes(
+ budget,
+ scalar_size,
+ metadata=metadata,
+ ):
+ return None, reason
+ if parent is None:
+ root = normalized
+ elif normalized is not source_item:
+ if type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized)
+ return root, None
+
+
+def _normalize_metadata_value(
+ payload: dict[str, Any], key: str, budget: _ResultBudget
+) -> str | None:
+ enum_slots: dict[str, frozenset[type[Any]]] = {
+ "status": frozenset({QueryStatus}),
+ "result_format": frozenset({ChartDataResultFormat}),
+ "coltypes": frozenset({GenericDataType}),
+ "applied_filters": frozenset({ExtraFiltersTimeColumnType}),
+ "rejected_filters": frozenset(
+ {ExtraFiltersReasonType, ExtraFiltersTimeColumnType}
+ ),
+ }
+ value = dict.__getitem__(payload, key)
+ normalized, reason = _normalize_value(
+ value,
+ budget,
+ enum_types=enum_slots.get(key, frozenset()),
+ metadata=True,
+ )
+ if reason is None and normalized is not value:
+ dict.__setitem__(payload, key, normalized)
+ return reason
+
+
+def _error_text(value: Any) -> str | None:
+ """Extract a bounded error from an already validated primitive tree."""
+ stack: list[Any] = [value]
+ parts: list[str] = []
+ used = 0
+ while stack and len(parts) < 3 and used < _MAX_ERROR_TEXT_BYTES:
+ item = stack.pop()
+ if type(item) is dict:
+ stack.extend(
+ reversed(
+ [dict.__getitem__(item, key) for key in _ERROR_KEYS if key in item]
+ )
+ )
+ continue
+ if type(item) is list:
+ stack.extend(
+ list.__getitem__(item, index)
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if item is None or item is False:
+ continue
+ if type(item) is str:
+ remaining = _MAX_ERROR_TEXT_BYTES - used - (2 if parts else 0)
+ encoded = str.encode(item, "utf-8")[:remaining]
+ text = bytes.decode(encoded, "utf-8", errors="ignore")
+ if text:
+ parts.append(text)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ return "; ".join(parts) or None
+
+
+def _failure_for_payload(payload: dict[str, Any], label: str) -> ChartError | None:
for key in ("error", "errors", "error_message"):
- if message := _query_error_text(payload.get(key)):
+ if key in payload and (message := _error_text(dict.__getitem__(payload, key))):
return ChartError(
error=f"{label} failed: {message}", error_type="QueryError"
)
- raw_status = payload.get("status")
- status = str(getattr(raw_status, "value", raw_status) or "")
+ raw_status = dict.get(payload, "status")
+ status = raw_status if type(raw_status) is str else ""
normalized_status = status.strip().casefold().replace("-", "_").replace(" ", "_")
if normalized_status in FAILED_QUERY_STATUSES:
message = (
- _query_error_text(payload.get("message"))
- or _query_error_text(payload.get("error_message"))
+ _error_text(dict.get(payload, "message"))
+ or _error_text(dict.get(payload, "error_message"))
or normalized_status
)
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
- if payload.get("success") is False:
- message = _query_error_text(payload.get("message")) or "request failed"
+ if dict.get(payload, "success") is False:
+ message = _error_text(dict.get(payload, "message")) or "request failed"
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if raw_status is None and "data" not in payload and "queries" not in payload:
+ if message := _error_text(dict.get(payload, "message")):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+ return None
+
+
+def _metadata_shape_error( # noqa: C901
+ payload: dict[str, Any], label: str, budget: _ResultBudget
+) -> ChartError | None:
+ if "success" in payload and type(dict.__getitem__(payload, "success")) is not bool:
+ return _invalid_metadata(label)
+ if "status" in payload and type(dict.__getitem__(payload, "status")) is not str:
+ return _invalid_metadata(label)
if (
- raw_status is None
- and "data" not in payload
- and "queries" not in payload
- and (message := _query_error_text(payload.get("message")))
+ "result_format" in payload
+ and dict.__getitem__(payload, "result_format") not in _RESULT_FORMAT_VALUES
):
- return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ return _invalid_metadata(label)
+ for count_key in ("rowcount", "sql_rowcount", "total_rows"):
+ if count_key in payload:
+ count = dict.__getitem__(payload, count_key)
+ if count is not None and not (
+ type(count) is int and 0 <= count <= MAX_QUERY_RESULT_ROWCOUNT
+ ):
+ return _invalid_metadata(label)
+ if "is_cached" in payload:
+ cached = dict.__getitem__(payload, "is_cached")
+ if cached is None:
+ if reason := _charge_json_bytes(
+ budget, 1, metadata=True
+ ): # ``false`` vs ``null``
+ return _invalid_result(reason)
+ dict.__setitem__(payload, "is_cached", False)
+ elif type(cached) is not bool:
+ return _invalid_metadata(label)
+ if "cache_timeout" in payload:
+ timeout = dict.__getitem__(payload, "cache_timeout")
+ if timeout is not None and not (
+ type(timeout) is int
+ and (
+ timeout == CACHE_DISABLED_TIMEOUT
+ or 0 <= timeout <= MAX_QUERY_RESULT_CACHE_TIMEOUT
+ )
+ ):
+ return _invalid_metadata(label)
+ if "cache_key" in payload:
+ cache_key = dict.__getitem__(payload, "cache_key")
+ if cache_key is not None and (type(cache_key) is not str or not cache_key):
+ return _invalid_metadata(label)
+ for timestamp_key in ("cached_dttm", "cache_dttm", "queried_dttm"):
+ if timestamp_key not in payload:
+ continue
+ timestamp = dict.__getitem__(payload, timestamp_key)
+ if timestamp is None:
+ continue
+ if (
+ type(timestamp) is not str
+ or not timestamp
+ or len(timestamp) > MAX_QUERY_RESULT_TIMESTAMP_LENGTH
+ ):
+ return _invalid_metadata(label)
+ normalized = f"{timestamp[:-1]}+00:00" if timestamp.endswith("Z") else timestamp
+ try:
+ parsed = datetime.fromisoformat(normalized)
+ except ValueError:
+ return _invalid_metadata(label)
+ if parsed.tzinfo is None:
+ parsed = parsed.replace(tzinfo=timezone.utc)
+ if parsed.utcoffset() != timedelta(0):
+ return _invalid_metadata(label)
+ canonical_timestamp = parsed.astimezone(timezone.utc).isoformat()
+ original_size = _json_string_size(timestamp, MAX_RESULT_STRING_LENGTH)
+ canonical_size = _json_string_size(
+ canonical_timestamp, MAX_RESULT_STRING_LENGTH
+ )
+ assert original_size is not None
+ assert canonical_size is not None
+ if canonical_size > original_size and (
+ reason := _charge_json_bytes(
+ budget, canonical_size - original_size, metadata=True
+ )
+ ):
+ return _invalid_result(reason)
+ dict.__setitem__(payload, timestamp_key, canonical_timestamp)
+ return None
+
+
+def _filter_metadata_is_valid(value: Any, *, rejected: bool) -> bool:
+ if type(value) is not list or list.__len__(value) > MAX_RESULT_VALUE_ITEMS:
+ return False
+ expected = {"column", "reason"} if rejected else {"column"}
+ for entry in value:
+ if type(entry) is not dict or set(dict.keys(entry)) != expected:
+ return False
+ if type(dict.__getitem__(entry, "column")) is not str:
+ return False
+ if rejected and type(dict.__getitem__(entry, "reason")) is not str:
+ return False
+ return True
+
+
+def _validate_query_metadata(query: dict[str, Any], index: int) -> ChartError | None:
+ label = f"Chart query {index}"
+ colnames_present = "colnames" in query
+ coltypes_present = "coltypes" in query
+ colnames = dict.get(query, "colnames", [])
+ coltypes = dict.get(query, "coltypes", [])
+ if (
+ type(colnames) is not list
+ or list.__len__(colnames) > MAX_QUERY_RESULT_COLUMNS
+ or any(type(column) is not str or not column for column in colnames)
+ ):
+ return _invalid_result(f"malformed column metadata for query {index}")
+ if (
+ type(coltypes) is not list
+ or list.__len__(coltypes) > MAX_QUERY_RESULT_COLUMNS
+ or any(
+ type(value) is not int or value not in _COLTYPE_VALUES for value in coltypes
+ )
+ ):
+ return _invalid_result(f"malformed column type metadata for query {index}")
+ if (colnames_present or coltypes_present) and (
+ not colnames_present
+ or not coltypes_present
+ or list.__len__(colnames) != list.__len__(coltypes)
+ ):
+ return _invalid_result(f"misaligned column metadata for query {index}")
+ if "applied_filters" in query and not _filter_metadata_is_valid(
+ dict.__getitem__(query, "applied_filters"), rejected=False
+ ):
+ return _invalid_metadata(label)
+ if "rejected_filters" in query and not _filter_metadata_is_valid(
+ dict.__getitem__(query, "rejected_filters"), rejected=True
+ ):
+ return _invalid_metadata(label)
+ if "rejected_filter_columns" in query:
+ rejected_columns = dict.__getitem__(query, "rejected_filter_columns")
+ if type(rejected_columns) is not list or any(
Review Comment:
`rejected_filter_columns` can hold the adhoc column dict itself: `models/helpers.py` appends the adhoc `flt_col` to `rejected_adhoc_filters_columns` when `adhoc_column_to_sqla` raises `ColumnNotFoundException`, and that list is concatenated into `rejected_filter_columns`. Requiring every entry to be a `str` makes such a result fail as malformed metadata, so `get_chart_data` on a chart with a custom-SQL filter referencing a since-removed column returns an error instead of the rows Explore still renders with a rejected-filter badge. Should non-string entries be tolerated here, as the `isinstance(column, str)` guard in `chart_helpers.py` already does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5963804908
@rusackas #44218 and #44219 have both merged. I merged master into this branch (no history rewrite), so the diff is now net of both, and CI is green on the current head. Ready for your review when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5968667564
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5497266749
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3927462037
##########
superset/mcp_service/chart/sunburst.py:
##########
@@ -0,0 +1,329 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Shared native Sunburst result-role resolution and validation."""
+
+import math
+from collections.abc import Mapping
+from dataclasses import dataclass
+from decimal import Decimal
+from typing import Any
+
+from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.common.error_schemas import DatasetContext
+
+
+@dataclass(frozen=True)
+class SunburstResultRoles:
+ """Resolved query-output fields required to render a Sunburst."""
+
+ hierarchy: tuple[str, ...]
+ primary_metric: str
+ secondary_metric: str | None = None
+
+
+def _column_result_label(column: Any) -> str | None:
+ """Resolve a bounded native column reference to its query output label."""
+ if isinstance(column, str) and column:
+ return column
+ if not isinstance(column, Mapping) or not 0 < len(column) <= 20:
+ return None
+ for key in ("label", "column_name", "columnName", "sqlExpression"):
+ value = column.get(key)
+ if isinstance(value, str) and value:
+ return value
+ if value not in (None, ""):
+ return None
+ return None
+
+
+def _metric_result_label(metric: Any) -> str | None:
+ """Resolve saved, SIMPLE, and SQL metric aliases like the frontend."""
+ if isinstance(metric, str) and metric:
+ return metric
+ if not isinstance(metric, Mapping) or not 0 < len(metric) <= 20:
+ return None
+
+ label = metric.get("label")
+ if isinstance(label, str) and label:
+ return label
+ if label not in (None, ""):
+ return None
+
+ expression_type = metric.get("expressionType")
+ if expression_type == "SIMPLE":
+ aggregate = metric.get("aggregate")
+ column = metric.get("column")
+ if (
+ not isinstance(aggregate, str)
+ or not aggregate
+ or len(aggregate) > 100
+ or not isinstance(column, Mapping)
+ or not 0 < len(column) <= 50
+ ):
+ return None
+ column_name = column.get("column_name") or column.get("columnName")
+ if not isinstance(column_name, str) or not column_name:
+ return None
+ return f"{aggregate}({column_name})"
+ if expression_type == "SQL":
+ sql_expression = metric.get("sqlExpression")
+ if (
+ isinstance(sql_expression, str)
+ and sql_expression
+ and len(sql_expression) <= 2000
+ ):
+ return sql_expression
+ return None
+
+
+def _metric_query_identity(metric: Any) -> tuple[str, ...] | None:
+ """Return a hook-free identity for one exact native metric value."""
+ if type(metric) is str and metric:
+ return ("saved", metric)
+ if type(metric) is not dict:
+ return None
+ expression_type = dict.get(metric, "expressionType")
+ if expression_type == "SQL":
+ expression = dict.get(metric, "sqlExpression")
+ return ("sql", expression) if type(expression) is str and expression else None
+ if expression_type == "SIMPLE":
+ aggregate = dict.get(metric, "aggregate")
+ column = dict.get(metric, "column")
+ if type(aggregate) is not str:
+ return None
+ column_name: Any
+ if type(column) is str:
+ column_name = column
+ elif type(column) is dict:
+ column_name = dict.get(column, "column_name")
+ if column_name is None:
+ column_name = dict.get(column, "columnName")
+ else:
+ return None
+ if type(column_name) is str and column_name:
+ return ("simple", aggregate, column_name)
+ return None
+
+
+def resolve_sunburst_result_roles(
+ form_data: Mapping[str, Any],
+) -> tuple[SunburstResultRoles | None, ChartError | None]:
+ """Resolve and validate all native Sunburst query-result roles."""
+ columns = form_data.get("columns")
+ if not isinstance(columns, list) or not columns:
+ return None, ChartError(
+ error="Sunburst form data requires one or more hierarchy columns.",
+ error_type="InvalidSunburstFormData",
+ )
+ hierarchy: list[str] = []
+ for index, column in enumerate(columns):
+ label = _column_result_label(column)
+ if label is None:
+ return None, ChartError(
+ error=f"Sunburst hierarchy column {index + 1} is malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ hierarchy.append(label)
+
+ primary = _metric_result_label(form_data.get("metric"))
+ if primary is None:
+ return None, ChartError(
+ error="Sunburst primary metric is missing or malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ primary_metric = form_data.get("metric")
+ secondary: str | None = None
+ if (secondary_metric := form_data.get("secondary_metric")) is not None:
+ secondary = _metric_result_label(secondary_metric)
+ if secondary is None:
+ return None, ChartError(
+ error="Sunburst secondary metric is malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ if (
+ secondary.casefold() == primary.casefold()
+ and _metric_query_identity(primary_metric)
+ == _metric_query_identity(secondary_metric)
+ and _metric_query_identity(primary_metric) is not None
+ ):
+ # The frontend interprets a repeated primary metric as categorical
+ # color mode and the query has one physical metric output.
+ secondary = None
+
+ labels = [*hierarchy, primary, *([secondary] if secondary else [])]
+ seen: dict[str, str] = {}
+ for label in labels:
+ folded = label.casefold()
+ if previous := seen.get(folded):
+ return None, ChartError(
+ error=(
+ f"Sunburst result label {label!r} is ambiguous with {previous!r}."
+ ),
+ error_type="InvalidSunburstFormData",
+ )
+ seen[folded] = label
+
+ return SunburstResultRoles(tuple(hierarchy), primary, secondary), None
+
+
+def _finite_number(value: Any) -> bool:
+ """Return whether a normalized database value is exactly numeric and finite."""
+ value_type = type(value)
+ if value_type is int:
+ return True
+ if value_type is float:
+ return math.isfinite(value)
+ if value_type is Decimal:
+ return Decimal.is_finite(value)
+ return False
+
+
+def _valid_hierarchy_value(value: Any) -> bool:
+ """Reject nested/container values that cannot form a stable node label."""
+ return not isinstance(value, (Mapping, list, tuple, set))
+
+
+def validate_sunburst_result_data(
+ data: Any, form_data: Mapping[str, Any]
+) -> tuple[SunburstResultRoles | None, ChartError | None]:
+ """Validate every Sunburst row and all resolved result aliases."""
+ roles, error = resolve_sunburst_result_roles(form_data)
+ if error is not None:
+ return None, error
+ assert roles is not None
+ if not isinstance(data, list):
+ return None, ChartError(
+ error="Sunburst query result data is not an array of rows.",
+ error_type="InvalidSunburstResult",
+ )
+
+ required_fields = [
+ *roles.hierarchy,
+ roles.primary_metric,
+ *([roles.secondary_metric] if roles.secondary_metric else []),
+ ]
+ for index, row in enumerate(data, start=1):
+ if not isinstance(row, Mapping):
+ return None, ChartError(
+ error=f"Sunburst result row {index} is not an object.",
+ error_type="InvalidSunburstResult",
+ )
+ missing = [field for field in required_fields if field not in row]
+ if missing:
+ return None, ChartError(
+ error=(
+ f"Sunburst result row {index} is missing required field(s): "
+ f"{', '.join(str(field) for field in missing)}."
+ ),
+ error_type="InvalidSunburstResult",
+ )
+ for field in roles.hierarchy:
+ if not _valid_hierarchy_value(row[field]):
+ return None, ChartError(
+ error=(
+ f"Sunburst result row {index} has a malformed hierarchy "
+ f"value for {field!r}."
+ ),
+ error_type="InvalidSunburstResult",
+ )
+ for field in (
+ roles.primary_metric,
+ *([roles.secondary_metric] if roles.secondary_metric else []),
+ ):
+ assert field is not None
+ if not _finite_number(row[field]):
Review Comment:
A NULL aggregate value now rejects the entire Sunburst result even though the frontend treats a non-numeric metric as zero. A group whose `SUM(revenue)` is NULL can therefore make generate/preview fail (and compile sampling makes it intermittent); could this normalize missing metric values to the frontend behavior before validating?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -0,0 +1,1520 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+
+import math
+import time as system_time
+from bisect import bisect_right
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
+from decimal import Decimal
+from enum import Enum
+from typing import Any
+from uuid import UUID
+from zoneinfo import ZoneInfo
+
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
+
+FAILED_QUERY_STATUSES = frozenset(
+ {"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
+)
+
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
+ return None
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
+
+
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = list.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (list.__getitem__(item, index), item, index, depth + 1, False)
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ children: list[tuple[Any, dict[str, Any], str, int, bool]] = []
+ for key, child in dict.items(item):
+ if type(key) is not str:
+ return None, "a non-string object key"
+ if reason := _charge_text(key, budget, key=True, metadata=metadata):
+ return None, reason
+ children.append((child, item, key, depth + 1, False))
+ stack.append((item, None, None, depth, True))
+ stack.extend(reversed(children))
+ continue
+
+ source_item = item
+ if type(item) in enum_types:
+ normalized = _safe_enum_value(item, enum_types)
+ if normalized is None:
+ return None, "an unsupported enum"
+ item = normalized
+ elif any(base is Enum for base in _type_mro(type(item))):
+ return None, "an enum outside its expected metadata slot"
+
+ normalized, reason = _normalize_scalar(item)
+ if reason is not None:
+ return None, reason
+ max_string_bytes = (
+ MAX_QUERY_RESULT_METADATA_BYTES if metadata else MAX_RESULT_STRING_LENGTH
+ )
+ if type(normalized) is str:
+ scalar_size = _json_string_size(normalized, max_string_bytes)
+ if scalar_size is None:
+ return None, "invalid text data"
+ else:
+ scalar_size = _normalized_scalar_json_size(normalized)
+ if reason := _charge_json_bytes(
+ budget,
+ scalar_size,
+ metadata=metadata,
+ ):
+ return None, reason
+ if parent is None:
+ root = normalized
+ elif normalized is not source_item:
+ if type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized)
+ return root, None
+
+
+def _normalize_metadata_value(
+ payload: dict[str, Any], key: str, budget: _ResultBudget
+) -> str | None:
+ enum_slots: dict[str, frozenset[type[Any]]] = {
+ "status": frozenset({QueryStatus}),
+ "result_format": frozenset({ChartDataResultFormat}),
+ "coltypes": frozenset({GenericDataType}),
+ "applied_filters": frozenset({ExtraFiltersTimeColumnType}),
+ "rejected_filters": frozenset(
+ {ExtraFiltersReasonType, ExtraFiltersTimeColumnType}
+ ),
+ }
+ value = dict.__getitem__(payload, key)
+ normalized, reason = _normalize_value(
+ value,
+ budget,
+ enum_types=enum_slots.get(key, frozenset()),
+ metadata=True,
+ )
+ if reason is None and normalized is not value:
+ dict.__setitem__(payload, key, normalized)
+ return reason
+
+
+def _error_text(value: Any) -> str | None:
+ """Extract a bounded error from an already validated primitive tree."""
+ stack: list[Any] = [value]
+ parts: list[str] = []
+ used = 0
+ while stack and len(parts) < 3 and used < _MAX_ERROR_TEXT_BYTES:
+ item = stack.pop()
+ if type(item) is dict:
+ stack.extend(
+ reversed(
+ [dict.__getitem__(item, key) for key in _ERROR_KEYS if key in item]
+ )
+ )
+ continue
+ if type(item) is list:
+ stack.extend(
+ list.__getitem__(item, index)
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if item is None or item is False:
+ continue
+ if type(item) is str:
+ remaining = _MAX_ERROR_TEXT_BYTES - used - (2 if parts else 0)
+ encoded = str.encode(item, "utf-8")[:remaining]
+ text = bytes.decode(encoded, "utf-8", errors="ignore")
+ if text:
+ parts.append(text)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ return "; ".join(parts) or None
+
+
+def _failure_for_payload(payload: dict[str, Any], label: str) -> ChartError | None:
+ for key in ("error", "errors", "error_message"):
+ if key in payload and (message := _error_text(dict.__getitem__(payload, key))):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+
+ raw_status = dict.get(payload, "status")
+ status = raw_status if type(raw_status) is str else ""
+ normalized_status = status.strip().casefold().replace("-", "_").replace(" ", "_")
+ if normalized_status in FAILED_QUERY_STATUSES:
+ message = (
+ _error_text(dict.get(payload, "message"))
+ or _error_text(dict.get(payload, "error_message"))
+ or normalized_status
+ )
+ return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if dict.get(payload, "success") is False:
+ message = _error_text(dict.get(payload, "message")) or "request failed"
+ return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if raw_status is None and "data" not in payload and "queries" not in payload:
+ if message := _error_text(dict.get(payload, "message")):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+ return None
+
+
+def _metadata_shape_error( # noqa: C901
+ payload: dict[str, Any], label: str, budget: _ResultBudget
+) -> ChartError | None:
+ if "success" in payload and type(dict.__getitem__(payload, "success")) is not bool:
+ return _invalid_metadata(label)
+ if "status" in payload and type(dict.__getitem__(payload, "status")) is not str:
+ return _invalid_metadata(label)
+ if (
+ "result_format" in payload
+ and dict.__getitem__(payload, "result_format") not in _RESULT_FORMAT_VALUES
+ ):
+ return _invalid_metadata(label)
+ for count_key in ("rowcount", "sql_rowcount", "total_rows"):
+ if count_key in payload:
+ count = dict.__getitem__(payload, count_key)
+ if count is not None and not (
+ type(count) is int and 0 <= count <= MAX_QUERY_RESULT_ROWCOUNT
+ ):
+ return _invalid_metadata(label)
+ if "is_cached" in payload:
+ cached = dict.__getitem__(payload, "is_cached")
+ if cached is None:
+ if reason := _charge_json_bytes(
+ budget, 1, metadata=True
+ ): # ``false`` vs ``null``
+ return _invalid_result(reason)
+ dict.__setitem__(payload, "is_cached", False)
+ elif type(cached) is not bool:
+ return _invalid_metadata(label)
+ if "cache_timeout" in payload:
+ timeout = dict.__getitem__(payload, "cache_timeout")
+ if timeout is not None and not (
Review Comment:
This rejects Superset's supported `cache_timeout = -1` sentinel, which `QueryContextProcessor` includes whenever caching is disabled. Any MCP chart query using that valid setting now fails the new result-envelope check; could the validator allow `CACHE_DISABLED_TIMEOUT`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3971738907
##########
superset/mcp_service/chart/chart_helpers.py:
##########
@@ -446,12 +564,200 @@ def _resolve_deck_gl_metrics(
if value:
metrics.append(value)
elif isinstance(prf, str) and _is_metric_ref(prf):
- # Legacy deck_scatter: point_radius_fixed as a bare non-numeric metric key
- logger.debug("Legacy point_radius_fixed string metric encountered: %s", prf)
metrics.append(prf)
return metrics
+def _deck_query_adapter( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any], viz_type: str
+) -> dict[str, Any]:
+ """Apply the native frontend builder for a single Deck.gl layer."""
+ base_columns = list(query.get("columns") or [])
+ base_metrics = list(query.get("metrics") or [])
+ filters = list(query.get("filters") or [])
+ tooltips = _deck_tooltip_columns(form_data.get("tooltip_contents"))
+
+ def add_null(column: str, *, value: Any = ...) -> None:
+ clause: dict[str, Any] = {"col": column, "op": "IS NOT NULL"}
+ if value is not ...:
+ clause["val"] = value
+ filters.append(clause)
+
+ if viz_type == "deck_geojson":
+ geometry = form_data.get("geojson")
+ if not isinstance(geometry, str) or not geometry:
+ raise ValueError("GeoJSON column is required for GeoJSON charts")
+ columns = _add_deck_columns(base_columns, [geometry] if geometry else [])
+ cross_filter = form_data.get("cross_filter_column")
+ if cross_filter:
+ columns = _add_deck_columns(columns, [cross_filter])
+ columns = _add_deck_columns(columns, tooltips)
+ if form_data.get("filter_nulls", True) and isinstance(geometry, str):
+ add_null(geometry)
+ query.update(
+ columns=columns,
+ metrics=[],
+ groupby=[],
+ filters=filters,
+ is_timeseries=False,
+ )
+ return query
+
+ if viz_type == "deck_polygon":
+ line_column = form_data.get("line_column")
+ if not isinstance(line_column, str) or not line_column:
+ raise ValueError("Polygon column is required for Polygon charts")
+ columns = _add_deck_columns(base_columns, [line_column] if line_column else [])
+ cross_filter = form_data.get("cross_filter_column")
+ if cross_filter:
+ columns = _add_deck_columns(columns, [cross_filter])
+ columns = _add_deck_columns(columns, tooltips)
+ metrics: list[Any] = []
+ if metric := form_data.get("metric"):
+ metrics.append(metric)
+ radius = form_data.get("point_radius_fixed")
+ if (
+ isinstance(radius, dict)
+ and radius.get("type") == "metric"
+ and radius.get("value") is not None
+ ):
+ metrics.append(radius["value"])
+ if form_data.get("filter_nulls", True) and isinstance(line_column, str):
+ add_null(line_column)
+ if metric:
+ add_null(_deck_metric_label(metric))
+ query.update(
+ columns=columns,
+ metrics=metrics,
+ filters=filters,
+ is_timeseries=False,
+ )
+ return query
+
+ if viz_type == "deck_path":
+ line_column = form_data.get("line_column")
+ if not isinstance(line_column, str) or not line_column:
+ raise ValueError("Line column is required for Path charts")
+ columns = list(base_columns)
+ metrics = list(base_metrics)
Review Comment:
For `deck_path`, `base_metrics` includes the aliased `size` value verbatim, so a saved chart with fixed `size="100"` emits a metric literally named `100` and changes a raw query into an aggregate. Can this filter scalar size values here as the scatter path does and keep the regression test aligned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5613893550
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5660116184
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5510715397
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5758240551
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5677210531
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4206811004
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -2158,6 +3048,56 @@ def map_mixed_timeseries_config(
_add_adhoc_filters(form_data, config.filters)
+ fields_set = config.model_fields_set
+ if "adhoc_filters_b" in fields_set:
+ form_data["adhoc_filters_b"] = config.adhoc_filters_b
+ elif "filters_secondary" in fields_set:
+ form_data["adhoc_filters_b"] = [
Review Comment:
Fixed in 31bff3652ae76ff7db15a230ad8eeeb9563d2fdd.
Mixed secondary filters now participate in column-reference extraction, the same physical-column filter validation as primary filters, and canonical-name normalization before adhoc_filters_b mapping. test_mixed_secondary_filters_validate_and_normalize rejects regoin and a saved-metric-only name, normalizes REGION to region, and checks the resulting Query B filter without leaking it into Query A. All three cases failed before and pass after.
Validation: MCP/common unit suites: 9,198 passed, 4 skipped; touched modules: 343 passed plus 351 Sunburst tests passed. Pre-commit passed on all 83 branch-changed files, including mypy.
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -2627,6 +3810,37 @@ class XYChartConfig(BaseChartConfig):
le=10000,
)
+ series_limit_metric: ColumnRef | None = Field(
+ None,
+ description="Metric used to rank limited series; null clears it",
+ )
+ timeseries_limit_metric: ColumnRef | None = Field(
+ None,
+ description="Native timeseries ranking metric; null clears it",
+ )
+
+ @field_validator("series_limit_metric", "timeseries_limit_metric", mode="before")
+ @classmethod
+ def coerce_series_ranking_metric(cls, value: Any) -> Any:
+ """Accept the same native metric references as the Y-axis."""
+ if isinstance(value, dict) and "expressionType" in value:
Review Comment:
Fixed in 31bff3652ae76ff7db15a230ad8eeeb9563d2fdd.
Both series_limit_metric and timeseries_limit_metric pass saved-metric strings through _coerce_native_metric, just like native metrics entries. test_xy_native_saved_ranking_metric_round_trips covers both spellings through the native GenerateChartRequest payload, verifies saved_metric semantics, and maps the ranking metric back to count. Both cases failed before and pass after; dimension-reference rejection remains covered.
Validation: MCP/common unit suites: 9,198 passed, 4 skipped; touched modules: 343 passed plus 351 Sunburst tests passed. Pre-commit passed on all 83 branch-changed files, including mypy.
##########
tests/unit_tests/mcp_service/chart/test_preview_utils.py:
##########
@@ -154,11 +172,41 @@ def test_build_query_columns_empty_columns_key_keeps_groupby():
) == ["country"]
+def test_generate_preview_seeds_form_data_before_query_execution():
+ """Preview execution seeds the form data consumed by virtual-dataset Jinja."""
+ with (
+ patch(
+ "superset.charts.data.form_data.set_query_context_form_data"
+ ) as mock_set_form_data,
+ patch(
+ "superset.commands.chart.data.get_data_command.ChartDataCommand"
+ ) as mock_cmd_cls,
+ patch(
+ "superset.common.query_context_factory.QueryContextFactory"
+ ) as mock_factory,
+ patch("superset.extensions.db") as mock_db,
+ ):
+ mock_db.session.get.return_value = MagicMock(id=12)
+ query_context = MagicMock()
+ mock_factory.return_value.create.return_value = query_context
+ mock_cmd_cls.return_value.run.return_value = chart_data_command_result(
+ rows=[], columns=["value"]
+ )
+
+ preview_utils.generate_preview_from_form_data(
+ form_data={"metrics": [{"label": "count"}]},
+ dataset_id=12,
+ preview_format="table",
+ )
+
+ mock_set_form_data.assert_called_once_with(query_context, 12, "table")
Review Comment:
Fixed in 31bff3652ae76ff7db15a230ad8eeeb9563d2fdd.
The unsaved-preview regression no longer mocks the seeder. At ChartDataCommand construction it reads the real ExtraCache.filter_values("region") and url_param("tenant") macros, checks EMEA/acme instead of deliberately stale inputs, and then asserts a successful one-row TablePreview containing count=2. The query-builder/command boundaries remain isolated. As a mutation check, moving real seeding after command.run() makes this strengthened test fail; restoring the correct order passes.
Validation: MCP/common unit suites: 9,198 passed, 4 skipped; touched modules: 343 passed plus 351 Sunburst tests passed. Pre-commit passed on all 83 branch-changed files, including mypy.
##########
docs/docs/using-superset/using-ai-with-superset.mdx:
##########
@@ -461,6 +472,24 @@ also have `data: null`. For each statement, `executed_sql` is `null` when it equ
- Restart your AI client if you recently changed the configuration
+### Sunburst chart data
+
+MCP chart tools support ordered Sunburst hierarchies with a primary metric and
+an optional secondary metric. When supplied, the secondary-to-primary ratio
+drives a sequential color scale (for example, profit divided by revenue shows
+margin); when omitted, arc colors are categorical. Use `chart_type: "sunburst"`
+in typed requests; the native chart uses `viz_type: "sunburst_v2"`.
+
+SQL NULL aggregate values are rendered as zero, matching the Sunburst frontend,
+including during compile checks, previews, and data exports. Missing result
+columns, numeric strings, booleans, and infinite metric values remain invalid.
Review Comment:
Fixed in 31bff3652ae76ff7db15a230ad8eeeb9563d2fdd.
The Sunburst documentation explicitly describes non-finite floating-point metrics becoming null and then zero during compile checks, previews, and exports, and warns that an overflowing aggregate can appear as zero rather than a validation error. Runtime normalization is unchanged. test_sunburst_docs_warn_about_nonfinite_metric_normalization fails against the previous documentation and passes with this wording; the existing compile boundary tests also pass.
Validation: MCP/common unit suites: 9,198 passed, 4 skipped; touched modules: 343 passed plus 351 Sunburst tests passed. Pre-commit passed on all 83 branch-changed files, including mypy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5867594549
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5497540597
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/43771?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `76.59574%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 79.21%. Comparing base ([`4395640`](https://app.codecov.io/gh/apache/superset/commit/439564041f54cafbc17f2f995c682c65b8bbbaab?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fb0a0e3`](https://app.codecov.io/gh/apache/superset/commit/fb0a0e3bd3ca6657c03dd1d860f15e01c73b75ba?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/43771?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/mcp\_service/chart/schemas.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fmcp_service%2Fchart%2Fschemas.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWNwX3NlcnZpY2UvY2hhcnQvc2NoZW1hcy5weQ==) | 80.00% | [15 Missing and 13 partials :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/mcp\_service/chart/plugins/sunburst.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fmcp_service%2Fchart%2Fplugins%2Fsunburst.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWNwX3NlcnZpY2UvY2hhcnQvcGx1Z2lucy9zdW5idXJzdC5weQ==) | 70.90% | [12 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/mcp\_service/chart/preview\_utils.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fmcp_service%2Fchart%2Fpreview_utils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWNwX3NlcnZpY2UvY2hhcnQvcHJldmlld191dGlscy5weQ==) | 68.62% | [9 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/mcp\_service/chart/chart\_helpers.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fmcp_service%2Fchart%2Fchart_helpers.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWNwX3NlcnZpY2UvY2hhcnQvY2hhcnRfaGVscGVycy5weQ==) | 60.00% | [2 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/mcp\_service/chart/chart\_utils.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fmcp_service%2Fchart%2Fchart_utils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWNwX3NlcnZpY2UvY2hhcnQvY2hhcnRfdXRpbHMucHk=) | 90.24% | [2 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/mcp\_service/chart/tool/get\_chart\_data.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fmcp_service%2Fchart%2Ftool%2Fget_chart_data.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWNwX3NlcnZpY2UvY2hhcnQvdG9vbC9nZXRfY2hhcnRfZGF0YS5weQ==) | 0.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...set/mcp\_service/chart/tool/update\_chart\_preview.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fmcp_service%2Fchart%2Ftool%2Fupdate_chart_preview.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWNwX3NlcnZpY2UvY2hhcnQvdG9vbC91cGRhdGVfY2hhcnRfcHJldmlldy5weQ==) | 33.33% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/common/form\_data\_query\_context.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fcommon%2Fform_data_query_context.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbW9uL2Zvcm1fZGF0YV9xdWVyeV9jb250ZXh0LnB5) | 75.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...erset/mcp\_service/chart/resources/chart\_configs.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fmcp_service%2Fchart%2Fresources%2Fchart_configs.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWNwX3NlcnZpY2UvY2hhcnQvcmVzb3VyY2VzL2NoYXJ0X2NvbmZpZ3MucHk=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/chart/validation/runtime/chart\_type\_suggester.py](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&filepath=superset%2Fmcp_service%2Fchart%2Fvalidation%2Fruntime%2Fchart_type_suggester.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWNwX3NlcnZpY2UvY2hhcnQvdmFsaWRhdGlvbi9ydW50aW1lL2NoYXJ0X3R5cGVfc3VnZ2VzdGVyLnB5) | 66.66% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/43771?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #43771 +/- ##
===========================================
+ Coverage 57.45% 79.21% +21.75%
===========================================
Files 2889 2890 +1
Lines 166762 167071 +309
Branches 38604 38681 +77
===========================================
+ Hits 95818 132343 +36525
+ Misses 70047 32184 -37863
- Partials 897 2544 +1647
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/43771/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage ฮ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/43771/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.77% <21.27%> (-0.06%)` | :arrow_down: |
| [presto](https://app.codecov.io/gh/apache/superset/pull/43771/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.66% <21.27%> (-0.07%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/43771/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.62% <76.59%> (+43.84%)` | :arrow_up: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/43771/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.05% <21.27%> (?)` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/43771/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.28% <76.59%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/43771?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5529425222
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4191416874
##########
superset/mcp_service/utils/serialization.py:
##########
@@ -170,24 +179,30 @@ def _sanitize_other(value: Any, depth: int) -> Any:
# FastMCP's ``fallback=str``, which the structured-content path
# (``to_jsonable_python``) does not apply.
return _sanitize_str(str(value))
- return sanitize_json_value(converted, depth + 1)
+ return sanitize_json_value(
+ converted, depth + 1, preserve_decimals=preserve_decimals
+ )
-def sanitize_json_value(value: Any, depth: int = 0) -> Any:
+def sanitize_json_value(
+ value: Any, depth: int = 0, *, preserve_decimals: bool = False
+) -> Any:
"""Return ``value`` with anything ``pydantic_core.to_json`` cannot encode
replaced by a JSON-safe equivalent."""
+ if preserve_decimals and type(value) is Decimal:
Review Comment:
Keeping exact `Decimal`s changes the wire format for every chart type that uses `ExactJsonSafeRows`: a Table or Pie cell of `Decimal("12.50")` previously came back from `get_chart_data` as the number `12.5` and now comes back as the string `"12.50"` (Pydantic's JSON mode), in `data`, `query_results[].data` and the column samples, while `schema_version` stays `"2.0"`. Clients that sum or chart those values will receive strings, and `query_dataset`/`get_table` rows still return numbers.
Is the string form intentional for existing chart types, and if so should it be called out in the docs/schema version, or should finite Decimals keep serializing as numbers?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1272 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": form_data.get("time_grain_sqla"),
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ query.setdefault("metrics", []).append(sort_metric)
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ )
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
+ extras = query.get("extras")
+ if isinstance(extras, dict):
+ extras.pop("time_grain_sqla", None)
+ if not extras:
+ query.pop("extras", None)
+ elif not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
query["post_processing"] = post_processing
- # ``granularity`` does two jobs downstream: it names the temporal column the
- # time range filters on, and it is the column ``time_grain_sqla`` buckets
- # (``models/helpers.py`` swaps a selected column for its timestamp expression
- # when that column equals ``granularity``). Only the first job depends on
- # there being an active range, so set it whenever form data carries one โ
- # matching ``extractExtras.ts``, which sets it unconditionally. Gating it on
- # ``time_range`` dropped the bucketing, so an ordinary "all-time totals by
- # month" chart exported one row per raw timestamp instead of one per month.
- if granularity := form_data.get("granularity") or form_data.get("granularity_sqla"):
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
Review Comment:
This branch is taken for every non-download rebuild, but `get_chart_data` never sets `result_format` on the form data it passes in. A cached Table with `server_pagination=true`, `server_page_length=10`, `row_limit=100` and 100 result rows, called as `get_chart_data(form_data_key=..., format="csv", limit=100)`, queries only 10 rows (and adds an `is_rowcount` query), so the CSV/Excel export is just the first page. The same branch also replaces the caller's `row_limit` with the saved page size, so `limit=1` against `server_page_length=1000` still fetches 1,000 groups.
Should the MCP rebuild treat the requested export format and explicit `limit` as non-paginated, the way the Excel-export caller now does?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,777 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ requested_filters = getattr(config, "filters", None)
+ if requested_filters is not None:
+ # Legacy ``filters``/``where``/``having`` are reconstructed into adhoc
+ # filters at query time, so an explicit clear or replacement removes
+ # every saved source the new form data does not set itself.
+ for key in _SAVED_PREDICATE_FORM_DATA_KEYS - new_form_data.keys():
+ merged.pop(key, None)
+ else:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool(
+ {"temporal_column", "time_grain", "time_range"} & fields_set
+ ),
+ )
+ if filters is not None:
+ merged["adhoc_filters"] = filters
+ return merged
+
+
+def retain_mixed_timeseries_secondary_update_state(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Return saved query-B controls a same-viz Mixed Timeseries update keeps.
+
+ Query B inherits unsuffixed controls only when the suffixed key is absent.
+ Preserve explicit native clears for controls the typed mapper did not
+ replace, so []/None never turns into accidental inheritance from query A.
+ Valid comparison state is also retained; malformed/stale dataset roles
+ remain fail-closed and are dropped.
+ """
+ from superset.common.form_data_query_context import (
+ MIXED_TIMESERIES_SECONDARY_QUERY_KEYS,
+ )
+
+ retained: Dict[str, Any] = {}
+ for key in MIXED_TIMESERIES_SECONDARY_QUERY_KEYS:
Review Comment:
Every `*_b` key is in `query_role_keys` and is stripped on a same-viz update, but this loop only restores explicit clears, `comparison_type_b`, and three list controls. Saved `time_range_b`, `row_limit_b`, `time_grain_sqla_b`, `rolling_*_b` and similar are lost whenever the update doesn't restate them. For example, a Mixed Timeseries with `time_range="No filter"` and `time_range_b="2025-01-01 : 2025-02-01"` updated with only `show_legend=false` ends up with no `time_range_b`, so query B inherits the unrestricted primary range and returns different data.
Should valid secondary-query controls the request didn't mention be preserved the way the primary query's are?
##########
tests/unit_tests/mcp_service/chart/test_native_form_data_and_dataset_rebind_contracts.py:
##########
@@ -0,0 +1,776 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from copy import deepcopy
+from unittest.mock import MagicMock, patch
+
+import pytest
+from pydantic import ValidationError
+
+from superset.mcp_service.chart.chart_helpers import (
+ build_query_context_from_form_data,
+ build_query_dicts_from_form_data,
+)
+from superset.mcp_service.chart.chart_utils import (
+ map_config_to_form_data,
+ merge_form_data_for_update,
+ scrub_dataset_bound_form_data,
+)
+from superset.mcp_service.chart.schemas import (
+ GenerateChartRequest,
+ SunburstChartConfig,
+ TableChartConfig,
+ UpdateChartPreviewRequest,
+ UpdateChartRequest,
+)
+
+
+@pytest.mark.parametrize(
+ "form_data,expected_columns,expected_metrics,expected_queries",
+ [
+ (
+ {
+ "viz_type": "histogram_v2",
+ "column": "revenue",
+ "groupby": ["region"],
+ "bins": 8,
+ },
+ ["region", "revenue"],
+ [],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "pivot_table_v2",
+ "groupbyRows": ["region"],
+ "groupbyColumns": ["product"],
+ "metrics": ["sum_revenue"],
+ },
+ ["region", "product"],
+ ["sum_revenue"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "waterfall",
+ "x_axis": "month",
+ "groupby": ["region"],
+ "metric": "sum_revenue",
+ },
+ ["month", "region"],
+ ["sum_revenue"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "gantt_chart",
+ "start_time": "started_at",
+ "end_time": "ended_at",
+ "y_axis": "task",
+ "series": ["team"],
+ "tooltip_columns": ["owner"],
+ "tooltip_metrics": ["duration"],
+ "order_by_cols": ['["started_at", true]'],
+ },
+ ["started_at", "ended_at", "task", "team", "owner"],
+ ["duration"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "mixed_timeseries",
+ "x_axis": "ds",
+ "groupby": ["region"],
+ "metrics": ["revenue"],
+ "groupby_b": ["product"],
+ "metrics_b": ["profit"],
+ },
+ ["ds", "region"],
+ ["revenue"],
+ 2,
+ ),
+ (
+ {
+ "viz_type": "table",
+ "query_mode": "raw",
+ "all_columns": ["region", "revenue"],
+ "order_by_cols": ['["revenue", false]'],
+ },
+ ["region", "revenue"],
+ [],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "echarts_timeseries_line",
+ "x_axis": "ds",
+ "groupby": ["region"],
+ "metrics": ["revenue"],
+ "series_limit": 5,
+ "series_limit_metric": "revenue",
+ "order_desc": False,
+ },
+ ["ds", "region"],
+ ["revenue"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "box_plot",
+ "columns": ["revenue"],
+ "groupby": ["region"],
+ "metrics": ["avg_revenue"],
+ "whiskerOptions": "Tukey",
+ },
+ ["revenue", "region"],
+ ["avg_revenue"],
+ 1,
+ ),
+ (
+ {
+ "viz_type": "ag-grid-pivot-table",
+ "groupby": ["region", "product"],
+ "metrics": ["revenue"],
+ },
+ ["region", "product"],
+ ["revenue"],
+ 1,
+ ),
+ ],
+ ids=[
+ "histogram",
+ "pivot",
+ "waterfall",
+ "gantt",
+ "mixed",
+ "raw-table",
+ "xy",
+ "box-plot",
+ "interactive-pivot",
+ ],
+)
+def test_product_query_context_uses_registered_frontend_adapters(
+ form_data: dict[str, object],
+ expected_columns: list[object],
+ expected_metrics: list[object],
+ expected_queries: int,
+) -> None:
+ """Exercise production QueryObject dictionaries at the factory boundary."""
+ factory = MagicMock()
+ factory.create.return_value = object()
+ query_form_data = {"datasource": "7__table", **deepcopy(form_data)}
+ with (
+ patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ),
+ patch(
+ "superset.common.query_context_factory.QueryContextFactory",
+ return_value=factory,
+ ),
+ ):
+ build_query_context_from_form_data(query_form_data)
+
+ queries = factory.create.call_args.kwargs["queries"]
+ assert len(queries) == expected_queries
+ if (
+ isinstance(form_data.get("x_axis"), str)
+ and form_data["x_axis"] in expected_columns
+ ):
+ expected_columns = [
+ {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": form_data["x_axis"],
+ "label": form_data["x_axis"],
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ if column == form_data["x_axis"]
+ else column
+ for column in expected_columns
+ ]
+ assert queries[0]["columns"] == expected_columns
+ assert queries[0]["metrics"] == expected_metrics
+
+ if form_data["viz_type"] == "histogram_v2":
+ assert queries[0]["post_processing"][0]["operation"] == "histogram"
+ if form_data["viz_type"] == "box_plot":
+ assert queries[0]["post_processing"][0]["operation"] == "boxplot"
+ if form_data["viz_type"] == "echarts_timeseries_line":
+ assert queries[0]["series_columns"] == ["region"]
+ assert queries[0]["orderby"] == [["revenue", True]]
+ assert [rule["operation"] for rule in queries[0]["post_processing"]] == [
+ "pivot",
+ "flatten",
+ ]
+ if form_data["viz_type"] == "waterfall":
+ assert queries[0]["orderby"] == [["month", True], ["region", True]]
+ if form_data["viz_type"] == "gantt_chart":
+ assert queries[0]["series_columns"] == ["team"]
+ assert queries[0]["orderby"] == [["started_at", True]]
+ if form_data["viz_type"] == "mixed_timeseries":
+ assert queries[1]["columns"] == [
+ {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": "ds",
+ "label": "ds",
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ },
+ "product",
+ ]
+ assert queries[1]["metrics"] == ["profit"]
+
+
+@pytest.mark.parametrize("secondary_key", ["adhoc_filters_b", "filters_b"])
+def test_mixed_secondary_explicit_filter_clear_never_inherits_primary(
+ secondary_key: str,
+) -> None:
+ primary_filter = {
+ "clause": "WHERE",
+ "expressionType": "SIMPLE",
+ "subject": "region",
+ "operator": "==",
+ "comparator": "EMEA",
+ }
+ form_data = {
+ "viz_type": "mixed_timeseries",
+ "x_axis": "ds",
+ "metrics": ["revenue"],
+ "metrics_b": ["profit"],
+ "adhoc_filters": [primary_filter],
+ secondary_key: [],
+ "time_range": "Last year",
+ "time_range_b": None,
+ }
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ primary, secondary = build_query_dicts_from_form_data(form_data, 7, "table")
+
+ assert primary["filters"] == [{"col": "region", "op": "==", "val": "EMEA"}]
+ assert secondary["filters"] == []
+ assert secondary.get("time_range") is None
+
+
+@pytest.mark.parametrize(
+ "viz_type,roles",
+ [
+ ("table", {"all_columns": ["old"], "order_by_cols": ['["old", true]']}),
+ ("pivot_table_v2", {"groupbyRows": ["old"], "groupbyColumns": ["old2"]}),
+ ("histogram_v2", {"column": "old", "groupby": ["old2"]}),
+ ("waterfall", {"x_axis": "old", "groupby": ["old2"]}),
+ ("echarts_timeseries_line", {"x_axis": "old", "series_columns": ["old2"]}),
+ ("mixed_timeseries", {"metrics_b": ["old"], "adhoc_filters_b": []}),
+ ("gantt_chart", {"start_time": "old", "tooltip_columns": ["old2"]}),
+ ("world_map", {"entity": "old", "metric": "old_metric"}),
+ ("deck_scatter", {"spatial": {"type": "latlong", "lonCol": "old"}}),
+ ],
+)
+def test_dataset_rebind_scrubs_complete_query_role_contracts(
+ viz_type: str, roles: dict[str, object]
+) -> None:
+ scrubbed = scrub_dataset_bound_form_data(
+ {"viz_type": viz_type, **roles, "show_legend": True}
+ )
+ assert set(roles).isdisjoint(scrubbed)
+ assert scrubbed == {"viz_type": viz_type, "show_legend": True}
+
+
+def test_dataset_rebind_fails_closed_without_a_complete_viz_contract() -> None:
+ with pytest.raises(ValueError, match="no complete dataset role contract"):
+ scrub_dataset_bound_form_data(
+ {"viz_type": "third_party_unknown", "mystery_column": "old"}
+ )
+
+
+_NATIVE_METRICS = [
+ "saved_revenue",
+ {
+ "expressionType": "SIMPLE",
+ "column": {"column_name": "revenue"},
+ "aggregate": "SUM",
+ "label": "SUM(revenue)",
+ "hasCustomLabel": False,
+ "optionName": "metric_revenue",
+ "datasourceWarning": False,
+ "sqlExpression": None,
+ },
+ {
+ "expressionType": "SQL",
+ "column": None,
+ "aggregate": None,
+ "sqlExpression": "COUNT(*)",
+ "label": "Count",
+ "hasCustomLabel": True,
+ "optionName": "metric_count",
+ "datasourceWarning": False,
+ },
+]
+
+
+@pytest.mark.parametrize("request_type", ["generate", "update", "preview"])
+@pytest.mark.parametrize("metric", _NATIVE_METRICS, ids=["saved", "simple", "sql"])
+def test_native_xy_form_data_round_trips_through_every_request_model(
+ request_type: str, metric: object
+) -> None:
+ config = {
+ "viz_type": "echarts_timeseries_bar",
+ "x_axis": "ds",
+ "x_axis_title": "Date",
+ "x_axis_format": "smart_date",
+ "y_axis_title": "Revenue",
+ "y_axis_scale": "log",
+ "metrics": [metric],
+ "groupby": ["region"],
+ "row_limit": 123,
+ }
+ payload: dict[str, object]
+ request_model: type[
+ GenerateChartRequest | UpdateChartRequest | UpdateChartPreviewRequest
+ ]
+ if request_type == "generate":
+ request_model = GenerateChartRequest
+ payload = {"dataset_id": 7, "config": config}
+ elif request_type == "update":
+ request_model = UpdateChartRequest
+ payload = {"identifier": 19, "config": config}
+ else:
+ request_model = UpdateChartPreviewRequest
+ payload = {"dataset_id": 7, "config": config}
+
+ request = request_model.model_validate(payload)
+ xy = request.config
+ assert xy is not None
+ assert xy.x is not None
+ assert xy.x.name == "ds"
+ assert xy.x_axis is not None
+ assert xy.x_axis.title == "Date"
+ assert xy.y_axis is not None
+ assert xy.y_axis.scale == "log"
+
+ with patch(
+ "superset.mcp_service.chart.chart_utils.is_column_truly_temporal",
+ return_value=True,
+ ):
+ mapped = map_config_to_form_data(xy, dataset_id=7)
+ assert mapped["x_axis"] == "ds"
+ assert mapped["x_axis_title"] == "Date"
+ assert mapped["x_axis_format"] == "smart_date"
+ assert mapped["metrics"][0]
+
+
+def test_native_xy_object_axis_config_remains_presentation_state() -> None:
+ request = GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "viz_type": "echarts_timeseries_line",
+ "x": {"name": "ds"},
+ "x_axis": {"title": "Date", "format": "smart_date"},
+ "metrics": ["saved_revenue"],
+ },
+ }
+ )
+ assert request.config.x is not None
+ assert request.config.x.name == "ds"
+ assert request.config.x_axis is not None
+ assert request.config.x_axis.title == "Date"
+
+
+@pytest.mark.parametrize(
+ "x_axis",
+ [
+ {"title": "Date", "formt": "smart_date"},
+ {"column_name": "ds", "column_nmae": "typo"},
+ {
+ "expressionType": "SIMPLE",
+ "column": {"column_name": "ds", "column_nmae": "typo"},
+ },
+ ],
+)
+def test_native_xy_rejects_malformed_nested_axis_state(x_axis: object) -> None:
+ with pytest.raises(ValidationError, match="Unknown"):
+ GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "viz_type": "echarts_timeseries_line",
+ "x_axis": x_axis,
+ "metrics": ["saved_revenue"],
+ },
+ }
+ )
+
+
+def test_native_xy_explicit_axis_reset_survives_sparse_merge() -> None:
+ request = UpdateChartRequest.model_validate(
+ {
+ "identifier": 19,
+ "config": {
+ "viz_type": "echarts_timeseries_line",
+ "x_axis": "ds",
+ "x_axis_title": None,
+ "metrics": ["saved_revenue"],
+ },
+ }
+ )
+ assert request.config is not None
+ with patch(
+ "superset.mcp_service.chart.chart_utils.is_column_truly_temporal",
+ return_value=True,
+ ):
+ mapped = map_config_to_form_data(request.config, dataset_id=7)
+ merged = merge_form_data_for_update(
+ {
+ "viz_type": "echarts_timeseries_line",
+ "x_axis": "old_ds",
+ "metrics": ["old_metric"],
+ "x_axis_title": "Old title",
+ },
+ mapped,
+ request.config,
+ )
+ assert "x_axis_title" not in merged
+
+
+def test_mixed_typed_secondary_nulls_remain_explicit_query_clears() -> None:
+ request = GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "chart_type": "mixed_timeseries",
+ "x": {"name": "ds"},
+ "y": [{"name": "revenue", "aggregate": "SUM"}],
+ "y_secondary": [{"name": "profit", "aggregate": "SUM"}],
+ "adhoc_filters_b": None,
+ "time_range_b": None,
+ "annotation_layers_b": [],
+ },
+ }
+ )
+ with patch(
+ "superset.mcp_service.chart.chart_utils.is_column_truly_temporal",
+ return_value=True,
+ ):
+ form_data = map_config_to_form_data(request.config, dataset_id=7)
+ form_data.update(
+ {
+ "adhoc_filters": [
+ {
+ "clause": "WHERE",
+ "expressionType": "SIMPLE",
+ "subject": "region",
+ "operator": "==",
+ "comparator": "EMEA",
+ }
+ ],
+ "time_range": "Last year",
+ }
+ )
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ _, secondary = build_query_dicts_from_form_data(form_data, 7, "table")
+ assert secondary["filters"] == []
+ assert "time_range" not in secondary
+ assert secondary["annotation_layers"] == []
+
+
+def test_mixed_secondary_receives_dashboard_extra_form_data() -> None:
+ form_data = {
+ "viz_type": "mixed_timeseries",
+ "x_axis": "ds",
+ "metrics": ["revenue"],
+ "metrics_b": ["profit"],
+ }
+ extra_form_data = {"filters": [{"col": "region", "op": "IN", "val": ["EMEA"]}]}
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ primary, secondary = build_query_dicts_from_form_data(
+ form_data, 7, "table", extra_form_data=extra_form_data
+ )
+ assert primary["filters"] == secondary["filters"]
+ assert secondary["filters"] == extra_form_data["filters"]
+
+
+def test_gantt_adapter_accepts_native_adhoc_axis_objects() -> None:
+ y_axis = {
+ "label": "Task",
+ "sqlExpression": "task_name",
+ "expressionType": "SQL",
+ }
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ query = build_query_dicts_from_form_data(
+ {
+ "viz_type": "gantt_chart",
+ "start_time": "started_at",
+ "end_time": "ended_at",
+ "y_axis": y_axis,
+ "series": "team",
+ "tooltip_metrics": ["duration"],
+ "order_by_cols": ['["started_at", true]'],
+ },
+ 7,
+ "table",
+ )[0]
+ assert query["columns"] == ["started_at", "ended_at", y_axis, "team"]
+ assert query["series_columns"] == ["team"]
+
+
+def test_pivot_non_additive_metrics_preserve_grouping_sets_contract() -> None:
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ query = build_query_dicts_from_form_data(
+ {
+ "viz_type": "pivot_table_v2",
+ "groupbyRows": ["region"],
+ "groupbyColumns": ["product"],
+ "metrics": ["saved_revenue"],
+ "rowTotals": True,
+ "colTotals": True,
+ },
+ 7,
+ "table",
+ )[0]
+ assert query["grouping_sets"] == [
+ [],
+ ["product"],
+ ["region"],
+ ["region", "product"],
+ ]
+
+
+def test_big_number_raw_aggregation_preserves_two_query_contract() -> None:
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ trend, overall = build_query_dicts_from_form_data(
+ {
+ "viz_type": "big_number",
+ "metric": "saved_ratio",
+ "granularity_sqla": "ds",
+ "aggregation": "raw",
+ },
+ 7,
+ "table",
+ )
+ assert trend["columns"] == []
+ assert trend["is_timeseries"] is True
+ assert trend["post_processing"][0]["options"]["index"] == ["__timestamp"]
+ assert [rule["operation"] for rule in trend["post_processing"]] == [
+ "pivot",
+ "flatten",
+ ]
+ assert overall["columns"] == []
+ assert overall["is_timeseries"] is False
+ assert overall["post_processing"] == []
+
+
+def _sunburst_replacement_config() -> SunburstChartConfig:
+ return SunburstChartConfig(
+ hierarchy=[{"name": "region"}, {"name": "country"}],
+ metric={"name": "sales", "aggregate": "SUM"},
+ )
+
+
+def test_cached_table_rebind_drops_legacy_sql_predicates() -> None:
+ """Top-level ``where``/``having`` reference the previous dataset."""
+ cached = {
+ "viz_type": "table",
+ "datasource": "10__table",
+ "query_mode": "aggregate",
+ "groupby": ["region"],
+ "metrics": ["count"],
+ "where": "region = 'EMEA'",
+ "having": "COUNT(*) > 5",
+ "show_cell_bars": True,
+ }
+ config = TableChartConfig(columns=[{"name": "region"}])
+ new_form_data = map_config_to_form_data(config)
+
+ merged = merge_form_data_for_update(
+ cached, new_form_data, config, dataset_rebind=True
+ )
+
+ assert "where" not in merged
+ assert "having" not in merged
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ (query,) = build_query_dicts_from_form_data(merged, 99, "table")
+ assert "region = 'EMEA'" not in str(query)
+ assert "COUNT(*) > 5" not in str(query)
+
+
+def test_cross_viz_rebind_from_unregistered_viz_uses_mapped_target() -> None:
+ """A complete config replaces a source viz that has no role contract."""
+ saved = {
+ "viz_type": "word_cloud",
+ "datasource": "10__table",
+ "series": "old_word",
+ "metric": "old_count",
+ "rotation": "square",
+ "adhoc_filters": [{"subject": "old_word"}],
+ "color_scheme": "supersetColors",
+ }
+ config = _sunburst_replacement_config()
+ new_form_data = map_config_to_form_data(config)
+
+ merged = merge_form_data_for_update(
+ saved, new_form_data, config, dataset_rebind=True
+ )
+
+ assert merged["viz_type"] == "sunburst_v2"
+ assert merged["columns"] == ["region", "country"]
+ assert merged["color_scheme"] == "supersetColors"
+ assert {"series", "rotation", "adhoc_filters"}.isdisjoint(merged)
+ assert "old_word" not in str(merged)
+ assert "old_count" not in str(merged)
+
+
+def test_cross_viz_dataset_only_scrub_still_requires_role_contract() -> None:
+ """Without a replacement viz, an unknown source still fails closed."""
+ with pytest.raises(ValueError, match="no complete dataset role contract"):
+ scrub_dataset_bound_form_data({"viz_type": "word_cloud", "series": "old"})
+
+
+def test_explicit_empty_filters_clear_legacy_predicates_on_cross_viz_update() -> None:
+ """``filters=[]`` removes adhoc, legacy structured, and free-form SQL state."""
+ saved = {
+ "viz_type": "table",
+ "datasource": "10__table",
+ "query_mode": "aggregate",
+ "groupby": ["region"],
+ "metrics": ["count"],
+ "adhoc_filters": [
+ {
+ "clause": "WHERE",
+ "expressionType": "SIMPLE",
+ "subject": "region",
+ "operator": "==",
+ "comparator": "EMEA",
+ }
+ ],
+ "filters": [{"col": "region", "op": "==", "val": "EMEA"}],
+ "where": "region = 'EMEA'",
+ "having": "COUNT(*) > 5",
+ }
+ config = SunburstChartConfig(
+ hierarchy=[{"name": "region"}, {"name": "country"}],
+ metric={"name": "sales", "aggregate": "SUM"},
+ filters=[],
+ )
+ new_form_data = map_config_to_form_data(config)
+
+ merged = merge_form_data_for_update(saved, new_form_data, config)
+
+ assert {"adhoc_filters", "filters", "where", "having"}.isdisjoint(merged)
+ with patch(
+ "superset.mcp_service.chart.chart_helpers.resolve_datasource_engine",
+ return_value="base",
+ ):
+ (query,) = build_query_dicts_from_form_data(merged, 10, "table")
+ assert query["filters"] == []
+ assert "EMEA" not in str(query)
+
+
+@pytest.mark.parametrize("chart_type", ["xy", "bar", "echarts_timeseries_bar"])
+def test_typed_xy_metrics_alias_keeps_column_semantics(chart_type: str) -> None:
+ """Only native Explore payloads read ``metrics`` strings as saved metrics."""
+ request = GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "chart_type": chart_type,
+ "x": {"name": "region"},
+ "metrics": ["revenue"],
+ },
+ }
+ )
+
+ assert request.config is not None
+ (metric,) = request.config.y
+ assert metric.name == "revenue"
+ assert metric.saved_metric is False
+ mapped = map_config_to_form_data(request.config, dataset_id=7)
+ (mapped_metric,) = mapped["metrics"]
+ assert mapped_metric["expressionType"] == "SIMPLE"
+ assert mapped_metric["aggregate"] == "SUM"
+ assert mapped_metric["column"]["column_name"] == "revenue"
+
+
+def test_native_xy_metrics_strings_remain_saved_metric_references() -> None:
+ request = GenerateChartRequest.model_validate(
+ {
+ "dataset_id": 7,
+ "config": {
+ "viz_type": "echarts_timeseries_bar",
+ "x_axis": "region",
+ "metrics": ["revenue"],
+ },
+ }
+ )
+
+ assert request.config is not None
+ (metric,) = request.config.y
+ assert metric.name == "revenue"
+ assert metric.saved_metric is True
+
+
+def test_replacement_filters_drop_legacy_predicates_on_cross_viz_update() -> None:
+ """Typed filters replace every saved predicate source, not only adhoc ones."""
+ saved = {
+ "viz_type": "table",
+ "datasource": "10__table",
+ "query_mode": "aggregate",
+ "groupby": ["region"],
+ "metrics": ["count"],
+ "filters": [{"col": "region", "op": "==", "val": "EMEA"}],
+ "extra_filters": [{"col": "region", "op": "in", "val": ["EMEA"]}],
+ "where": "region = 'EMEA'",
+ "having": "COUNT(*) > 5",
+ }
+ config = SunburstChartConfig(
+ hierarchy=[{"name": "region"}, {"name": "country"}],
+ metric={"name": "sales", "aggregate": "SUM"},
+ filters=[{"column": "region", "op": "=", "value": "APAC"}],
+ )
+ new_form_data = map_config_to_form_data(config)
+
+ merged = merge_form_data_for_update(saved, new_form_data, config)
+
+ assert {"filters", "extra_filters", "where", "having"}.isdisjoint(merged)
Review Comment:
This replacement test goes Table โ Sunburst, where the cross-viz allowlist already excludes `where`/`having`, so the assertions here pass even if `where` and `having` are removed from `_SAVED_PREDICATE_FORM_DATA_KEYS`. The same-viz path that the new `requested_filters is not None` branch actually serves has no case: update an aggregate Table on the same dataset that has saved `filters`, `where="region = 'EMEA'"` and `having="COUNT(*) > 5"` with `filters=[APAC]`, then assert via `merge_form_data_for_update` and the query builder that only APAC remains and neither SQL clause survives.
Could you add that same-viz case?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1272 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": form_data.get("time_grain_sqla"),
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ query.setdefault("metrics", []).append(sort_metric)
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ )
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
+ extras = query.get("extras")
+ if isinstance(extras, dict):
+ extras.pop("time_grain_sqla", None)
+ if not extras:
+ query.pop("extras", None)
+ elif not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
query["post_processing"] = post_processing
- # ``granularity`` does two jobs downstream: it names the temporal column the
- # time range filters on, and it is the column ``time_grain_sqla`` buckets
- # (``models/helpers.py`` swaps a selected column for its timestamp expression
- # when that column equals ``granularity``). Only the first job depends on
- # there being an active range, so set it whenever form data carries one โ
- # matching ``extractExtras.ts``, which sets it unconditionally. Gating it on
- # ``time_range`` dropped the bucketing, so an ordinary "all-time totals by
- # month" chart exported one row per raw timestamp instead of one per month.
- if granularity := form_data.get("granularity") or form_data.get("granularity_sqla"):
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
+ page_size = int(form_data.get("server_page_length") or 0)
+ configured_limit = int(form_data.get("row_limit") or 0)
+ query["row_limit"] = (
+ min(page_size, configured_limit) if configured_limit else page_size
+ )
+ query["row_offset"] = 0
+
+ queries = [query]
+ if form_data.get("percent_metric_calculation") == "all_records" and percent_metrics:
+ all_records = dict(query)
+ all_records.update(
+ {
+ "columns": [],
+ "metrics": percent_metrics,
+ "post_processing": [],
+ "row_limit": 0,
+ "row_offset": 0,
+ "orderby": [],
+ "is_timeseries": False,
+ }
+ )
+ queries.append(all_records)
+ if form_data.get("show_totals") and metrics:
+ totals_aggregate = form_data.get("totals_aggregate")
+ if totals_aggregate not in {"SUM", "AVG"}:
+ totals_aggregate = None
+ totals_metrics = [
+ {
+ **metric,
+ "aggregate": totals_aggregate,
+ }
+ if totals_aggregate
+ and isinstance(metric, dict)
+ and metric.get("expressionType") == "SIMPLE"
+ else metric
+ for metric in metrics
+ ]
+ totals = dict(query)
+ totals.update(
+ {
+ "columns": [],
+ "metrics": totals_metrics,
+ "row_limit": 0,
+ "row_offset": 0,
+ "post_processing": [contribution] if contribution else [],
+ }
+ )
+ totals.pop("order_desc", None)
+ totals.pop("orderby", None)
+ queries.append(totals)
+ if form_data.get("server_pagination") and not is_download:
+ rowcount = dict(query)
+ rowcount.update(
+ {
+ "time_offsets": [],
+ "row_limit": int(form_data.get("row_limit") or 0),
+ "row_offset": 0,
+ "post_processing": [],
+ "is_rowcount": True,
+ }
+ )
+ queries.insert(1, rowcount)
+ return queries
+
+
+def _treemap_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ """Apply Treemap's bounded metric and hierarchy ordering contract."""
+ hierarchy = _as_list(form_data.get("groupby"))
+ metric = form_data.get("metric")
+ query["columns"] = hierarchy
+ query["metrics"] = [metric] if metric is not None else []
+ granularity = form_data.get("granularity", form_data.get("granularity_sqla"))
+ if granularity is not None:
query["granularity"] = granularity
- if form_data.get("row_limit"):
- query["row_limit"] = form_data["row_limit"]
+ ordering = [[metric, False]] if form_data.get("sort_by_metric") and metric else []
+ ordering.extend(
+ [column, True] for column in hierarchy if isinstance(column, str) and column
+ )
+ try:
+ bounded = float(query.get("row_limit") or 0) != 0
+ except (ValueError, TypeError):
+ bounded = True
+ query.pop("orderby", None)
+ if bounded and ordering:
+ query["orderby"] = ordering
+
+
+def _bubble_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ """Preserve Bubble's entity/series roles and explicit sort-metric direction."""
+ query["columns"] = _deduplicate_fields(
+ [*_as_list(form_data.get("entity")), *_as_list(form_data.get("series"))]
+ )
+ query.pop("orderby", None)
+ if sort_metric := next(iter(_as_list(form_data.get("orderby"))), None):
+ query["orderby"] = [[sort_metric, not query.get("order_desc", True)]]
+
+
+def _entity_map_query(
+ form_data: dict[str, Any], query: dict[str, Any], *, world_map: bool
+) -> None:
+ """Group World/Country Map queries by their ``entity`` role.
+
+ Mirrors the World Map and Country Map ``buildQuery`` transforms: the
+ ``entity`` control is the only grouping column, Country Map selects one
+ ``metric``, and World Map adds a distinct ``secondary_metric`` and orders by
+ the primary metric only when ``sort_by_metric`` is enabled.
+ """
+ entity = form_data.get("entity")
+ query["columns"] = [entity] if entity else []
+ metric = form_data.get("metric")
+ metrics = [metric] if metric else []
+ if world_map:
+ secondary_metric = form_data.get("secondary_metric")
+ if secondary_metric and (
+ not metric
+ or _label(secondary_metric, metric=True) != _label(metric, metric=True)
+ ):
+ metrics.append(secondary_metric)
+ query.pop("orderby", None)
+ if form_data.get("sort_by_metric") and metric:
+ query["orderby"] = [[metric, False]]
+ query["metrics"] = metrics
+
+
+def build_query_objects_from_form_data( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ viz_type: str | None = None,
+ row_limit: int | None = None,
+ order_desc: bool | None = None,
+ filters_prepared: bool = False,
+ secondary_form_data: dict[str, Any] | None = None,
+) -> list[dict[str, Any]]:
+ """Build frontend-equivalent QueryObject dictionaries from form data.
+
+ Every real MCP product path and the legacy common fallback share this
+ extractor and the chart adapters below. This prevents compile/SQL/preview
+ and saved/fallback data paths from silently selecting different fields.
+ """
+ effective_viz = viz_type or str(form_data.get("viz_type") or "")
+ extraction_form_data = form_data
+ if is_raw_query_mode(form_data):
+ # ``extractQueryFields`` is strict, while the legacy server fallback has
+ # always kept old saved tables executable by dropping malformed order
+ # entries. Normalize that bounded raw-table input before invoking the
+ # shared extractor; valid entries still use the centralized contract.
+ extraction_form_data = {
+ **form_data,
+ "query_mode": "raw",
+ "order_by_cols": orderby_from_form_data(form_data, [], effective_viz),
+ }
+ query = _base_query_object(
+ extraction_form_data,
+ row_limit=row_limit,
+ order_desc=order_desc,
+ filters_prepared=filters_prepared,
+ )
+
+ if effective_viz in {"table", "ag-grid-table"}:
+ return _finalize_query_objects(form_data, _table_queries(form_data, query))
+ if effective_viz == "histogram_v2":
+ _histogram_query(form_data, query)
+ elif effective_viz == "box_plot":
+ _box_plot_query(form_data, query)
+ elif effective_viz == "pivot_table_v2":
+ _pivot_query(form_data, query)
+ elif effective_viz == "waterfall":
+ _waterfall_query(form_data, query)
+ elif effective_viz in {"gantt", "gantt_chart"}:
+ _gantt_query(form_data, query)
+ elif effective_viz.startswith("echarts_timeseries") or effective_viz in {
+ "echarts_area",
+ "mixed_timeseries",
+ }:
+ _timeseries_query(form_data, query)
+ elif effective_viz == "big_number":
+ return _finalize_query_objects(form_data, _big_number_queries(form_data, query))
+ elif effective_viz == "big_number_total":
+ query["columns"] = []
+ elif effective_viz == "handlebars":
+ _normalize_query_orderby(query)
+ elif effective_viz == "pie":
+ if form_data.get("sort_by_metric") and form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ query["post_processing"] = _pie_contribution_post_processing(
+ list(query.get("metrics") or [])
+ )
+ elif effective_viz == "gauge_chart":
+ if form_data.get("sort_by_metric") and form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ elif effective_viz == "treemap_v2":
+ _treemap_query(form_data, query)
+ elif effective_viz == "sunburst_v2":
+ if not form_data.get("sort_by_metric"):
+ query["orderby"] = []
+ elif form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ elif effective_viz in {"bubble", "bubble_v2"}:
+ _bubble_query(form_data, query)
+ elif effective_viz in {"world_map", "country_map"}:
+ _entity_map_query(form_data, query, world_map=effective_viz == "world_map")
+ elif effective_viz == "ag-grid-pivot-table":
+ query["columns"] = _temporalized_columns(
+ form_data, _as_list(form_data.get("groupby"))
+ )
+
+ if not effective_viz and not query.get("orderby"):
Review Comment:
The metric-ordering fallback only runs when `effective_viz` is empty. Any saved chart whose viz type has no branch above (Funnel, for example) now gets no `orderby` at all, whereas the previous `orderby_from_form_data(form_data, metrics, viz_type)` call applied to every type. A Funnel with `groupby=["stage"]`, `metric="revenue"`, `sort_by_metric=true`, `row_limit=2` rebuilt through the fallback `get_chart_data` or a preview now issues `LIMIT 2` with no `ORDER BY`, so it can return different stages than Explore.
Should this fallback also run when a viz type is set but no adapter handled ordering?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4191870218
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
Review Comment:
`build_query_dicts_from_form_data` runs `prepare_form_data_for_query` first, and its `merge_extra_filters` โ `merge_extra_form_data` pops `extra_form_data` from the form data. By the time this runs, a saved table with `time_compare=["inherit"]` and `extra_form_data={"time_compare": "1 year ago"}` has no `extra_form_data` left, so `time_offsets` comes out empty and the prior-year column is missing. The new inheritance test calls the common builder directly and skips that preparation step. Should the inherited offset be captured before the merge, with a test through `build_query_dicts_from_form_data`?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
Review Comment:
This resample operator never passes `fill_time_range`, which the frontend emits when `resample_fill_time_range` is true and which `QueryObject` turns into the range boundaries. For a Jan 1โ8 range with daily zero-fill and data only on Jan 3โ5, Explore returns Jan 1โ7 with zero edge buckets while this rebuild returns only Jan 3โ5; the Big Number copy at line 1336 has the same gap, so previews and fallback data disagree with the saved chart. Should both adapters forward `fill_time_range` when the form data sets it?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": form_data.get("time_grain_sqla"),
Review Comment:
This takes the Prophet `time_grain` only from `time_grain_sqla`. A saved line chart with `forecastEnabled=True` whose adhoc x-axis carries `timeGrain="P1M"` and no panel-level grain gets `time_grain=None`, so the forecast step fails with "Time grain missing" where Explore uses the axis grain. Should this fall back to the x-axis grain, and then to the frontend's daily default, with a forecast test?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
Review Comment:
The synthesized `BASE_AXIS` temporal column omits `isColumnReference: True`, which the native Table query supplies for semantic-layer datasources (the Waterfall path at line 1007 sets it). The semantic mapper then treats it as an adhoc dimension and rejects it, so a saved Table grouped by a temporal column with a grain fails in MCP preview/data fallback even though it works in Explore. Should this set `isColumnReference` like the other `BASE_AXIS` builders?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
Review Comment:
The shared builder now pivots grouped timeseries and flattens them to wide columns like `revenue, East` / `revenue, West`, but the Vega-Lite consumer still expects long-form rows (`preview_utils.py` around line 1711 picks one numeric column and only adds color when `groupby[0]` is a column). For `x_axis="ds"`, `metrics=["revenue"]`, `groupby=["region"]` the XY preview draws one region's line and drops the others without any error. Should the preview adapt to the wide shape (or melt it back) in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4202059805
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1305 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ extra_metric = sort_metric
+ else:
+ extra_metric = None
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ sort_metric=extra_metric,
+ )
+ if extra_metric is not None:
+ query.setdefault("metrics", []).append(extra_metric)
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
+ extras = query.get("extras")
+ if isinstance(extras, dict):
+ extras.pop("time_grain_sqla", None)
+ if not extras:
+ query.pop("extras", None)
+ elif not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
query["post_processing"] = post_processing
- # ``granularity`` does two jobs downstream: it names the temporal column the
- # time range filters on, and it is the column ``time_grain_sqla`` buckets
- # (``models/helpers.py`` swaps a selected column for its timestamp expression
- # when that column equals ``granularity``). Only the first job depends on
- # there being an active range, so set it whenever form data carries one โ
- # matching ``extractExtras.ts``, which sets it unconditionally. Gating it on
- # ``time_range`` dropped the bucketing, so an ordinary "all-time totals by
- # month" chart exported one row per raw timestamp instead of one per month.
- if granularity := form_data.get("granularity") or form_data.get("granularity_sqla"):
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
+ page_size = int(form_data.get("server_page_length") or 0)
+ configured_limit = int(form_data.get("row_limit") or 0)
+ query["row_limit"] = (
+ min(page_size, configured_limit) if configured_limit else page_size
+ )
+ query["row_offset"] = 0
+
+ queries = [query]
+ if form_data.get("percent_metric_calculation") == "all_records" and percent_metrics:
+ all_records = dict(query)
+ all_records.update(
+ {
+ "columns": [],
+ "metrics": percent_metrics,
+ "post_processing": [],
+ "row_limit": 0,
+ "row_offset": 0,
+ "orderby": [],
+ "is_timeseries": False,
+ }
+ )
+ queries.append(all_records)
+ if form_data.get("show_totals") and metrics:
+ totals_aggregate = form_data.get("totals_aggregate")
+ if totals_aggregate not in {"SUM", "AVG"}:
+ totals_aggregate = None
+ totals_metrics = [
+ {
+ **metric,
+ "aggregate": totals_aggregate,
+ }
+ if totals_aggregate
+ and isinstance(metric, dict)
+ and metric.get("expressionType") == "SIMPLE"
+ else metric
+ for metric in metrics
+ ]
+ totals = dict(query)
+ totals.update(
+ {
+ "columns": [],
+ "metrics": totals_metrics,
+ "row_limit": 0,
+ "row_offset": 0,
+ "post_processing": [contribution] if contribution else [],
+ }
+ )
+ totals.pop("order_desc", None)
+ totals.pop("orderby", None)
+ queries.append(totals)
+ if form_data.get("server_pagination") and not is_download:
+ rowcount = dict(query)
+ rowcount.update(
+ {
+ "time_offsets": [],
+ "row_limit": int(form_data.get("row_limit") or 0),
+ "row_offset": 0,
+ "post_processing": [],
+ "is_rowcount": True,
+ }
+ )
+ queries.insert(1, rowcount)
+ return queries
+
+
+def _treemap_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ """Apply Treemap's bounded metric and hierarchy ordering contract."""
+ hierarchy = _as_list(form_data.get("groupby"))
+ metric = form_data.get("metric")
+ query["columns"] = hierarchy
+ query["metrics"] = [metric] if metric is not None else []
+ granularity = form_data.get("granularity", form_data.get("granularity_sqla"))
+ if granularity is not None:
query["granularity"] = granularity
- if form_data.get("row_limit"):
- query["row_limit"] = form_data["row_limit"]
+ ordering = [[metric, False]] if form_data.get("sort_by_metric") and metric else []
+ ordering.extend(
+ [column, True] for column in hierarchy if isinstance(column, str) and column
+ )
+ try:
+ bounded = float(query.get("row_limit") or 0) != 0
+ except (ValueError, TypeError):
+ bounded = True
+ query.pop("orderby", None)
+ if bounded and ordering:
+ query["orderby"] = ordering
+
+
+def _bubble_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ """Preserve Bubble's entity/series roles and explicit sort-metric direction."""
+ query["columns"] = _deduplicate_fields(
+ [*_as_list(form_data.get("entity")), *_as_list(form_data.get("series"))]
+ )
+ query.pop("orderby", None)
+ if sort_metric := next(iter(_as_list(form_data.get("orderby"))), None):
+ query["orderby"] = [[sort_metric, not query.get("order_desc", True)]]
+
+
+def _entity_map_query(
+ form_data: dict[str, Any], query: dict[str, Any], *, world_map: bool
+) -> None:
+ """Group World/Country Map queries by their ``entity`` role.
+
+ Mirrors the World Map and Country Map ``buildQuery`` transforms: the
+ ``entity`` control is the only grouping column, Country Map selects one
+ ``metric``, and World Map adds a distinct ``secondary_metric`` and orders by
+ the primary metric only when ``sort_by_metric`` is enabled.
+ """
+ entity = form_data.get("entity")
+ query["columns"] = [entity] if entity else []
+ metric = form_data.get("metric")
+ metrics = [metric] if metric else []
+ if world_map:
+ secondary_metric = form_data.get("secondary_metric")
+ if secondary_metric and (
+ not metric
+ or _label(secondary_metric, metric=True) != _label(metric, metric=True)
+ ):
+ metrics.append(secondary_metric)
+ query.pop("orderby", None)
+ if form_data.get("sort_by_metric") and metric:
+ query["orderby"] = [[metric, False]]
+ query["metrics"] = metrics
+
+
+def build_query_objects_from_form_data( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ viz_type: str | None = None,
+ row_limit: int | None = None,
+ order_desc: bool | None = None,
+ filters_prepared: bool = False,
+ secondary_form_data: dict[str, Any] | None = None,
+) -> list[dict[str, Any]]:
+ """Build frontend-equivalent QueryObject dictionaries from form data.
+
+ Every real MCP product path and the legacy common fallback share this
+ extractor and the chart adapters below. This prevents compile/SQL/preview
+ and saved/fallback data paths from silently selecting different fields.
+ """
+ effective_viz = viz_type or str(form_data.get("viz_type") or "")
+ extraction_form_data = form_data
+ if is_raw_query_mode(form_data):
+ # ``extractQueryFields`` is strict, while the legacy server fallback has
+ # always kept old saved tables executable by dropping malformed order
+ # entries. Normalize that bounded raw-table input before invoking the
+ # shared extractor; valid entries still use the centralized contract.
+ extraction_form_data = {
+ **form_data,
+ "query_mode": "raw",
+ "order_by_cols": orderby_from_form_data(form_data, [], effective_viz),
+ }
+ query = _base_query_object(
+ extraction_form_data,
+ row_limit=row_limit,
+ order_desc=order_desc,
+ filters_prepared=filters_prepared,
+ )
+
+ if effective_viz in {"table", "ag-grid-table"}:
+ return _finalize_query_objects(form_data, _table_queries(form_data, query))
+ if effective_viz == "histogram_v2":
+ _histogram_query(form_data, query)
+ elif effective_viz == "box_plot":
+ _box_plot_query(form_data, query)
+ elif effective_viz == "pivot_table_v2":
+ _pivot_query(form_data, query)
+ elif effective_viz == "waterfall":
+ _waterfall_query(form_data, query)
+ elif effective_viz in {"gantt", "gantt_chart"}:
+ _gantt_query(form_data, query)
+ elif effective_viz.startswith("echarts_timeseries") or effective_viz in {
+ "echarts_area",
+ "mixed_timeseries",
+ }:
+ _timeseries_query(form_data, query)
+ elif effective_viz == "big_number":
+ return _finalize_query_objects(form_data, _big_number_queries(form_data, query))
+ elif effective_viz == "big_number_total":
+ query["columns"] = []
+ elif effective_viz == "handlebars":
+ _normalize_query_orderby(query)
+ elif effective_viz == "pie":
+ if form_data.get("sort_by_metric") and form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ query["post_processing"] = _pie_contribution_post_processing(
+ list(query.get("metrics") or [])
+ )
+ elif effective_viz == "gauge_chart":
+ if form_data.get("sort_by_metric") and form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ elif effective_viz == "treemap_v2":
+ _treemap_query(form_data, query)
+ elif effective_viz == "sunburst_v2":
+ if not form_data.get("sort_by_metric"):
+ query["orderby"] = []
+ elif form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ elif effective_viz in {"bubble", "bubble_v2"}:
+ _bubble_query(form_data, query)
+ elif effective_viz in {"world_map", "country_map"}:
+ _entity_map_query(form_data, query, world_map=effective_viz == "world_map")
+ elif effective_viz == "ag-grid-pivot-table":
+ query["columns"] = _temporalized_columns(
+ form_data, _as_list(form_data.get("groupby"))
+ )
+
+ elif not query.get("orderby"):
+ # Unadapted visualizations retain the legacy top-N ordering fallback.
+ # Handled adapters may deliberately leave ordering empty.
+ query["orderby"] = orderby_from_form_data(
+ form_data, list(query.get("metrics") or []), effective_viz
+ )
+
+ if effective_viz != "mixed_timeseries":
+ return _finalize_query_objects(form_data, [query])
+
+ secondary = secondary_form_data or retain_mixed_timeseries_secondary_form_data(
+ form_data
+ )
+ # Query A's saved ordering references metrics query B may not select, so the
+ # secondary series orders by ``orderby_b`` or not at all.
+ if (orderby_b := form_data.get("orderby_b")) is not None:
+ secondary["orderby"] = orderby_b
+ elif "orderby_b" not in form_data:
+ secondary.pop("orderby", None)
+ query_b = _base_query_object(
+ secondary,
+ row_limit=row_limit,
+ order_desc=order_desc,
Review Comment:
Fixed in f2acaab6defc45cd7680662feced6b74dae95d02.
Query B no longer receives Query A's order_desc override; its own order_desc_b wins. Regression: test_mixed_secondary_sort_direction_ignores_primary_override checks opposite directions with series_limit_b=1.
Validation: 851 tests passed across the 11 touched/related pytest modules; pre-commit passed for all changed files. The regression coverage fails against the previous implementation and passes with this fix.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1305 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ extra_metric = sort_metric
+ else:
+ extra_metric = None
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ sort_metric=extra_metric,
+ )
+ if extra_metric is not None:
+ query.setdefault("metrics", []).append(extra_metric)
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
Review Comment:
Fixed in f2acaab6defc45cd7680662feced6b74dae95d02.
The Big Number pivot label is derived from the same explicit_x_axis used to select the column. Regression: test_big_number_native_axis_pivot_references_selected_column covers the native column_name mapping with granularity_sqla and verifies the pivot index is ds, not __timestamp.
Validation: 851 tests passed across the 11 touched/related pytest modules; pre-commit passed for all changed files. The regression coverage fails against the previous implementation and passes with this fix.
##########
superset/mcp_service/chart/tool/update_chart.py:
##########
@@ -206,209 +219,66 @@ def _append_table_columns(
def _merge_replacement_config(
existing_form_data: dict[str, Any],
new_form_data: dict[str, Any],
- parsed_config: ChartConfig,
+ parsed_config: Any,
*,
dataset_rebind: bool = False,
) -> dict[str, Any]:
- """Delegate update semantics to the shared form-data merge helper."""
- return merge_chart_form_data(
+ """Merge a replacement config through the plugin-aware update merge.
+
+ An explicit empty filter list clears saved filters; plugins that generate
+ their own time binding (Gantt) keep it through that clear.
+ """
+ return merge_form_data_for_update(
existing_form_data,
new_form_data,
parsed_config,
dataset_rebind=dataset_rebind,
)
-def _valid_dataset_reference(
- value: Any,
- columns: set[str],
- metrics: set[str],
- *,
- allow_metric: bool = False,
-) -> bool:
- """Return whether a form_data reference resolves against a dataset."""
- if not isinstance(value, str):
- return True
- normalized = value.casefold()
- return normalized in columns or (allow_metric and normalized in metrics)
-
-
-def _inherited_metrics_match_dataset(
- existing_form_data: dict[str, Any],
- columns: set[str],
- metrics: set[str],
-) -> bool:
- for metric in existing_form_data.get("metrics") or []:
- if isinstance(metric, str) and not _valid_dataset_reference(
- metric, columns, metrics, allow_metric=True
- ):
- return False
- if isinstance(metric, dict):
- column = metric.get("column")
- if isinstance(column, dict) and not _valid_dataset_reference(
- column.get("column_name"), columns, metrics
- ):
- return False
- return True
-
-
-def _inherited_sort_matches_dataset(
- order_by_cols: Any, columns: set[str], metrics: set[str]
-) -> bool:
- for order_by in order_by_cols or []:
- try:
- column = json.loads(order_by)[0]
- except (TypeError, ValueError, IndexError):
- return False
- if not _valid_dataset_reference(column, columns, metrics, allow_metric=True):
- return False
- return True
-
-
-def _inherited_filters_match_dataset(
- filters: Any, columns: set[str], metrics: set[str]
-) -> bool:
- for filter_ in filters or []:
- if not isinstance(filter_, dict):
- return False
- if filter_.get("expressionType") not in (None, "SIMPLE"):
- return False
- subject = filter_.get("subject") or filter_.get("col")
- allow_metric = str(filter_.get("clause", "WHERE")).upper() == "HAVING"
- if not _valid_dataset_reference(
- subject, columns, metrics, allow_metric=allow_metric
- ):
- return False
- return True
-
-
-#: form_data keys carrying query roles, mapped to the config field that
-#: sets them explicitly. An explicit field is the caller's stated intent,
-#: so it is never treated as inherited state.
-_INHERITED_QUERY_ROLE_FIELDS = {
- "groupby": "group_by",
- "groupby_b": "group_by_secondary",
- "all_columns": None,
- "columns": None,
- "x_axis": None,
- "granularity_sqla": None,
- "metrics": None,
- "order_by_cols": "sort_by",
- "adhoc_filters": "filters",
-}
-
-_INHERITED_COLUMN_LIST_KEYS = frozenset(
- {"groupby", "groupby_b", "all_columns", "columns"}
-)
-_INHERITED_COLUMN_SCALAR_KEYS = frozenset({"x_axis", "granularity_sqla"})
+def _is_dataset_rebind(request: UpdateChartRequest, chart: Any) -> bool:
+ """Return whether a request changes the chart's datasource identity."""
+ return request.dataset_id is not None and str(request.dataset_id) != str(
+ getattr(chart, "datasource_id", None)
+ )
-def _inherited_state_invalid_keys(
- existing_form_data: dict[str, Any],
- new_form_data: dict[str, Any],
- parsed_config: ChartConfig,
- dataset_id: int,
-) -> set[str]:
- """Return inherited query fields that are invalid for a new dataset.
-
- A dataset rebind only has to discard the state that cannot resolve
- against the replacement dataset; everything else stays valid and is
- preserved so the update does not silently reset the chart.
- """
- fields_set = parsed_config.model_fields_set
- inherited_keys = {
- key
- for key, config_field in _INHERITED_QUERY_ROLE_FIELDS.items()
- if key not in new_form_data
- and config_field not in fields_set
- and existing_form_data.get(key)
- }
- if not inherited_keys:
- return set()
-
- from superset.daos.dataset import DatasetDAO
- from superset.mcp_service.chart.validation.dataset_validator import (
- build_dataset_context_from_orm,
+def _add_columns_rebind_error() -> GenerateChartResponse:
+ """Require a complete replacement config when changing table datasets."""
+ return _validation_error_response(
+ message="Cannot combine 'add_columns' with a dataset rebind.",
+ details=(
+ "Dataset-bound roles from the previous dataset cannot be reused. "
+ "Provide 'config' with the complete table configuration for the "
+ "target dataset."
+ ),
)
- context = build_dataset_context_from_orm(DatasetDAO.find_by_id(dataset_id))
- if context is None:
- # The replacement dataset cannot be inspected, so no inherited
- # reference can be shown to be safe.
- return inherited_keys
- columns = {column["name"].casefold() for column in context.available_columns}
- metrics = {metric["name"].casefold() for metric in context.available_metrics}
-
- invalid_keys: set[str] = set()
- for key in inherited_keys & _INHERITED_COLUMN_LIST_KEYS:
- values = existing_form_data.get(key)
- if isinstance(values, list) and not all(
- _valid_dataset_reference(value, columns, metrics) for value in values
- ):
- invalid_keys.add(key)
- for key in inherited_keys & _INHERITED_COLUMN_SCALAR_KEYS:
- if not _valid_dataset_reference(existing_form_data.get(key), columns, metrics):
- invalid_keys.add(key)
- if "metrics" in inherited_keys and not _inherited_metrics_match_dataset(
- existing_form_data, columns, metrics
- ):
- invalid_keys.add("metrics")
- if "order_by_cols" in inherited_keys and not _inherited_sort_matches_dataset(
- existing_form_data.get("order_by_cols"), columns, metrics
- ):
- invalid_keys.add("order_by_cols")
- if "adhoc_filters" in inherited_keys and not _inherited_filters_match_dataset(
- existing_form_data.get("adhoc_filters"), columns, metrics
- ):
- invalid_keys.add("adhoc_filters")
- return invalid_keys
-
-
-def _build_replacement_form_data(
- existing_form_data: dict[str, Any],
- parsed_config: ChartConfig,
- effective_dataset_id: int | None,
- replacement_dataset_id: int | None = None,
+
+def _build_dataset_rebind_payload(
+ request: UpdateChartRequest, chart: Any
) -> dict[str, Any]:
- """Map and merge a replacement config for preview and save paths."""
- new_form_data = map_config_to_form_data(
- parsed_config, dataset_id=effective_dataset_id, include_disabled=True
- )
- new_form_data.pop("_mcp_warnings", None)
- dataset_rebind = replacement_dataset_id is not None
- config_plugin = get_registry().get(parsed_config.chart_type, include_disabled=True)
- if replacement_dataset_id is not None and not (
- config_plugin is not None and config_plugin.strict_dataset_rebind
- ):
- # Drop only the inherited state the replacement dataset cannot
- # resolve, then merge as a same-dataset update. Plugins with a strict
- # rebind contract handle the rebind in merge_update_form_data.
- invalid_keys = _inherited_state_invalid_keys(
- existing_form_data,
- new_form_data,
- parsed_config,
- replacement_dataset_id,
- )
- existing_form_data = {
- key: value
- for key, value in existing_form_data.items()
- if key not in invalid_keys
- }
- dataset_rebind = False
- merge_table_column_config(existing_form_data, new_form_data)
- merge_interactive_pivot_ui_config(existing_form_data, new_form_data)
- merged = _merge_replacement_config(
- existing_form_data,
- new_form_data,
- parsed_config,
- dataset_rebind=dataset_rebind,
+ """Rebind a chart and keep operation-owned params aligned with the target."""
+ assert request.dataset_id is not None
+ payload: dict[str, Any] = {
+ "datasource_id": request.dataset_id,
+ "datasource_type": "table",
+ }
+ existing_form_data = _get_existing_form_data(chart)
+ canonical_form_data = canonicalize_operation_form_data(
+ scrub_dataset_bound_form_data(existing_form_data),
Review Comment:
Fixed in f2acaab6defc45cd7680662feced6b74dae95d02.
Dataset changes now require a complete config for every viz type, in both preview and save paths, rather than scrubbing all roles into an unusable chart. Same-dataset requests remain idempotent. Updated the dataset_id description, MCP docs, and migration notes. Regressions: test_dataset_only_rebind_rejects_populated_roles_in_preview_and_save and test_dataset_only_rebind_requires_complete_config; complete-config forwarding and target-access tests also pass.
Validation: 851 tests passed across the 11 touched/related pytest modules; pre-commit passed for all changed files. The regression coverage fails against the previous implementation and passes with this fix.
##########
superset/mcp_service/chart/tool/update_chart_preview.py:
##########
@@ -441,55 +486,60 @@ def update_chart_preview( # noqa: C901
"schema_version": "2.0",
"api_version": "v1",
}
- return result
+ return _finalize_response(result)
except OAuth2RedirectError as ex:
logger.warning(
"Chart preview update requires OAuth authentication: form_data_key=%s",
request.form_data_key,
)
- return {
- "chart": None,
- "error": build_oauth2_redirect_message(ex),
- "success": False,
- "schema_version": "2.0",
- "api_version": "v1",
- }
+ return _finalize_response(
+ {
+ "chart": None,
+ "error": build_oauth2_redirect_message(ex),
+ "success": False,
+ "schema_version": "2.0",
+ "api_version": "v1",
+ }
+ )
except OAuth2Error:
logger.warning(
"OAuth2 configuration error: form_data_key=%s", request.form_data_key
)
- return {
- "chart": None,
- "error": OAUTH2_CONFIG_ERROR_MESSAGE,
- "success": False,
- "schema_version": "2.0",
- "api_version": "v1",
- }
+ return _finalize_response(
+ {
+ "chart": None,
+ "error": OAUTH2_CONFIG_ERROR_MESSAGE,
+ "success": False,
+ "schema_version": "2.0",
+ "api_version": "v1",
+ }
+ )
except GanttSemanticNormalizationError as ex:
execution_time = int((time.time() - start_time) * 1000)
- return {
- "chart": None,
- "error": {
- "error_type": "gantt_semantic_validation_error",
- "message": "Gantt chart column roles are invalid",
- "details": str(ex),
- "suggestions": [
- "Use different physical columns for start_time and end_time",
- "Use different physical columns for category and series",
- "Use exact dataset column casing when names differ only by case",
- ],
- "error_code": "GANTT_SEMANTIC_VALIDATION_ERROR",
- },
- "performance": {
- "query_duration_ms": execution_time,
- "cache_status": "error",
- "optimization_suggestions": [],
- },
- "success": False,
- "schema_version": "2.0",
- "api_version": "v1",
- }
+ return _finalize_response(
+ {
+ "chart": None,
+ "error": {
+ "error_type": "gantt_semantic_validation_error",
+ "message": "Gantt chart column roles are invalid",
+ "details": str(ex),
+ "suggestions": [
+ "Use different physical columns for start_time and end_time",
+ "Use different physical columns for category and series",
+ "Use exact dataset column casing when names differ only "
+ "by case",
+ ],
+ "error_code": "GANTT_SEMANTIC_VALIDATION_ERROR",
+ },
+ "performance": {
+ "query_duration_ms": execution_time,
+ "cache_status": "error",
+ "optimization_suggestions": [],
+ },
+ "success": False,
Review Comment:
Fixed in f2acaab6defc45cd7680662feced6b74dae95d02.
Restored schema_version=2.0 and api_version=v1 on the Gantt semantic validation-error branch. Regression: test_gantt_validation_error_includes_response_versions exercises that exception response.
Validation: 851 tests passed across the 11 touched/related pytest modules; pre-commit passed for all changed files. The regression coverage fails against the previous implementation and passes with this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4192207102
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
Review Comment:
Fixed in a855ac13ea9932d86f0b016558497dc65d231519. Both the timeseries (including Mixed) and Big Number adapters forward `fill_time_range` when enabled. The regression executes the rebuilt QueryObject pipeline with Jan 1โ8 bounds and data on Jan 3โ5, asserting Jan 1โ7 with zero edge buckets; disabling range fill retains Jan 3โ5.
The regression fails against the original code and passes with this fix. Full MCP suite: 8,124 passed, 4 skipped; pre-commit passed.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
Review Comment:
Fixed in a855ac13ea9932d86f0b016558497dc65d231519. The Table plugin resolves the merged inherited comparison before filter preparation consumes `extra_form_data`. Regression tests go through `build_query_dicts_from_form_data` for Table and AG Grid Table, including request-level override precedence and charts whose viz type comes from the saved chart rather than form data. The offset and comparison metric columns survive preparation.
The regression fails against the original code and passes with this fix. Full MCP suite: 8,124 passed, 4 skipped; pre-commit passed.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": form_data.get("time_grain_sqla"),
Review Comment:
Fixed in a855ac13ea9932d86f0b016558497dc65d231519. Prophet uses the frontend precedence: adhoc x-axis `timeGrain`, resolved query extras, panel `time_grain_sqla`, then `P1D`. Forecast regression tests cover an axis-only monthly grain, missing-grain daily fallback, and precedence when multiple sources exist.
The regression fails against the original code and passes with this fix. Full MCP suite: 8,124 passed, 4 skipped; pre-commit passed.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
Review Comment:
Fixed in a855ac13ea9932d86f0b016558497dc65d231519. The XY plugin Vega-Lite renderer folds every numeric/nullable wide series column and binds color to the series label, instead of selecting one region. Regression tests execute the actual pivot/rename/flatten pipeline for one and two grouping dimensions, with and without metric truncation. Long-form results retain their existing encoding, and generated fold fields cannot overwrite the x-axis.
The regression fails against the original code and passes with this fix. Full MCP suite: 8,124 passed, 4 skipped; pre-commit passed.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1274 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
Review Comment:
Fixed in a855ac13ea9932d86f0b016558497dc65d231519. Synthesized physical temporal Table `BASE_AXIS` columns include `isColumnReference: True`. The regression covers Table and AG Grid Table with a semantic-view datasource and verifies the semantic mapper accepts the generated column as the `ds` dimension.
The regression fails against the original code and passes with this fix. Full MCP suite: 8,124 passed, 4 skipped; pre-commit passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3952052043
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -0,0 +1,1525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+
+import math
+import time as system_time
+from bisect import bisect_right
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
+from decimal import Decimal
+from enum import Enum
+from typing import Any
+from uuid import UUID
+from zoneinfo import ZoneInfo
+
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
+from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
+
+FAILED_QUERY_STATUSES = frozenset(
+ {"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
+)
+
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
+ return None
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
+
+
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
Review Comment:
Fixed in 9f7a3a7ac6840564732d93b0c845bd9fa88704fa. The timestamp path accepts the existing hook-free pytz named-zone normalization, including static generated zones; the canonical UTC singleton is normalized explicitly. Stored wall/instant offsets and nanoseconds remain intact. Regressions cover UTC, US/Pacific, Etc/GMT-3, and both New York DST-fold occurrences with nanoseconds. Focused suite: 845 passed; staged and branch-wide pre-commit passed. The broader local suite passed 5,040 tests. All 13 required checks passed on this exact head; no check failures or pending checks remain.
##########
superset/mcp_service/chart/preview_utils.py:
##########
@@ -160,11 +205,17 @@ def _generate_ascii_preview_from_data(
content = _generate_safe_ascii_line_chart(data)
elif viz_type == "pie":
content = _generate_safe_ascii_pie_chart(data)
+ elif viz_type == "sunburst_v2":
Review Comment:
Fixed in 9f7a3a7ac6840564732d93b0c845bd9fa88704fa. Sunburst reserves two headers and, when needed, a footer before selecting rows (17 rows plus a notice at default height for larger results). Tiny canvases retain an omission notice. Display-column-aware clipping marks truncated lines, and embedded line breaks cannot consume the footer. Regressions cover 1โ25-line canvases, 1โ30 rows, CJK/emoji/combining characters, and clipped paths with an intact footer. Focused suite: 845 passed; pre-commit passed. The broader local suite passed 5,040 tests. All 13 required checks passed on this exact head; no check failures or pending checks remain.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,6 +461,1156 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
+) -> dict[str, Any]:
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
+
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
+ """
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
+
+
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
+ query: dict[str, Any] = {
+ "columns": columns,
+ "metrics": metrics,
+ }
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
+ if extras:
+ query["extras"] = extras
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": form_data.get("time_grain_sqla"),
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ query.setdefault("metrics", []).append(sort_metric)
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ )
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ explicit_x_axis = _frontend_x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
+ query["columns"] = time_column
+ if not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
+ query["post_processing"] = post_processing
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
Review Comment:
Fixed in 9f7a3a7ac6840564732d93b0c845bd9fa88704fa at the Excel rebuild caller: a copied form-data payload sets result_format=xlsx before shared query construction. This preserves the configured row limit and excludes interactive page/count queries without changing interactive MCP query semantics or saved chart params. Aggregate/raw Table regressions assert one query, the full 1000-row limit rather than page length 10, no count query, and unchanged saved params. Focused suite: 845 passed; pre-commit passed. The broader local suite passed 5,040 tests. All 13 required checks passed on this exact head; no check failures or pending checks remain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5658734358
Thanks โ I audited the full 77-file diff rather than only relabeling it.
I found 3 Sunburst-only files (+5,866 lines, mostly the exhaustive regression suite), 30 mixed Sunburst/shared chart files (+7,651/-2,185), 33 shared MCP contract files (+10,016/-611), and 11 clearly incidental files (+249/-32).
I separated the independently testable incidental work without rewriting this ready PR:
- #44218: dashboard Excel pagination rebuild (3 files, focused test: 49 passed). Those files are removed from this PR.
- #44219: strict DataFrame/non-finite JSON projection (6 files, focused tests: 150 passed). This should land before #43771; a normal merge of master then removes the duplicated files here.
- Dropped the unrelated Python-unit timeout edit entirely.
I tested removing the DataFrame foundation from #43771 before #44219 lands; it produces 29 focused failures across the real query-context materialization and Sunburst data paths. I therefore did not leave this PR red or hide that dependency behind a mechanical split. With only the Excel extraction and timeout removal applied, the broad focused matrix passes (890 passed, 1 skipped), including Sunburst, both Gauge suites, query-result/get-data, and dashboard export coverage; staged-file pre-commit also passes.
The PR body now has the complete four-group review map, dependency explanation, and explicit landing order: #44218 and #44219 first, normal-merge master into #43771, then #43771. The remaining mixed/shared code meets at the compile, preview, update, and typed query-result boundaries, so further extraction should use the same land-first/normal-merge sequence rather than duplicating code or rewriting history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5864959242
Follow-up test-only commit preserves the Unicode-ID regression through the consolidated dataset lookup rather than mocking that lookup away. The generate/update-preview files pass together (75 tests); staged-file pre-commit passed. No runtime behavior changed from the live-tested merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5964784640
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4188365145
##########
superset/mcp_service/chart/validation/dataset_validator.py:
##########
@@ -252,13 +332,43 @@ def validate_against_dataset(
# handlebars / big number slip through ``SUM(non_numeric)`` patterns
# for the fast-path tools that skip Tier 2.
aggregation_errors = DatasetValidator._validate_aggregations(
- column_refs, dataset_context
+ column_refs,
+ dataset_context,
+ require_numeric_metrics=isinstance(config, SunburstChartConfig),
Review Comment:
**P3:** Master's new `_extract_metric_references` lists metric slots by field name, and `SunburstChartConfig.secondary_metric` is not in the list. A sunburst `secondary_metric` that is a near miss of a saved metric (with no `saved_metric: true`) therefore still ends with "No matching columns found" and no saved-metric hint, while `metric` on the same config does get the hint. Consider adding `"secondary_metric"` to the field tuple in `_extract_metric_references`, ideally with a sunburst test.
##########
superset/mcp_service/chart/compile.py:
##########
@@ -237,23 +281,47 @@ def _validate_adhoc_filter_columns(
# (column, clause) pairs: the clause decides whether a saved metric is a
# legal reference, and so whether metrics belong in the suggestions.
invalid: list[tuple[str, str]] = []
+ has_simple_having = False
for f in adhoc_filters:
# SIMPLE filters expose the column via "subject"; SQL-expression
# filters carry a free-form ``sqlExpression`` we can't safely parse,
# so skip those.
if f.get("expressionType") and f.get("expressionType") != "SIMPLE":
continue
+ clause = f.get("clause", "WHERE")
+ if not isinstance(clause, str) or clause not in {"WHERE", "HAVING"}:
+ return ChartGenerationError(
+ error_type="invalid_filter_clause",
+ message="SIMPLE filter clause must be 'WHERE' or 'HAVING'",
+ details=(
+ f"A SIMPLE filter has malformed clause {clause!r}; the clause "
+ "is never coerced or defaulted when explicitly set."
+ ),
+ suggestions=["Use clause='WHERE'"],
+ error_code="INVALID_FILTER_CLAUSE",
+ )
+ has_simple_having = has_simple_having or clause == "HAVING"
column = f.get("subject") or f.get("col")
if not column or not isinstance(column, str):
continue
- clause = f.get("clause", "WHERE").upper()
try:
if not _adhoc_filter_column_valid(column, clause, dataset_context):
invalid.append((column, clause))
except AmbiguousDatasetReferenceError as ex:
return DatasetValidator._build_ambiguous_reference_error(ex)
if not invalid:
+ if has_simple_having:
Review Comment:
**P3:** Because the HAVING rejection is deferred until after column validation, the invalid-column path below still runs first and passes `include_metrics=True` for HAVING (from master's `_get_column_suggestions` call). So a HAVING subject that is a near miss of a saved metric gets told to use the metric. Probe: `{"clause": "HAVING", "subject": "sum_boyz"}` on the `_orm_dataset()` fixture returns `CHART_VALIDATION_FAILED` with suggestions `['sum_boys']`. After following that suggestion, the caller gets `UNSUPPORTED_FILTER_CLAUSE`, so the guidance is a dead end. Suggest checking `has_simple_having` before the invalid-column error, or leaving metrics out of the suggestions while HAVING is unsupported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-6002332705
Commit c0c84e1e66b4624ac6ec9767adc28c1a7caed70c closes both P3 findings from review 5420035652:
- HAVING hint dead end (discussion_r4188365138): unsupported SIMPLE HAVING filters no longer suggest saved metrics. Existing WHERE behavior is preserved, and the regression test fails before the fix and passes afterward.
- Sunburst secondary_metric hint (discussion_r4188365145): secondary_metric receives the same saved-metric near-miss guidance as the primary metric, without changing other chart types. Both sunburst metric slots are covered by a regression test.
Validation: tests/unit_tests/mcp_service/chart plus test_tool_inventory.py โ 3979 passed, 3 skipped. pre-commit run --files on all four touched files passed after formatting fixes (including mypy). Documentation updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3971724688
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1249,6 +1336,659 @@ def map_gauge_config(config: GaugeChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def preserve_previous_adhoc_filters(
+ new_form_data: Dict[str, Any], previous_form_data: Mapping[str, Any]
+) -> None:
+ """Compatibility entry point backed by the shared filter merge."""
+ filters = _merge_preserved_adhoc_filters(
+ previous_form_data,
+ new_form_data,
+ drop_existing_temporal=False,
+ )
+ if filters is not None:
+ new_form_data["adhoc_filters"] = filters
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update( # noqa: C901
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+ """
+ if dataset_rebind:
+ existing_form_data = scrub_dataset_bound_form_data(existing_form_data)
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
Review Comment:
Gauge updates no longer use the old Gauge-specific merge, but `GaugeChartConfig` is absent from `_MODELED_UPDATE_CONTROL_PATHS`. A partial Gauge update therefore maps defaults such as `show_progress=True` and `split_number=10` over saved non-default values; can Gauge be included in the modeled update semantics and covered through `update_chart`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3971744441
##########
superset/mcp_service/chart/tool/get_chart_preview.py:
##########
@@ -220,6 +232,30 @@ def _authorize_guest_query(self, query_context: Any) -> None:
if (dashboard_id := guest_scope.guest_dashboard_id(self.chart)) is not None:
guest_scope.authorize_query(query_context, dashboard_id, self.chart)
+ def _canonical_form_data(self, form_data: dict[str, Any]) -> dict[str, Any]:
+ """Bind saved-preview identity to the resolved chart and datasource."""
+ canonical = canonicalize_operation_form_data(
+ form_data,
+ datasource_id=self.chart.datasource_id,
+ datasource_type=self.chart.datasource_type,
+ chart_id=self.chart.id,
+ )
+ if self.chart.viz_type == "gauge_chart":
Review Comment:
Only Gauge restores the saved chart's `viz_type` here. For a Sunburst chart whose stored params omit or retain a stale `viz_type`, strategy dispatch uses `chart.viz_type` while the renderer/query builder uses form data, so the preview falls back to a generic table or builds the wrong query. Can this apply the chart viz type for every saved preview?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5502679192
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5526277252
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5767604855
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5648872499
This is a pretty massive diff for "add typed Sunburst support" โ 77 files, +23k/-2.8k, touching `dataframe.py`, `query_context_processor.py`, and the Excel export task too. Any chance this splits into the actual Sunburst feature plus the incidental fixes as separate PRs? Would make this a lot easier to actually review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4201662781
##########
superset/mcp_service/chart/tool/update_chart.py:
##########
@@ -206,209 +219,66 @@ def _append_table_columns(
def _merge_replacement_config(
existing_form_data: dict[str, Any],
new_form_data: dict[str, Any],
- parsed_config: ChartConfig,
+ parsed_config: Any,
*,
dataset_rebind: bool = False,
) -> dict[str, Any]:
- """Delegate update semantics to the shared form-data merge helper."""
- return merge_chart_form_data(
+ """Merge a replacement config through the plugin-aware update merge.
+
+ An explicit empty filter list clears saved filters; plugins that generate
+ their own time binding (Gantt) keep it through that clear.
+ """
+ return merge_form_data_for_update(
existing_form_data,
new_form_data,
parsed_config,
dataset_rebind=dataset_rebind,
)
-def _valid_dataset_reference(
- value: Any,
- columns: set[str],
- metrics: set[str],
- *,
- allow_metric: bool = False,
-) -> bool:
- """Return whether a form_data reference resolves against a dataset."""
- if not isinstance(value, str):
- return True
- normalized = value.casefold()
- return normalized in columns or (allow_metric and normalized in metrics)
-
-
-def _inherited_metrics_match_dataset(
- existing_form_data: dict[str, Any],
- columns: set[str],
- metrics: set[str],
-) -> bool:
- for metric in existing_form_data.get("metrics") or []:
- if isinstance(metric, str) and not _valid_dataset_reference(
- metric, columns, metrics, allow_metric=True
- ):
- return False
- if isinstance(metric, dict):
- column = metric.get("column")
- if isinstance(column, dict) and not _valid_dataset_reference(
- column.get("column_name"), columns, metrics
- ):
- return False
- return True
-
-
-def _inherited_sort_matches_dataset(
- order_by_cols: Any, columns: set[str], metrics: set[str]
-) -> bool:
- for order_by in order_by_cols or []:
- try:
- column = json.loads(order_by)[0]
- except (TypeError, ValueError, IndexError):
- return False
- if not _valid_dataset_reference(column, columns, metrics, allow_metric=True):
- return False
- return True
-
-
-def _inherited_filters_match_dataset(
- filters: Any, columns: set[str], metrics: set[str]
-) -> bool:
- for filter_ in filters or []:
- if not isinstance(filter_, dict):
- return False
- if filter_.get("expressionType") not in (None, "SIMPLE"):
- return False
- subject = filter_.get("subject") or filter_.get("col")
- allow_metric = str(filter_.get("clause", "WHERE")).upper() == "HAVING"
- if not _valid_dataset_reference(
- subject, columns, metrics, allow_metric=allow_metric
- ):
- return False
- return True
-
-
-#: form_data keys carrying query roles, mapped to the config field that
-#: sets them explicitly. An explicit field is the caller's stated intent,
-#: so it is never treated as inherited state.
-_INHERITED_QUERY_ROLE_FIELDS = {
- "groupby": "group_by",
- "groupby_b": "group_by_secondary",
- "all_columns": None,
- "columns": None,
- "x_axis": None,
- "granularity_sqla": None,
- "metrics": None,
- "order_by_cols": "sort_by",
- "adhoc_filters": "filters",
-}
-
-_INHERITED_COLUMN_LIST_KEYS = frozenset(
- {"groupby", "groupby_b", "all_columns", "columns"}
-)
-_INHERITED_COLUMN_SCALAR_KEYS = frozenset({"x_axis", "granularity_sqla"})
+def _is_dataset_rebind(request: UpdateChartRequest, chart: Any) -> bool:
+ """Return whether a request changes the chart's datasource identity."""
+ return request.dataset_id is not None and str(request.dataset_id) != str(
+ getattr(chart, "datasource_id", None)
+ )
-def _inherited_state_invalid_keys(
- existing_form_data: dict[str, Any],
- new_form_data: dict[str, Any],
- parsed_config: ChartConfig,
- dataset_id: int,
-) -> set[str]:
- """Return inherited query fields that are invalid for a new dataset.
-
- A dataset rebind only has to discard the state that cannot resolve
- against the replacement dataset; everything else stays valid and is
- preserved so the update does not silently reset the chart.
- """
- fields_set = parsed_config.model_fields_set
- inherited_keys = {
- key
- for key, config_field in _INHERITED_QUERY_ROLE_FIELDS.items()
- if key not in new_form_data
- and config_field not in fields_set
- and existing_form_data.get(key)
- }
- if not inherited_keys:
- return set()
-
- from superset.daos.dataset import DatasetDAO
- from superset.mcp_service.chart.validation.dataset_validator import (
- build_dataset_context_from_orm,
+def _add_columns_rebind_error() -> GenerateChartResponse:
+ """Require a complete replacement config when changing table datasets."""
+ return _validation_error_response(
+ message="Cannot combine 'add_columns' with a dataset rebind.",
+ details=(
+ "Dataset-bound roles from the previous dataset cannot be reused. "
+ "Provide 'config' with the complete table configuration for the "
+ "target dataset."
+ ),
)
- context = build_dataset_context_from_orm(DatasetDAO.find_by_id(dataset_id))
- if context is None:
- # The replacement dataset cannot be inspected, so no inherited
- # reference can be shown to be safe.
- return inherited_keys
- columns = {column["name"].casefold() for column in context.available_columns}
- metrics = {metric["name"].casefold() for metric in context.available_metrics}
-
- invalid_keys: set[str] = set()
- for key in inherited_keys & _INHERITED_COLUMN_LIST_KEYS:
- values = existing_form_data.get(key)
- if isinstance(values, list) and not all(
- _valid_dataset_reference(value, columns, metrics) for value in values
- ):
- invalid_keys.add(key)
- for key in inherited_keys & _INHERITED_COLUMN_SCALAR_KEYS:
- if not _valid_dataset_reference(existing_form_data.get(key), columns, metrics):
- invalid_keys.add(key)
- if "metrics" in inherited_keys and not _inherited_metrics_match_dataset(
- existing_form_data, columns, metrics
- ):
- invalid_keys.add("metrics")
- if "order_by_cols" in inherited_keys and not _inherited_sort_matches_dataset(
- existing_form_data.get("order_by_cols"), columns, metrics
- ):
- invalid_keys.add("order_by_cols")
- if "adhoc_filters" in inherited_keys and not _inherited_filters_match_dataset(
- existing_form_data.get("adhoc_filters"), columns, metrics
- ):
- invalid_keys.add("adhoc_filters")
- return invalid_keys
-
-
-def _build_replacement_form_data(
- existing_form_data: dict[str, Any],
- parsed_config: ChartConfig,
- effective_dataset_id: int | None,
- replacement_dataset_id: int | None = None,
+
+def _build_dataset_rebind_payload(
+ request: UpdateChartRequest, chart: Any
) -> dict[str, Any]:
- """Map and merge a replacement config for preview and save paths."""
- new_form_data = map_config_to_form_data(
- parsed_config, dataset_id=effective_dataset_id, include_disabled=True
- )
- new_form_data.pop("_mcp_warnings", None)
- dataset_rebind = replacement_dataset_id is not None
- config_plugin = get_registry().get(parsed_config.chart_type, include_disabled=True)
- if replacement_dataset_id is not None and not (
- config_plugin is not None and config_plugin.strict_dataset_rebind
- ):
- # Drop only the inherited state the replacement dataset cannot
- # resolve, then merge as a same-dataset update. Plugins with a strict
- # rebind contract handle the rebind in merge_update_form_data.
- invalid_keys = _inherited_state_invalid_keys(
- existing_form_data,
- new_form_data,
- parsed_config,
- replacement_dataset_id,
- )
- existing_form_data = {
- key: value
- for key, value in existing_form_data.items()
- if key not in invalid_keys
- }
- dataset_rebind = False
- merge_table_column_config(existing_form_data, new_form_data)
- merge_interactive_pivot_ui_config(existing_form_data, new_form_data)
- merged = _merge_replacement_config(
- existing_form_data,
- new_form_data,
- parsed_config,
- dataset_rebind=dataset_rebind,
+ """Rebind a chart and keep operation-owned params aligned with the target."""
+ assert request.dataset_id is not None
+ payload: dict[str, Any] = {
+ "datasource_id": request.dataset_id,
+ "datasource_type": "table",
+ }
+ existing_form_data = _get_existing_form_data(chart)
+ canonical_form_data = canonicalize_operation_form_data(
+ scrub_dataset_bound_form_data(existing_form_data),
Review Comment:
A dataset-only rebind now strips every saved metric and groupby even when the target dataset has the same columns, leaving a chart with nothing to query, while the `dataset_id` description still promises a rebind "without altering the config". Gauge and Treemap return an explicit complete-config error for this case; should other viz types do the same (and the description say so), or keep the roles that still resolve on the target?
##########
superset/mcp_service/chart/chart_helpers.py:
##########
@@ -815,6 +1152,14 @@ def build_mixed_timeseries_secondary(
)
+def with_x_axis_column(form_data: dict[str, Any], groupby: list[Any]) -> list[Any]:
Review Comment:
With query building now routed through `build_query_objects_from_form_data`, `with_x_axis_column`, `build_mixed_timeseries_secondary` and `_DECK_TIMESERIES_VIZ_TYPES` here, plus `_columns_and_metrics` in `form_data_query_context.py`, have no remaining callers (this PR removed master's call sites). Can these be deleted so fixes don't keep landing in a second builder that can't affect any query?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1305 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ extra_metric = sort_metric
+ else:
+ extra_metric = None
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ sort_metric=extra_metric,
+ )
+ if extra_metric is not None:
+ query.setdefault("metrics", []).append(extra_metric)
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
+ extras = query.get("extras")
+ if isinstance(extras, dict):
+ extras.pop("time_grain_sqla", None)
+ if not extras:
+ query.pop("extras", None)
+ elif not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
query["post_processing"] = post_processing
- # ``granularity`` does two jobs downstream: it names the temporal column the
- # time range filters on, and it is the column ``time_grain_sqla`` buckets
- # (``models/helpers.py`` swaps a selected column for its timestamp expression
- # when that column equals ``granularity``). Only the first job depends on
- # there being an active range, so set it whenever form data carries one โ
- # matching ``extractExtras.ts``, which sets it unconditionally. Gating it on
- # ``time_range`` dropped the bucketing, so an ordinary "all-time totals by
- # month" chart exported one row per raw timestamp instead of one per month.
- if granularity := form_data.get("granularity") or form_data.get("granularity_sqla"):
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
+ page_size = int(form_data.get("server_page_length") or 0)
+ configured_limit = int(form_data.get("row_limit") or 0)
+ query["row_limit"] = (
+ min(page_size, configured_limit) if configured_limit else page_size
+ )
+ query["row_offset"] = 0
+
+ queries = [query]
+ if form_data.get("percent_metric_calculation") == "all_records" and percent_metrics:
+ all_records = dict(query)
+ all_records.update(
+ {
+ "columns": [],
+ "metrics": percent_metrics,
+ "post_processing": [],
+ "row_limit": 0,
+ "row_offset": 0,
+ "orderby": [],
+ "is_timeseries": False,
+ }
+ )
+ queries.append(all_records)
+ if form_data.get("show_totals") and metrics:
+ totals_aggregate = form_data.get("totals_aggregate")
+ if totals_aggregate not in {"SUM", "AVG"}:
+ totals_aggregate = None
+ totals_metrics = [
+ {
+ **metric,
+ "aggregate": totals_aggregate,
+ }
+ if totals_aggregate
+ and isinstance(metric, dict)
+ and metric.get("expressionType") == "SIMPLE"
+ else metric
+ for metric in metrics
+ ]
+ totals = dict(query)
+ totals.update(
+ {
+ "columns": [],
+ "metrics": totals_metrics,
+ "row_limit": 0,
+ "row_offset": 0,
+ "post_processing": [contribution] if contribution else [],
+ }
+ )
+ totals.pop("order_desc", None)
+ totals.pop("orderby", None)
+ queries.append(totals)
+ if form_data.get("server_pagination") and not is_download:
+ rowcount = dict(query)
+ rowcount.update(
+ {
+ "time_offsets": [],
+ "row_limit": int(form_data.get("row_limit") or 0),
+ "row_offset": 0,
+ "post_processing": [],
+ "is_rowcount": True,
+ }
+ )
+ queries.insert(1, rowcount)
+ return queries
+
+
+def _treemap_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ """Apply Treemap's bounded metric and hierarchy ordering contract."""
+ hierarchy = _as_list(form_data.get("groupby"))
+ metric = form_data.get("metric")
+ query["columns"] = hierarchy
+ query["metrics"] = [metric] if metric is not None else []
+ granularity = form_data.get("granularity", form_data.get("granularity_sqla"))
+ if granularity is not None:
query["granularity"] = granularity
- if form_data.get("row_limit"):
- query["row_limit"] = form_data["row_limit"]
+ ordering = [[metric, False]] if form_data.get("sort_by_metric") and metric else []
+ ordering.extend(
+ [column, True] for column in hierarchy if isinstance(column, str) and column
+ )
+ try:
+ bounded = float(query.get("row_limit") or 0) != 0
+ except (ValueError, TypeError):
+ bounded = True
+ query.pop("orderby", None)
+ if bounded and ordering:
+ query["orderby"] = ordering
+
+
+def _bubble_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ """Preserve Bubble's entity/series roles and explicit sort-metric direction."""
+ query["columns"] = _deduplicate_fields(
+ [*_as_list(form_data.get("entity")), *_as_list(form_data.get("series"))]
+ )
+ query.pop("orderby", None)
+ if sort_metric := next(iter(_as_list(form_data.get("orderby"))), None):
+ query["orderby"] = [[sort_metric, not query.get("order_desc", True)]]
+
+
+def _entity_map_query(
+ form_data: dict[str, Any], query: dict[str, Any], *, world_map: bool
+) -> None:
+ """Group World/Country Map queries by their ``entity`` role.
+
+ Mirrors the World Map and Country Map ``buildQuery`` transforms: the
+ ``entity`` control is the only grouping column, Country Map selects one
+ ``metric``, and World Map adds a distinct ``secondary_metric`` and orders by
+ the primary metric only when ``sort_by_metric`` is enabled.
+ """
+ entity = form_data.get("entity")
+ query["columns"] = [entity] if entity else []
+ metric = form_data.get("metric")
+ metrics = [metric] if metric else []
+ if world_map:
+ secondary_metric = form_data.get("secondary_metric")
+ if secondary_metric and (
+ not metric
+ or _label(secondary_metric, metric=True) != _label(metric, metric=True)
+ ):
+ metrics.append(secondary_metric)
+ query.pop("orderby", None)
+ if form_data.get("sort_by_metric") and metric:
+ query["orderby"] = [[metric, False]]
+ query["metrics"] = metrics
+
+
+def build_query_objects_from_form_data( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ viz_type: str | None = None,
+ row_limit: int | None = None,
+ order_desc: bool | None = None,
+ filters_prepared: bool = False,
+ secondary_form_data: dict[str, Any] | None = None,
+) -> list[dict[str, Any]]:
+ """Build frontend-equivalent QueryObject dictionaries from form data.
+
+ Every real MCP product path and the legacy common fallback share this
+ extractor and the chart adapters below. This prevents compile/SQL/preview
+ and saved/fallback data paths from silently selecting different fields.
+ """
+ effective_viz = viz_type or str(form_data.get("viz_type") or "")
+ extraction_form_data = form_data
+ if is_raw_query_mode(form_data):
+ # ``extractQueryFields`` is strict, while the legacy server fallback has
+ # always kept old saved tables executable by dropping malformed order
+ # entries. Normalize that bounded raw-table input before invoking the
+ # shared extractor; valid entries still use the centralized contract.
+ extraction_form_data = {
+ **form_data,
+ "query_mode": "raw",
+ "order_by_cols": orderby_from_form_data(form_data, [], effective_viz),
+ }
+ query = _base_query_object(
+ extraction_form_data,
+ row_limit=row_limit,
+ order_desc=order_desc,
+ filters_prepared=filters_prepared,
+ )
+
+ if effective_viz in {"table", "ag-grid-table"}:
+ return _finalize_query_objects(form_data, _table_queries(form_data, query))
+ if effective_viz == "histogram_v2":
+ _histogram_query(form_data, query)
+ elif effective_viz == "box_plot":
+ _box_plot_query(form_data, query)
+ elif effective_viz == "pivot_table_v2":
+ _pivot_query(form_data, query)
+ elif effective_viz == "waterfall":
+ _waterfall_query(form_data, query)
+ elif effective_viz in {"gantt", "gantt_chart"}:
+ _gantt_query(form_data, query)
+ elif effective_viz.startswith("echarts_timeseries") or effective_viz in {
+ "echarts_area",
+ "mixed_timeseries",
+ }:
+ _timeseries_query(form_data, query)
+ elif effective_viz == "big_number":
+ return _finalize_query_objects(form_data, _big_number_queries(form_data, query))
+ elif effective_viz == "big_number_total":
+ query["columns"] = []
+ elif effective_viz == "handlebars":
+ _normalize_query_orderby(query)
+ elif effective_viz == "pie":
+ if form_data.get("sort_by_metric") and form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ query["post_processing"] = _pie_contribution_post_processing(
+ list(query.get("metrics") or [])
+ )
+ elif effective_viz == "gauge_chart":
+ if form_data.get("sort_by_metric") and form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ elif effective_viz == "treemap_v2":
+ _treemap_query(form_data, query)
+ elif effective_viz == "sunburst_v2":
+ if not form_data.get("sort_by_metric"):
+ query["orderby"] = []
+ elif form_data.get("metric") is not None:
+ query["orderby"] = [[form_data["metric"], False]]
+ elif effective_viz in {"bubble", "bubble_v2"}:
+ _bubble_query(form_data, query)
+ elif effective_viz in {"world_map", "country_map"}:
+ _entity_map_query(form_data, query, world_map=effective_viz == "world_map")
+ elif effective_viz == "ag-grid-pivot-table":
+ query["columns"] = _temporalized_columns(
+ form_data, _as_list(form_data.get("groupby"))
+ )
+
+ elif not query.get("orderby"):
+ # Unadapted visualizations retain the legacy top-N ordering fallback.
+ # Handled adapters may deliberately leave ordering empty.
+ query["orderby"] = orderby_from_form_data(
+ form_data, list(query.get("metrics") or []), effective_viz
+ )
+
+ if effective_viz != "mixed_timeseries":
+ return _finalize_query_objects(form_data, [query])
+
+ secondary = secondary_form_data or retain_mixed_timeseries_secondary_form_data(
+ form_data
+ )
+ # Query A's saved ordering references metrics query B may not select, so the
+ # secondary series orders by ``orderby_b`` or not at all.
+ if (orderby_b := form_data.get("orderby_b")) is not None:
+ secondary["orderby"] = orderby_b
+ elif "orderby_b" not in form_data:
+ secondary.pop("orderby", None)
+ query_b = _base_query_object(
+ secondary,
+ row_limit=row_limit,
+ order_desc=order_desc,
Review Comment:
Query B takes its sort direction from the `order_desc` passed in for query A instead of from `order_desc_b`, so an unsaved or fallback Mixed chart with `order_desc=true`, `order_desc_b=false` and `series_limit_b=1` keeps the highest series for B where the frontend keeps the lowest. Should B ignore the A override so `order_desc_b` wins?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1305 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ extra_metric = sort_metric
+ else:
+ extra_metric = None
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ sort_metric=extra_metric,
+ )
+ if extra_metric is not None:
+ query.setdefault("metrics", []).append(extra_metric)
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
Review Comment:
A saved Big Number with `x_axis={"column_name": "ds"}` and `granularity_sqla` set selects `ds` and drops the granularity binding above, but this strict label still falls back to `__timestamp`, so the pivot's `index` names a column the query never returns and fails with "Referenced columns not available in DataFrame". Should the label come from the same `explicit_x_axis` that picked the column?
##########
superset/common/form_data_query_context.py:
##########
@@ -23,30 +23,188 @@
carry no ``query_context``, so server-side consumers that need to run the query
(e.g. the dashboard Excel export) have nothing to execute.
-This module rebuilds a best-effort query context from the form data โ columns,
-metrics, filters (including free-form SQL and the time range), ordering and time
-grain โ mirroring the shared parts of the viz plugins' ``buildQuery``. Beyond
-that shared core it reproduces exactly one piece of plugin post-processing, Pie's
-unconditional ``contribution`` operator (see
-:func:`_pie_contribution_post_processing`). It does **not** reproduce any other
-post-processing (pivot, percent-metric transforms, rolling/forecast) or
-multi-query fan-out, so callers must restrict it to viz types whose data maps
-faithfully to a single query.
+This module rebuilds query contexts from form data โ columns, metrics, filters,
+ordering, temporal state, limits, query fan-out, and registered visualization
+post-processing โ mirroring the shared frontend extractor and explicit plugin
+``buildQuery`` adapters. MCP compile, SQL, preview, and data fallback products
+consume the same QueryObject contract as the legacy common fallback.
The mirrored logic lives on the frontend in
-``superset-frontend/plugins/plugin-chart-table/src/buildQuery.ts`` (query mode,
-ordering), ``superset-frontend/packages/superset-ui-core/src/query/`` (field
-extraction, ``processFilters``). There is no automated tripwire tying the two
-across the language boundary; the per-helper pointers below must be kept in sync
-when that frontend logic changes.
+``superset-frontend/plugins/*/buildQuery.ts`` and
+``superset-frontend/packages/superset-ui-core/src/query/`` (field extraction,
+``processFilters``). The adapter tests pin the Python QueryObject dictionaries;
+the per-helper pointers below must still be kept in sync when frontend contracts
+change.
"""
from __future__ import annotations
+from collections.abc import Mapping
from typing import Any
from superset.utils import json
-from superset.utils.core import as_list, get_metric_name
+from superset.utils.core import as_list, DTTM_ALIAS, get_column_name, get_metric_name
+
+# Keep this mapping identical to ``queryFieldAliases`` in
+# ``superset-ui-core/src/query/extractQueryFields.ts``. It is the shared
+# frontend contract for every form-data key that can contribute a metric,
+# column, or ordering expression to a QueryObject. Server-side consumers and
+# MCP replacement cleanup import this contract rather than maintaining partial
+# chart-specific copies.
+FORM_DATA_QUERY_FIELD_ALIASES: dict[str, str] = {
+ "metric": "metrics",
+ "metric_2": "metrics",
+ "secondary_metric": "metrics",
+ "left_metric": "metrics",
+ "right_metric": "metrics",
Review Comment:
The comment above says this mapping is identical to `queryFieldAliases`, but the frontend also maps `open`, `close`, `high` and `low` to metrics (the Candlestick controls). A saved Candlestick without a `query_context` therefore builds its fallback query without the OHLC metrics; should those four be added here?
##########
superset/mcp_service/chart/tool/update_chart_preview.py:
##########
@@ -441,55 +486,60 @@ def update_chart_preview( # noqa: C901
"schema_version": "2.0",
"api_version": "v1",
}
- return result
+ return _finalize_response(result)
except OAuth2RedirectError as ex:
logger.warning(
"Chart preview update requires OAuth authentication: form_data_key=%s",
request.form_data_key,
)
- return {
- "chart": None,
- "error": build_oauth2_redirect_message(ex),
- "success": False,
- "schema_version": "2.0",
- "api_version": "v1",
- }
+ return _finalize_response(
+ {
+ "chart": None,
+ "error": build_oauth2_redirect_message(ex),
+ "success": False,
+ "schema_version": "2.0",
+ "api_version": "v1",
+ }
+ )
except OAuth2Error:
logger.warning(
"OAuth2 configuration error: form_data_key=%s", request.form_data_key
)
- return {
- "chart": None,
- "error": OAUTH2_CONFIG_ERROR_MESSAGE,
- "success": False,
- "schema_version": "2.0",
- "api_version": "v1",
- }
+ return _finalize_response(
+ {
+ "chart": None,
+ "error": OAUTH2_CONFIG_ERROR_MESSAGE,
+ "success": False,
+ "schema_version": "2.0",
+ "api_version": "v1",
+ }
+ )
except GanttSemanticNormalizationError as ex:
execution_time = int((time.time() - start_time) * 1000)
- return {
- "chart": None,
- "error": {
- "error_type": "gantt_semantic_validation_error",
- "message": "Gantt chart column roles are invalid",
- "details": str(ex),
- "suggestions": [
- "Use different physical columns for start_time and end_time",
- "Use different physical columns for category and series",
- "Use exact dataset column casing when names differ only by case",
- ],
- "error_code": "GANTT_SEMANTIC_VALIDATION_ERROR",
- },
- "performance": {
- "query_duration_ms": execution_time,
- "cache_status": "error",
- "optimization_suggestions": [],
- },
- "success": False,
- "schema_version": "2.0",
- "api_version": "v1",
- }
+ return _finalize_response(
+ {
+ "chart": None,
+ "error": {
+ "error_type": "gantt_semantic_validation_error",
+ "message": "Gantt chart column roles are invalid",
+ "details": str(ex),
+ "suggestions": [
+ "Use different physical columns for start_time and end_time",
+ "Use different physical columns for category and series",
+ "Use exact dataset column casing when names differ only "
+ "by case",
+ ],
+ "error_code": "GANTT_SEMANTIC_VALIDATION_ERROR",
+ },
+ "performance": {
+ "query_duration_ms": execution_time,
+ "cache_status": "error",
+ "optimization_suggestions": [],
+ },
+ "success": False,
Review Comment:
The Gantt validation-error response dropped `schema_version` and `api_version`, which master returned here and every sibling error branch still includes, so clients keying on them miss this error.
```suggestion
"success": False,
"schema_version": "2.0",
"api_version": "v1",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4225386352
##########
superset/mcp_service/chart/plugins/mixed_timeseries.py:
##########
@@ -28,20 +28,36 @@
map_mixed_timeseries_config,
)
from superset.mcp_service.chart.plugin import BaseChartPlugin
-from superset.mcp_service.chart.schemas import ColumnRef, MixedTimeseriesChartConfig
+from superset.mcp_service.chart.schemas import (
+ ChartError,
+ ColumnRef,
+ MixedTimeseriesChartConfig,
+ VegaLitePreview,
+)
from superset.mcp_service.chart.validation.dataset_validator import DatasetValidator
from superset.mcp_service.common.error_schemas import ChartGenerationError
class MixedTimeseriesChartPlugin(BaseChartPlugin):
"""Plugin for mixed_timeseries chart type."""
+ query_role_keys = BaseChartPlugin.query_role_keys
chart_type = "mixed_timeseries"
display_name = "Mixed Timeseries"
native_viz_types: ClassVar[Mapping[str, str]] = {
"mixed_timeseries": "Mixed Timeseries Chart",
}
+ def vega_lite_preview(
Review Comment:
Adding this `vega_lite_preview` hook means saved Mixed Timeseries previews now go through `_create_plugin_preview`, but the plugin keeps the default `resizes_saved_preview = False`, so the width/height/description framing is skipped. A `get_chart_preview` call with `format="vega_lite", width=800, height=200` returns `width="container"` and the generator's own height instead of the requested size; before this hook the saved-chart fallback honored it. XY hit the same problem and sets `resizes_saved_preview = True`; should Mixed Timeseries do the same?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -2158,6 +3046,56 @@ def map_mixed_timeseries_config(
_add_adhoc_filters(form_data, config.filters)
+ fields_set = config.model_fields_set
+ if "adhoc_filters_b" in fields_set:
+ form_data["adhoc_filters_b"] = config.adhoc_filters_b
Review Comment:
`adhoc_filters_b` is copied into the form data as-is, but the tier-1 check only walks `form_data["adhoc_filters"]`. That check rejects a SIMPLE `HAVING` filter for query A ("SIMPLE HAVING filters are unsupported"), while the same filter in `adhoc_filters_b` passes validation and is then dropped by the `where_only=True` conversion when query B is built. For example, `adhoc_filters_b=[{"expressionType":"SIMPLE","clause":"HAVING","subject":"orders","operator":">","comparator":100}]` compiles cleanly and query B returns groups below the threshold. Should the B filters go through the same validation as the primary ones?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1784,799 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "series_limit_metric": (("series_limit_metric",), ("timeseries_limit_metric",)),
+ "timeseries_limit_metric": (
+ ("series_limit_metric",),
+ ("timeseries_limit_metric",),
+ ),
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "groupby": (("group_by",),),
+ "groupby_b": (("group_by_secondary",),),
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
Review Comment:
Bubble isn't in `_MODELED_UPDATE_CONTROL_PATHS`, and `series` is now stripped with the other query roles, so a same-viz `update_chart` that omits the optional `series` silently drops a saved grouping. `map_bubble_config` also always emits `row_limit` and `color_scheme` (defaulting to `supersetColors`), so an update that doesn't restate them resets a saved palette and row limit; the old merge kept both when they weren't in `fields_set`. For example, a `bubble_v2` saved with `series="region"` and a custom `color_scheme` comes back ungrouped on the default palette after an update that only changes `x`. Should Bubble get a modeled-control entry for `series`, `row_limit` and `color_scheme` like Pie/XY?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1548 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+import base64
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.utils.serialization import BINARY_PREFIX
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _canonical_binary(
+ value: bytes | bytearray | memoryview,
+) -> tuple[str | None, str | None]:
+ """Render exact binary cells as UTF-8 text or ``base64:``-prefixed text.
+
+ Matches the MCP response serializer: valid UTF-8 is returned as text and
+ anything else is losslessly base64-encoded. The raw size is bounded before
+ copying so the rendered string stays within the per-cell text budget.
+ """
+ size = value.nbytes if type(value) is memoryview else len(value)
+ if size > MAX_RESULT_STRING_LENGTH:
+ return None, "an oversized binary value"
+ try:
+ raw = bytes(value)
+ except (BufferError, TypeError, ValueError):
+ return None, "an invalid binary value"
+ try:
+ text = raw.decode("utf-8")
+ except UnicodeDecodeError:
+ text = BINARY_PREFIX + base64.b64encode(raw).decode("ascii")
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized binary value"
+ return text, None
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ return (value, None) if math.isfinite(value) else (None, None)
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+ if value_type is bytes or value_type is bytearray or value_type is memoryview:
+ return _canonical_binary(value)
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, None)
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
Review Comment:
Anything outside this exact-type whitelist now fails the whole response with `InvalidQueryResult`, where it used to be stringified. For a saved raw Table with a nullable `int4range[]` column whose first row is NULL and second row is `ARRAY[int4range(1,10,'[)')]`, the nested psycopg range objects survive result-set conversion and reach this branch, so `get_chart_data(format="json")` returns an error instead of `{"bands": ["[1, 10)"]}`. Is rejecting ordinary driver-native values like ranges intended, or should concrete range types get bounded text normalization here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5967332757
Bito review-body nits: the `coltypes` empty-list default now uses `is None` (already in the branch), and the duplicated unsaved get_chart_data stub setup in the timezone tests is now a shared helper, fixed in 2f8a8afb859f7ded07602a653269be6fb42eaeb5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-6008294248
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5518841694
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3925215579
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -1045,6 +1064,831 @@ def reject_sql_expression_on_dimensions(self) -> "PieChartConfig":
return self
+class SunburstStandardizedControls(UnknownFieldCheckMixin):
+ """Bounded shared controls retained by Explore across viz changes."""
+
+ model_config = ConfigDict(extra="ignore")
+
+ metrics: list[JsonValue] = Field(default_factory=list)
+ columns: list[JsonValue] = Field(default_factory=list)
+
+
+class SunburstStandardizedFormData(UnknownFieldCheckMixin):
+ """Validated shape of Explore's cross-plugin UI memory."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ controls: SunburstStandardizedControls
+ memorized_form_data: list[tuple[str, dict[str, JsonValue]]] = Field(
+ default_factory=list,
+ validation_alias=AliasChoices("memorized_form_data", "memorizedFormData"),
+ serialization_alias="memorizedFormData",
+ )
+
+
+_NATIVE_COLUMN_META_MAX_DEPTH = 8
+_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS = 128
+_NATIVE_COLUMN_META_MAX_TOTAL_VALUES = 512
+_NATIVE_COLUMN_META_MAX_KEY_BYTES = 1_024
+_NATIVE_COLUMN_META_MAX_STRING_BYTES = 16 * 1_024
+_NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES = 64 * 1_024
+_NATIVE_COLUMN_META_MAX_INT_BITS = 64
+_NATIVE_METRIC_ALLOWED_KEYS = frozenset(
+ {
+ "aggregate",
+ "column",
+ "datasourceWarning",
+ "expressionType",
+ "hasCustomLabel",
+ "label",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_EXPRESSION_TYPES = frozenset({"SIMPLE", "SQL"})
+_NATIVE_METRIC_DISCRIMINATOR_KEYS = frozenset(
+ {
+ "column",
+ "datasourceWarning",
+ "hasCustomLabel",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_MISSING = object()
+
+
+class _UnsafeNativeMetricInputError(ValueError):
+ """Signal input that must be removed before Pydantic renders an error."""
+
+
+def _native_column_meta_string_bytes(value: str, *, key: bool = False) -> int:
+ """Return exact UTF-8 size for a trusted built-in string."""
+ kind = "key" if key else "string"
+ try:
+ size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(
+ f"Sunburst native metric column metadata {kind} is invalid"
+ ) from ex
+ limit = (
+ _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ if key
+ else _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ )
+ if size > limit:
+ raise ValueError(f"Sunburst native metric column metadata {kind} is too long")
+ return size
+
+
+def _validate_native_metric_string(
+ value: Any, field: str, max_length: int
+) -> str | None:
+ """Validate one optional exact string in the closed native metric wrapper."""
+ if value is None:
+ return None
+ if type(value) is not str:
+ raise ValueError(f"Sunburst native metric {field} must be a string")
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(f"Sunburst native metric {field} is invalid") from ex
+ if str.__len__(value) > max_length or encoded_size > max_length * 4:
+ raise ValueError(f"Sunburst native metric {field} is too long")
+ return value
+
+
+def _inspect_native_metric_expression_type(value: Any) -> tuple[object, bool]:
+ """Inspect an exact wrapper without hashing or comparing hostile objects."""
+ if type(value) is not dict:
+ return _NATIVE_METRIC_MISSING, False
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+ expression_type: object = _NATIVE_METRIC_MISSING
+ has_native_discriminator = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise _UnsafeNativeMetricInputError(
+ "Sunburst native metric keys must be strings"
+ )
+ _validate_native_metric_string(key, "field name", 255)
+ if key in _NATIVE_METRIC_DISCRIMINATOR_KEYS:
+ has_native_discriminator = True
+ if key == "expressionType":
+ if type(item) is not str:
+ if issubclass(type(item), str):
+ raise _UnsafeNativeMetricInputError(
+ "Sunburst native metric expressionType must be a string"
+ )
+ raise ValueError(
+ "Sunburst native metric expressionType must be a string"
+ )
+ if item not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError(
+ "Sunburst native metric expressionType must be SIMPLE or SQL"
+ )
+ expression_type = item
+ return expression_type, has_native_discriminator
+
+
+def _validate_native_metric_column_value(value: Any) -> None:
+ """Validate a wrapper column without dispatching container or scalar hooks."""
+ if type(value) is str:
+ _validate_native_metric_string(value, "column", 255)
+ elif type(value) is dict:
+ _validate_bounded_native_column_meta(value)
+ elif value is not None:
+ raise ValueError("Sunburst native metric column must be a string or object")
+
+
+def _validate_native_metric_item(key: str, item: Any) -> object:
+ """Validate one value after its wrapper key is known to be an exact string."""
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError("Sunburst native metric expressionType must be a string")
+ _validate_native_metric_string(item, "expressionType", 10)
+ return item
+ if key in ("label", "optionName"):
+ _validate_native_metric_string(item, key, 500)
+ elif key == "sqlExpression":
+ _validate_native_metric_string(item, key, 2_000)
+ elif key == "aggregate":
+ _validate_native_metric_string(item, key, 32)
+ elif key in ("datasourceWarning", "hasCustomLabel"):
+ if item is not None and type(item) is not bool:
+ raise ValueError(f"Sunburst native metric {key} must be a boolean")
+ elif key == "column":
+ _validate_native_metric_column_value(item)
+ return _NATIVE_METRIC_MISSING
+
+
+def _validate_native_metric_wrapper(value: Any) -> dict[str, Any]:
+ """Validate and copy the closed native metric wrapper without calling hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric must be a string or object")
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+
+ validated: dict[str, Any] = {}
+ expression_type: object = _NATIVE_METRIC_MISSING
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key not in _NATIVE_METRIC_ALLOWED_KEYS:
+ raise ValueError(f"Unknown Sunburst native metric field: {key}")
+ item_expression_type = _validate_native_metric_item(key, item)
+ if item_expression_type is not _NATIVE_METRIC_MISSING:
+ expression_type = item_expression_type
+ validated[key] = item
+
+ if expression_type is _NATIVE_METRIC_MISSING:
+ raise ValueError("Sunburst native metric requires expressionType")
+ if expression_type not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError("Sunburst native metric expressionType must be SIMPLE or SQL")
+ return validated
+
+
+def _validate_non_dict_native_metric(value: Any) -> ColumnRef:
+ """Accept an existing model while rejecting hook-bearing dict subclasses."""
+ if issubclass(type(value), dict):
+ raise _UnsafeNativeMetricInputError(
+ "Sunburst native metric must be an exact object"
+ )
+ if isinstance(value, ColumnRef):
+ return value
+ raise ValueError("Sunburst native metric must be a string or object")
+
+
+def _replace_unsafe_native_metric(
+ data: dict[str, Any], original_data: dict[str, Any] | None, key: str
+) -> None:
+ """Remove one hook-bearing value from working and Pydantic input mappings."""
+ dict.__setitem__(data, key, 0)
+ if original_data is not None:
+ dict.__setitem__(original_data, key, 0)
+
+
+def _native_column_meta_dict_children(
+ value: dict[Any, Any], depth: int
+) -> tuple[list[tuple[Any, int]], int]:
+ """Read exact-dict children while validating bounded primitive keys."""
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata object is too large")
+ children: list[tuple[Any, int]] = []
+ string_bytes = 0
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ raise ValueError(
+ "Sunburst native metric column metadata keys must be strings"
+ )
+ string_bytes += _native_column_meta_string_bytes(key, key=True)
+ children.append((nested, depth + 1))
+ return children, string_bytes
+
+
+def _native_column_meta_list_children(
+ value: list[Any], depth: int
+) -> list[tuple[Any, int]]:
+ """Read exact-list children without dispatching subclass hooks."""
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata array is too large")
+ return [
+ (list.__getitem__(value, index), depth + 1)
+ for index in range(list.__len__(value))
+ ]
+
+
+def _validate_native_column_meta_scalar(value: Any) -> int:
+ """Validate one exact JSON scalar and return its string-byte contribution."""
+ value_type = type(value)
+ if value_type is str:
+ return _native_column_meta_string_bytes(value)
+ if value_type is int:
+ if int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ raise ValueError(
+ "Sunburst native metric column metadata integer is too large"
+ )
+ return 0
+ if value_type is float:
+ if not math.isfinite(value):
+ raise ValueError(
+ "Sunburst native metric column metadata number must be finite"
+ )
+ return 0
+ if value_type not in (bool, type(None)):
+ raise ValueError(
+ "Sunburst native metric column metadata contains an invalid value"
+ )
+ return 0
+
+
+def _validate_bounded_native_column_meta(value: Any) -> Any:
+ """Validate an extensible frontend ColumnMeta object without invoking hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric column must be an object")
+
+ total_values = 0
+ total_string_bytes = 0
+ stack: list[tuple[Any, int]] = [(value, 0)]
+ while stack:
+ item, depth = stack.pop()
+ total_values += 1
+ if total_values > _NATIVE_COLUMN_META_MAX_TOTAL_VALUES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ if depth > _NATIVE_COLUMN_META_MAX_DEPTH:
+ raise ValueError(
+ "Sunburst native metric column metadata is too deeply nested"
+ )
+
+ item_type = type(item)
+ if item_type is dict:
+ children, key_bytes = _native_column_meta_dict_children(item, depth)
+ total_string_bytes += key_bytes
+ stack.extend(children)
+ elif item_type is list:
+ stack.extend(_native_column_meta_list_children(item, depth))
+ else:
+ total_string_bytes += _validate_native_column_meta_scalar(item)
+
+ if total_string_bytes > _NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ return value
+
+
+class SunburstNativeMetricColumn(BaseModel):
+ """Owned projection of the frontend's intentionally extensible ColumnMeta."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ type: StrictStr | None = Field(None, max_length=255)
+
+ @model_validator(mode="before")
+ @classmethod
+ def validate_bounded_column_meta(cls, value: Any) -> Any:
+ """Bound the full open object before projecting the fields we own."""
+ value = _validate_bounded_native_column_meta(value)
+ snake_name = dict.get(value, "column_name", _NATIVE_METRIC_MISSING)
+ camel_name = dict.get(value, "columnName", _NATIVE_METRIC_MISSING)
+ if (
+ snake_name is not _NATIVE_METRIC_MISSING
+ and camel_name is not _NATIVE_METRIC_MISSING
+ ):
+ if type(snake_name) is not str or type(camel_name) is not str:
+ raise ValueError(
+ "column_name and columnName must both be exact strings"
+ )
+ if snake_name != camel_name:
+ raise ValueError(
+ "column_name and columnName must match when both are provided"
+ )
+ return value
+
+
+class XYNativeMetricColumn(UnknownFieldCheckMixin):
+ """Closed legacy ColumnMeta projection retained for native XY axes."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ id: StrictInt | None = Field(None, ge=0)
+ type: StrictStr | None = Field(None, max_length=255)
+ type_generic: StrictInt | None = Field(None, ge=0, le=10)
+ groupby: StrictBool | None = None
+ is_dttm: StrictBool | None = None
+ filterable: StrictBool | None = None
+ verbose_name: StrictStr | None = Field(None, max_length=500)
+ description: StrictStr | None = Field(None, max_length=2000)
+ expression: StrictStr | None = Field(None, max_length=2000)
+ database_expression: StrictStr | None = Field(None, max_length=2000)
+ python_date_format: StrictStr | None = Field(None, max_length=255)
+ option_name: StrictStr | None = Field(
+ None,
+ max_length=500,
+ validation_alias=AliasChoices("option_name", "optionName"),
+ )
+ filter_by: StrictStr | None = Field(
+ None,
+ max_length=255,
+ validation_alias=AliasChoices("filter_by", "filterBy"),
+ )
+ value: StrictStr | None = Field(None, max_length=500)
+ advanced_data_type: StrictStr | None = Field(None, max_length=255)
+ uuid: StrictStr | None = Field(None, max_length=64)
+
+
+class SunburstLegacySavedFields(BaseModel):
+ """Known obsolete controls tolerated only while reading saved Sunbursts."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ country_fieldtype: StrictStr | None = Field(None, max_length=255)
+ entity: StrictStr | None = Field(None, max_length=255)
+ granularity: StrictStr | None = Field(None, max_length=255)
+ limit: StrictInt | StrictStr | None = None
+ markup_type: Literal["markdown", "html"] | None = None
+ show_bubbles: StrictBool | None = None
+
+ @field_validator("limit")
+ @classmethod
+ def validate_legacy_limit(cls, value: int | str | None) -> int | str | None:
+ """Bound the ignored legacy row limit and reject non-numeric strings."""
+ if value is None:
+ return None
+ if isinstance(value, str):
+ if not value.isascii() or not value.isdigit():
+ raise ValueError("legacy Sunburst limit must be an integer")
+ numeric_value = int(value)
+ else:
+ numeric_value = value
+ if not 1 <= numeric_value <= 50000:
+ raise ValueError("legacy Sunburst limit must be between 1 and 50000")
+ return value
+
+
+_NATIVE_FORM_DATA_MARKER = "_mcp_native_form_data"
+_SUNBURST_IGNORED_LEGACY_FIELDS = frozenset(
+ {
+ "country_fieldtype",
+ "entity",
+ "granularity",
+ "limit",
+ "markup_type",
+ "show_bubbles",
+ }
+)
+
+
+class SunburstChartConfig(BaseChartConfig):
+ """Config for the ECharts Sunburst plugin (viz_type ``sunburst_v2``).
+
+ ``hierarchy`` follows the frontend ``columns`` control: the first entry is
+ the innermost ring and each later entry adds a child level. The primary
+ metric sizes arcs; an optional secondary metric colors arcs by the
+ secondary/primary ratio.
+ """
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ chart_type: Literal["sunburst"] = "sunburst"
+ viz_type: Literal["sunburst_v2"] = Field(
+ "sunburst_v2",
+ description="Exact Superset frontend visualization tag",
+ )
+ hierarchy: List[ColumnRef] = Field(
+ ...,
+ min_length=1,
+ description=(
+ "Hierarchy dimensions in ring order, from the innermost/root level "
+ "to the outermost/leaf level"
+ ),
+ validation_alias=AliasChoices("hierarchy", "columns", "groupby"),
+ )
+ metric: ColumnRef = Field(
+ ...,
+ description=(
+ "Primary metric used to size arcs. Use aggregate for a SIMPLE "
+ "adhoc metric, saved_metric=True for a dataset metric, or "
+ "sql_expression plus label for a SQL metric."
+ ),
+ )
+ secondary_metric: ColumnRef | None = Field(
+ None,
+ description=(
+ "Optional metric used to color arcs by secondary/primary ratio. "
+ "When omitted, colors are categorical."
+ ),
+ validation_alias=AliasChoices("secondary_metric", "secondaryMetric"),
+ )
+ filters: List[FilterConfig] | None = Field(
+ None,
+ description=(
+ "Structured WHERE filters (column/op/value/clause). An omitted list is "
+ "preserved on updates; an explicit [] clears saved filters."
+ ),
+ )
+ time_range: str | None = Field(
+ None,
+ min_length=1,
+ max_length=1000,
+ description=(
+ "Superset time range, for example 'Last year', "
+ "'2025-01-01 : 2025-12-31', or 'No filter'"
+ ),
+ )
+ time_grain: TimeGrain | None = Field(
+ None,
+ description="Optional bucket for temporal hierarchy columns",
+ validation_alias=AliasChoices("time_grain", "time_grain_sqla"),
+ )
+ sort_by_metric: bool = Field(
+ False,
+ description=(
+ "Order hierarchy rows by the primary metric descending before "
+ "applying row_limit, matching the frontend buildQuery transform"
+ ),
+ )
+ row_limit: int = Field(10000, description="Maximum hierarchy rows", ge=1, le=50000)
+ color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Categorical scheme used when secondary_metric is omitted",
+ )
+ linear_color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Sequential scheme used when secondary_metric is present",
+ )
+ show_labels: bool = False
+ show_labels_threshold: float = Field(
+ 5,
+ ge=0,
+ le=100,
+ description="Minimum arc size in percentage points for showing a label",
+ )
+ show_total: bool = False
+ show_null_values: bool = Field(
+ True,
+ description="Keep null-valued hierarchy nodes in the rendered tree",
+ )
+ label_type: Literal["key", "value", "key_value"] = "key"
+ number_format: str = Field("SMART_NUMBER", min_length=1, max_length=50)
+ date_format: str = Field("smart_date", min_length=1, max_length=50)
+ currency_format: CurrencyFormat | None = None
+ # Bounded native Explore envelope/UI state. These keys occur in real saved
+ # form_data and are safe to round-trip, but remain typed so native mode does
+ # not become an escape hatch for arbitrary plugin keys or malformed nesting.
+ annotation_layers: list[dict[str, JsonValue]] | None = None
+ dashboard_id: int | None = Field(
+ None,
+ validation_alias=AliasChoices("dashboard_id", "dashboardId"),
+ serialization_alias="dashboardId",
+ )
+ dashboards: list[int] | None = None
+ datasource: str | None = Field(None, min_length=1, max_length=255)
+ extra_form_data: dict[str, JsonValue] | None = None
+ slice_id: int | None = None
+ url_params: dict[str, JsonValue] | None = None
+ standardized_form_data: SunburstStandardizedFormData | None = Field(
+ None,
+ validation_alias=AliasChoices("standardized_form_data", "standardizedFormData"),
+ serialization_alias="standardizedFormData",
+ )
+ since: str | None = Field(None, max_length=1000)
+ until: str | None = Field(None, max_length=1000)
+ time_compare: str | list[str] | None = None
+ compare_lag: int | str | None = None
+ compare_suffix: str | None = Field(None, max_length=255)
+
+ @staticmethod
+ def _looks_like_native_form_data(data: Any) -> bool:
+ """Identify saved Explore payloads without weakening typed typo checks."""
+ if type(data) is not dict:
+ return False
+ native_marker = dict.get(data, _NATIVE_FORM_DATA_MARKER)
+ viz_type = dict.get(data, "viz_type")
+ if not (
+ (type(native_marker) is bool and native_marker)
+ or (type(viz_type) is str and viz_type == "sunburst_v2")
+ ):
+ return False
+ metric = dict.get(data, "metric")
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(metric)
+ )
+ return (
+ any(
+ key in data
+ for key in (
+ "adhoc_filters",
+ "annotation_layers",
+ "datasource",
+ "extra_form_data",
+ "since",
+ "slice_id",
+ "standardizedFormData",
+ "until",
+ )
+ )
+ or type(metric) is str
+ or expression_type is not _NATIVE_METRIC_MISSING
+ or has_native_discriminator
+ )
+
+ @staticmethod
+ def _coerce_native_metric(
+ value: Any, *, allow_extensible_column_meta: bool = True
+ ) -> Any:
+ """Accept saved metric names and native SIMPLE/SQL metric objects."""
+ if type(value) is str:
+ return {"name": value, "saved_metric": True}
+ if type(value) is dict:
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(value)
+ )
+ if expression_type is _NATIVE_METRIC_MISSING:
+ if has_native_discriminator:
+ raise ValueError("Sunburst native metric requires expressionType")
+ return value
+ value = _validate_native_metric_wrapper(value)
+ else:
+ return _validate_non_dict_native_metric(value)
+
+ expression_type = value.get("expressionType")
+ label = value.get("label")
+ has_custom_label = value.get("hasCustomLabel")
+ if expression_type == "SQL":
+ sql_expression = value.get("sqlExpression")
+ return {
+ "sql_expression": sql_expression,
+ # The frontend still assigns a result label when the label is
+ # not custom. Preserve a nonempty native label first, then use
+ # the SQL expression as the same effective fallback.
+ "label": label or sql_expression,
+ "has_custom_label": has_custom_label,
+ }
+ if expression_type == "SIMPLE":
+ column = value.get("column")
+ if type(column) is dict:
+ column_model = (
+ SunburstNativeMetricColumn
+ if allow_extensible_column_meta
+ else XYNativeMetricColumn
+ )
+ column_metadata = column_model.model_validate(column)
+ name = column_metadata.column_name
+ dtype = column_metadata.type
+ elif type(column) is str:
+ name = column
+ dtype = None
+ else:
+ raise ValueError(
+ "Sunburst SIMPLE metric column must be a column name or object"
+ )
+ return {
+ "name": name,
+ "aggregate": value.get("aggregate"),
+ "label": label,
+ "has_custom_label": has_custom_label,
+ "dtype": dtype,
+ }
+ return value
+
+ @staticmethod
+ def _coerce_native_hierarchy(data: dict[str, Any]) -> None:
+ """Normalize native ``columns``/``groupby`` dimension shortcuts."""
+ hierarchy_key = next(
+ (key for key in ("hierarchy", "columns", "groupby") if key in data),
+ None,
+ )
+ if hierarchy_key is None:
+ return
+ hierarchy = data[hierarchy_key]
+ if isinstance(hierarchy, (str, dict)):
+ hierarchy = [hierarchy]
+ if isinstance(hierarchy, list):
+ data[hierarchy_key] = [
+ {"name": item} if isinstance(item, str) else item for item in hierarchy
+ ]
+ if hierarchy_key != "groupby":
+ # Saved Sunburst payloads sometimes retain an empty groupby from a
+ # previous plugin. The explicit columns/hierarchy control wins.
+ data.pop("groupby", None)
+
+ @staticmethod
+ def _coerce_native_filter(
+ native_filter: Any, data: dict[str, Any]
+ ) -> dict[str, Any] | None:
+ """Convert one native SIMPLE filter, extracting temporal controls."""
+ if not isinstance(native_filter, dict):
+ raise ValueError("Each Sunburst native adhoc_filter must be an object")
+ allowed_keys = {
+ "clause",
+ "comparator",
+ "datasourceWarning",
+ "expressionType",
+ "filterOptionName",
+ "isExtra",
+ "operator",
+ "sqlExpression",
+ "subject",
+ }
+ if unknown := set(native_filter) - allowed_keys:
+ raise ValueError(
+ "Unknown Sunburst native adhoc_filter field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ if native_filter.get("expressionType") != "SIMPLE":
+ raise ValueError(
+ "Sunburst native adhoc_filters round-trip supports SIMPLE filters "
+ "only; express filters with the typed 'filters' field"
+ )
+ operator = native_filter.get("operator")
+ clause = native_filter.get("clause", "WHERE")
+ if not isinstance(clause, str) or clause != "WHERE":
+ raise ValueError(
+ "Sunburst native SIMPLE filter clause must be 'WHERE'; "
+ "SIMPLE HAVING is unsupported because the shared query mapper "
+ "cannot preserve HAVING semantics"
+ )
+ if operator == "TEMPORAL_RANGE":
+ if "temporal_column" not in data and "granularity_sqla" not in data:
+ data["temporal_column"] = native_filter.get("subject")
+ data.setdefault("time_range", native_filter.get("comparator"))
+ return None
+ mapped_operator = "=" if operator == "==" else operator
+ comparator = native_filter.get("comparator")
+ if mapped_operator in {"IS NULL", "IS NOT NULL"}:
+ comparator = None
+ return {
+ "column": native_filter.get("subject"),
+ "clause": clause,
+ "op": mapped_operator,
+ "value": comparator,
+ }
+
+ @staticmethod
+ def _discard_legacy_saved_fields(data: dict[str, Any]) -> None:
+ """Validate and ignore only the known obsolete saved-chart controls."""
+ legacy = {
+ key: data[key] for key in _SUNBURST_IGNORED_LEGACY_FIELDS if key in data
+ }
+ if legacy:
+ SunburstLegacySavedFields.model_validate(legacy)
+ for key in legacy:
+ data.pop(key, None)
+
+ @classmethod
+ def _coerce_native_filters(cls, data: dict[str, Any]) -> None:
+ """Round-trip native saved filters into the typed filter contract."""
+ native_filters = data.pop("adhoc_filters", None)
+ if native_filters is None or "filters" in data:
+ return
+ if not isinstance(native_filters, list):
+ raise ValueError("Sunburst native adhoc_filters must be a list")
+ filters = [
+ converted
+ for native_filter in native_filters or []
+ if (converted := cls._coerce_native_filter(native_filter, data)) is not None
+ ]
+ data["filters"] = filters
+
+ @model_validator(mode="before")
+ @classmethod
+ def accept_native_form_data(cls, data: Any) -> Any:
+ """Translate native saved Sunburst form_data into the typed contract."""
+ if not isinstance(data, dict):
+ return data
+ original_data = data if type(data) is dict else None
+ data = dict(data)
+ try:
+ is_native_form_data = cls._looks_like_native_form_data(data)
+ except _UnsafeNativeMetricInputError:
+ # Pydantic includes rejected input in its rendered error. Remove
+ # hook-bearing objects before handing control back to its validators.
+ _replace_unsafe_native_metric(data, original_data, "metric")
+ is_native_form_data = True
+ data.pop(_NATIVE_FORM_DATA_MARKER, None)
+
+ if is_native_form_data:
+ cls._discard_legacy_saved_fields(data)
+ cls._coerce_native_hierarchy(data)
+
+ for key in ("metric", "secondary_metric", "secondaryMetric"):
+ if key in data and data[key] is not None:
+ try:
+ data[key] = cls._coerce_native_metric(data[key])
+ except _UnsafeNativeMetricInputError:
+ _replace_unsafe_native_metric(data, original_data, key)
+
+ # Native saved filters can be round-tripped when they use the SIMPLE
+ # controls. SQL filter clauses intentionally remain outside the typed
+ # MCP filter contract rather than being accepted as arbitrary SQL.
+ cls._coerce_native_filters(data)
+
+ if "time_range" not in data and (data.get("since") or data.get("until")):
+ data["time_range"] = (
+ f"{data.get('since') or ''} : {data.get('until') or ''}"
+ )
+ if "temporal_column" not in data and "granularity_sqla" in data:
+ data["temporal_column"] = data["granularity_sqla"]
+ data.pop("granularity_sqla", None)
+ # Native mode intentionally accepts only the explicitly typed keys
+ # above. Leave unknown keys in place so UnknownFieldCheckMixin rejects
+ # typos and stale type-specific payload rather than silently dropping it.
+ return data
+
+ @field_validator("time_range")
+ @classmethod
+ def sanitize_time_range(cls, value: str | None) -> str | None:
+ """Sanitize a human-readable Superset time-range expression."""
+ return sanitize_user_input(
+ value,
+ "Time range",
+ max_length=1000,
+ allow_empty=True,
+ )
+
+ @model_validator(mode="after")
+ def validate_roles(self) -> "SunburstChartConfig":
+ """Enforce dimension/metric roles and unique query output labels."""
+ hierarchy_names: list[str] = []
+ for index, dimension in enumerate(self.hierarchy):
+ _reject_sql_expression_on_dimension(dimension, f"hierarchy[{index}]")
+ if dimension.saved_metric or dimension.aggregate:
+ raise ValueError(
+ f"hierarchy[{index}] must be a physical dimension column, "
+ "not a metric"
+ )
+ hierarchy_names.append(dimension.name or "")
+
+ folded_hierarchy = [name.casefold() for name in hierarchy_names]
+ if len(folded_hierarchy) != len(set(folded_hierarchy)):
+ raise ValueError("Sunburst hierarchy dimensions must be unique")
+
+ metrics = [("metric", self.metric)]
+ if self.secondary_metric is not None:
+ metrics.append(("secondary_metric", self.secondary_metric))
+ for role, metric in metrics:
+ if not metric.is_metric:
+ raise ValueError(
+ f"{role} must define aggregate, saved_metric=True, or "
+ "sql_expression with label"
+ )
+
+ output_labels: list[tuple[str, str]] = [
+ (f"hierarchy[{index}]", name) for index, name in enumerate(hierarchy_names)
+ ]
+ output_labels.extend(
+ (role, _metric_display_label(metric)) for role, metric in metrics
+ )
+ seen: dict[str, str] = {}
+ for role, label in output_labels:
Review Comment:
This rejects a valid Sunburst configuration when `secondary_metric` is the same metric as `metric`, even though the frontend deliberately supports that mode by falling back to categorical colors. Could this duplicate-label check allow the primary/secondary pair to share the same metric?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5560883172
Finite maintainer disposition of Bito run #686bfa (six additional suggestions), implemented/reverified in bc8ac02f9a0dd3da8bdc3909079de1a99459efad:
- **Extreme fractional Decimal profiling: addressed.** Bounded exponent/digit values retain an exact oversized-fraction identity instead of discarding all column sampling. `test_bounded_extreme_decimals_preserve_other_columns` checks both exponent signs and unrelated-column metadata.
- **ASCII dimensions: addressed.** Rendered content is bounded by the requested width/height, with `test_ascii_preview_content_respects_requested_dimensions`.
- **Aggregate alias comparison: addressed.** STDDEV/STDDEV_SAMP and VAR/VAR_SAMP compare using their mapped query semantics; `test_same_sunburst_metric_accepts_aggregate_aliases` covers both pairs.
- **Stubbed repeated-metric validation: no production change needed.** The test's stub explicitly calls the real `GenerateChartRequest.model_validate(payload)`, so it does exercise `SunburstChartConfig.validate_roles`; it stubs the unrelated dataset pipeline, not Pydantic role validation. The registered FastMCP test and direct native request/update/explore matrices remain passing.
- **Repeated pure-function call: addressed.** Primary metric identity is computed once.
- **Ineffective SQL test monkeypatch: removed.** Dedicated complete-response boundary tests remain in place.
Focused validation: **776 passed**. SQL/query suites: **51 passed**. Full branch-diff pre-commit (including MyPy, Ruff, Pylint and workflow checks) and staged pre-commit pass. Master was merged normally in `4dde97173719e8015647e59709023fee12f85f88`; Gauge registration/query-ordering and shared normalization contracts are preserved. Bullet #43770 was inspected read-only at `703bb6cd8c9a96c19e0e76c0b1e3ec968ec26489`; no Bullet branch/file/commit/PR was changed. No additional review session requested; final exact-head CI is being tracked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3952052043
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -0,0 +1,1525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+
+import math
+import time as system_time
+from bisect import bisect_right
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
+from decimal import Decimal
+from enum import Enum
+from typing import Any
+from uuid import UUID
+from zoneinfo import ZoneInfo
+
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
+from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
+
+FAILED_QUERY_STATUSES = frozenset(
+ {"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
+)
+
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
+ return None
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
+
+
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
Review Comment:
Fixed in 9f7a3a7ac6840564732d93b0c845bd9fa88704fa. The timestamp path accepts the existing hook-free pytz named-zone normalization, including static generated zones; the canonical UTC singleton is normalized explicitly. Stored wall/instant offsets and nanoseconds remain intact. Regressions cover UTC, US/Pacific, Etc/GMT-3, and both New York DST-fold occurrences with nanoseconds. Focused suite: 845 passed; staged and branch-wide pre-commit passed. Exact-head CI is running.
##########
superset/mcp_service/chart/preview_utils.py:
##########
@@ -160,11 +205,17 @@ def _generate_ascii_preview_from_data(
content = _generate_safe_ascii_line_chart(data)
elif viz_type == "pie":
content = _generate_safe_ascii_pie_chart(data)
+ elif viz_type == "sunburst_v2":
Review Comment:
Fixed in 9f7a3a7ac6840564732d93b0c845bd9fa88704fa. Sunburst reserves two headers and, when needed, a footer before selecting rows (17 rows plus a notice at default height for larger results). Tiny canvases retain an omission notice. Display-column-aware clipping marks truncated lines, and embedded line breaks cannot consume the footer. Regressions cover 1โ25-line canvases, 1โ30 rows, CJK/emoji/combining characters, and clipped paths with an intact footer. Focused suite: 845 passed; pre-commit passed. Exact-head CI is running.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,6 +461,1156 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
+) -> dict[str, Any]:
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
+
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
+ """
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
+
+
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
+ query: dict[str, Any] = {
+ "columns": columns,
+ "metrics": metrics,
+ }
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
+ if extras:
+ query["extras"] = extras
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": form_data.get("time_grain_sqla"),
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ query.setdefault("metrics", []).append(sort_metric)
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ )
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ explicit_x_axis = _frontend_x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
+ query["columns"] = time_column
+ if not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
+ query["post_processing"] = post_processing
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
Review Comment:
Fixed in 9f7a3a7ac6840564732d93b0c845bd9fa88704fa at the Excel rebuild caller: a copied form-data payload sets result_format=xlsx before shared query construction. This preserves the configured row limit and excludes interactive page/count queries without changing interactive MCP query semantics or saved chart params. Aggregate/raw Table regressions assert one query, the full 1000-row limit rather than page length 10, no count query, and unchanged saved params. Focused suite: 845 passed; pre-commit passed. Exact-head CI is running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4225739260
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1784,799 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "series_limit_metric": (("series_limit_metric",), ("timeseries_limit_metric",)),
+ "timeseries_limit_metric": (
+ ("series_limit_metric",),
+ ("timeseries_limit_metric",),
+ ),
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "groupby": (("group_by",),),
+ "groupby_b": (("group_by_secondary",),),
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
Review Comment:
Confirmed and fixed in [e7d7b41af2](https://github.com/apache/superset/commit/e7d7b41af23775a45b97f75fc252725dd5d966a9). Bubble now has modeled update controls for `series`, `row_limit`, and `color_scheme`, so omitted fields retain saved values while explicit replacements and `series: null` take effect. `test_bubble_update_preserves_omitted_controls_and_applies_clears` exercises the actual update merge, including dataset-rebind and cross-viz isolation.
##########
superset/mcp_service/chart/plugins/mixed_timeseries.py:
##########
@@ -28,20 +28,36 @@
map_mixed_timeseries_config,
)
from superset.mcp_service.chart.plugin import BaseChartPlugin
-from superset.mcp_service.chart.schemas import ColumnRef, MixedTimeseriesChartConfig
+from superset.mcp_service.chart.schemas import (
+ ChartError,
+ ColumnRef,
+ MixedTimeseriesChartConfig,
+ VegaLitePreview,
+)
from superset.mcp_service.chart.validation.dataset_validator import DatasetValidator
from superset.mcp_service.common.error_schemas import ChartGenerationError
class MixedTimeseriesChartPlugin(BaseChartPlugin):
"""Plugin for mixed_timeseries chart type."""
+ query_role_keys = BaseChartPlugin.query_role_keys
chart_type = "mixed_timeseries"
display_name = "Mixed Timeseries"
native_viz_types: ClassVar[Mapping[str, str]] = {
"mixed_timeseries": "Mixed Timeseries Chart",
}
+ def vega_lite_preview(
Review Comment:
Confirmed and fixed in [e7d7b41af2](https://github.com/apache/superset/commit/e7d7b41af23775a45b97f75fc252725dd5d966a9). Mixed Timeseries sets `resizes_saved_preview = True`. The saved-timeseries regression covers default dimensions, explicit 800ร200 dimensions, and the saved chart description, while checking that unsaved previews retain their container sizing.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -2158,6 +3046,56 @@ def map_mixed_timeseries_config(
_add_adhoc_filters(form_data, config.filters)
+ fields_set = config.model_fields_set
+ if "adhoc_filters_b" in fields_set:
+ form_data["adhoc_filters_b"] = config.adhoc_filters_b
Review Comment:
Confirmed and fixed in [e7d7b41af2](https://github.com/apache/superset/commit/e7d7b41af23775a45b97f75fc252725dd5d966a9). Tier-1 validation now walks the secondary query form data supplied by the owning plugin, applying the same clause and column checks to query B. Regressions cover SIMPLE HAVING, malformed clauses, missing columns, and valid WHERE filters with compile checks both enabled and disabled; invalid filters are rejected before query execution.
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1548 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+import base64
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.utils.serialization import BINARY_PREFIX
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _canonical_binary(
+ value: bytes | bytearray | memoryview,
+) -> tuple[str | None, str | None]:
+ """Render exact binary cells as UTF-8 text or ``base64:``-prefixed text.
+
+ Matches the MCP response serializer: valid UTF-8 is returned as text and
+ anything else is losslessly base64-encoded. The raw size is bounded before
+ copying so the rendered string stays within the per-cell text budget.
+ """
+ size = value.nbytes if type(value) is memoryview else len(value)
+ if size > MAX_RESULT_STRING_LENGTH:
+ return None, "an oversized binary value"
+ try:
+ raw = bytes(value)
+ except (BufferError, TypeError, ValueError):
+ return None, "an invalid binary value"
+ try:
+ text = raw.decode("utf-8")
+ except UnicodeDecodeError:
+ text = BINARY_PREFIX + base64.b64encode(raw).decode("ascii")
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized binary value"
+ return text, None
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ return (value, None) if math.isfinite(value) else (None, None)
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+ if value_type is bytes or value_type is bytearray or value_type is memoryview:
+ return _canonical_binary(value)
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, None)
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
Review Comment:
Confirmed and fixed in [e7d7b41af2](https://github.com/apache/superset/commit/e7d7b41af23775a45b97f75fc252725dd5d966a9). Concrete psycopg2 and psycopg range types receive bounded driver-style text normalization, including nested range arrays. Endpoints are validated without arbitrary conversion hooks, subclasses remain rejected, and rendered text uses the existing cell and aggregate byte budgets. The saved raw Table regression runs a first-NULL/second-range-array DataFrame through producer conversion and the FastMCP JSON response; additional tests cover both drivers, temporal/decimal/empty/unbounded ranges, hostile endpoints, and size limits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4220719622
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -1480,19 +1488,89 @@ def _write_excel_headers(ws: Any, columns: List[str]) -> None:
ws.cell(row=1, column=idx, value=col)
-def _write_excel_data(ws: Any, data: List[Dict[str, Any]], columns: List[str]) -> None:
+def _export_scalar(value: Any) -> Any:
+ """Project a validated result scalar to CSV/Excel writer-safe values."""
+ if value is None or (isinstance(value, float) and not math.isfinite(value)):
Review Comment:
Non-finite floats are exported as an empty string here (and are already turned into `None` in `_normalize_scalar`), so `inf` in a CSV/XLSX export is indistinguishable from SQL NULL. `docs/admin_docs/configuration/mcp-server.mdx` (Gauge section) says exports write `nan`/`inf`/`-inf` as text to keep them distinct from blank NULL cells. A Gauge row with `score = inf` exports a blank cell today. Is the blank intended, in which case the docs line should change, or should the text form be preserved for exports?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1784,801 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "series_limit_metric": (("series_limit_metric",), ("timeseries_limit_metric",)),
+ "timeseries_limit_metric": (
+ ("series_limit_metric",),
+ ("timeseries_limit_metric",),
+ ),
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "groupby": (("group_by",),),
+ "groupby_b": (("group_by_secondary",),),
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ requested_filters = getattr(config, "filters", None)
+ if requested_filters is not None:
+ # Legacy ``filters``/``where``/``having`` are reconstructed into adhoc
+ # filters at query time, so an explicit clear or replacement removes
+ # every saved source the new form data does not set itself.
+ for key in _SAVED_PREDICATE_FORM_DATA_KEYS - new_form_data.keys():
+ merged.pop(key, None)
+ else:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool(
Review Comment:
Passing only `time_grain` on a same-viz update now sets `drop_existing_temporal`, so `_merge_preserved_adhoc_filters` removes every saved `TEMPORAL_RANGE` adhoc filter, not just a stale generated binding. With a saved XY chart that has an `ds TEMPORAL_RANGE "Last year"` filter, resending the same x/y roles with `time_grain="P1M"` and no `filters` or `time_range` replaces that filter with the mapper's neutral binding. The update succeeds but the chart then queries all dates. Should the saved range be kept unless `time_range` or `temporal_column` is also supplied?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +427,1295 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are normalized for old server/native payloads.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Normalize saved physical references before the frontend column predicate."""
+ x_axis = _x_axis_column(form_data)
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
Review Comment:
Compile validation reuses this pipeline with `compile_row_limit` of 2 for XY charts, so rolling and forecast now run on a two-row sample. A saved line chart with `rolling_type="mean"`, `rolling_periods=7`, `min_periods=7` and `forecastEnabled` leaves no rows after the rolling step, and Prophet then raises "Forecast requires at least 2 data points". An update to a chart that renders fine in Explore gets rejected at compile. Should the compile sample skip these analytics, or be sized to what they need?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3972016420
##########
superset/mcp_service/chart/tool/get_chart_preview.py:
##########
@@ -220,6 +232,30 @@ def _authorize_guest_query(self, query_context: Any) -> None:
if (dashboard_id := guest_scope.guest_dashboard_id(self.chart)) is not None:
guest_scope.authorize_query(query_context, dashboard_id, self.chart)
+ def _canonical_form_data(self, form_data: dict[str, Any]) -> dict[str, Any]:
+ """Bind saved-preview identity to the resolved chart and datasource."""
+ canonical = canonicalize_operation_form_data(
+ form_data,
+ datasource_id=self.chart.datasource_id,
+ datasource_type=self.chart.datasource_type,
+ chart_id=self.chart.id,
+ )
+ if self.chart.viz_type == "gauge_chart":
Review Comment:
Addressed: saved preview canonicalization stamps `chart.viz_type` for every chart type, not only Gauge. Existing parameterized Gauge stale/missing-viz tests and Sunburst preview coverage exercise the shared path; MyPy and Ruff pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4189744171
##########
superset/mcp_service/chart/chart_helpers.py:
##########
@@ -853,55 +1206,28 @@ def build_query_dicts_from_form_data(
if plugin_queries is not None:
return plugin_queries
- metrics, groupby = resolve_metrics_and_groupby(form_data, chart)
-
# Deck.gl charts use spatial column configs rather than the standard
# metrics / groupby fields. Extract columns from the spatial controls.
if viz_type.startswith("deck_"):
- deck_columns = resolve_deck_gl_columns(form_data)
- deck_metrics = _resolve_deck_gl_metrics(form_data, viz_type)
- qd = build_single_query_dict(
- form_data,
- deck_columns,
- deck_metrics,
- row_limit=row_limit,
- order_desc=order_desc,
- )
- if deck_metrics:
- # Mirror BaseDeckGLViz.query_obj(): order by first metric descending
- qd["orderby"] = [(deck_metrics[0], not form_data.get("order_desc", True))]
- if viz_type in _DECK_TIMESERIES_VIZ_TYPES and (
- time_grain := form_data.get("time_grain_sqla")
- ):
- qd["is_timeseries"] = True
- qd["granularity"] = form_data.get("granularity_sqla")
- qd.setdefault("extras", {})["time_grain_sqla"] = time_grain
- if form_data.get("filter_nulls", True):
- null_filters = _deck_gl_null_filters(form_data)
- if null_filters:
- qd["filters"] = [*(qd.get("filters") or []), *null_filters]
- return [qd]
-
- if viz_type.startswith("echarts_timeseries"):
- groupby = with_x_axis_column(form_data, groupby)
-
- return [
- build_single_query_dict(
+ from superset.common.form_data_query_context import normalize_time_column
+
+ qd = build_query_objects_from_form_data(
form_data,
- groupby,
- metrics,
+ viz_type=viz_type,
row_limit=row_limit,
order_desc=order_desc,
- )
- ]
-
-
-def with_x_axis_column(form_data: dict[str, Any], groupby: list[Any]) -> list[Any]:
- """Prepend a time-series chart's x-axis column to its query columns."""
- x_axis_col = extract_x_axis_col(form_data)
- if x_axis_col and x_axis_col not in groupby:
- return [x_axis_col, *groupby]
- return groupby
+ filters_prepared=True,
+ )[0]
+ qd = _deck_query_adapter(form_data, qd, viz_type)
+ return [normalize_time_column(form_data, qd)]
+ return build_query_objects_from_form_data(
Review Comment:
Rebuilding a saved World Map or Country Map without `query_context` now drops its `entity` column, so a chart with `entity="country_code"` and `metric="count"` returns one dataset-wide total instead of per-country values. Could the shared builder preserve the native map grouping before replacing the old resolver?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1773,741 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(existing_form_data)
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ if getattr(config, "filters", None) == []:
+ merged.pop("adhoc_filters", None)
+ elif getattr(config, "filters", None) is None:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool(
+ {"temporal_column", "time_grain", "time_range"} & fields_set
+ ),
+ )
+ if filters is not None:
+ merged["adhoc_filters"] = filters
+ return merged
+
+
+def retain_mixed_timeseries_secondary_update_state(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Return saved query-B controls a same-viz Mixed Timeseries update keeps.
+
+ Query B inherits unsuffixed controls only when the suffixed key is absent.
+ Preserve explicit native clears for controls the typed mapper did not
+ replace, so []/None never turns into accidental inheritance from query A.
+ Valid comparison state is also retained; malformed/stale dataset roles
+ remain fail-closed and are dropped.
+ """
+ from superset.common.form_data_query_context import (
+ MIXED_TIMESERIES_SECONDARY_QUERY_KEYS,
+ )
+
+ retained: Dict[str, Any] = {}
+ for key in MIXED_TIMESERIES_SECONDARY_QUERY_KEYS:
+ if key in new_form_data or key not in existing_form_data:
+ continue
+ value = existing_form_data[key]
+ is_explicit_clear = value is None or value in ([], {}, "")
+ is_valid_comparison = key == "comparison_type_b" and value in {
+ "values",
+ "difference",
+ "percentage",
+ "ratio",
+ }
+ is_valid_list_state = key in {
+ "adhoc_filters_b",
+ "annotation_layers_b",
+ "time_compare_b",
+ } and isinstance(value, list)
+ if is_explicit_clear or is_valid_comparison or is_valid_list_state:
+ retained[key] = value
+ return retained
+
+
+def finalize_sunburst_update_form_data( # noqa: C901
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ merged: Dict[str, Any],
+ config: SunburstChartConfig,
+) -> Dict[str, Any]:
+ """Apply Sunburst omission, explicit-clear and temporal-pair semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ temporal_fields = {"time_grain", "temporal_column"}
+ for field_name, form_key in _SUNBURST_UPDATE_FIELD_KEYS.items():
+ if field_name in temporal_fields:
+ continue
+ if field_name not in fields_set:
+ if same_viz:
+ if form_key in existing_form_data:
+ merged[form_key] = existing_form_data[form_key]
+ else:
+ merged.pop(form_key, None)
+ continue
+
+ value = getattr(config, field_name)
+ if value is None:
+ merged.pop(form_key, None)
+ if field_name == "time_range":
+ # The generic query-context mapper reconstructs time_range from
+ # these legacy keys. A clear must remove all three sources.
+ merged.pop("since", None)
+ merged.pop("until", None)
+
+ # A null temporal control is an atomic clear. Apply it after every merge so
+ # cached/native aliases and extra-form-data overrides cannot recreate time
+ # state in QueryContextFactory.
+ explicit_temporal_clear = any(
+ field_name in fields_set and getattr(config, field_name) is None
+ for field_name in ("temporal_column", "time_grain", "time_range")
+ )
+ if explicit_temporal_clear:
+ merged = _scrub_temporal_form_data(merged)
+
+ # Merge the temporal subject and grain as one final-state control pair.
+ temporal_set = "temporal_column" in fields_set
+ grain_set = "time_grain" in fields_set
+ if temporal_set and config.temporal_column is not None:
+ merged["granularity_sqla"] = config.temporal_column
+
+ if grain_set and config.time_grain is not None:
+ merged["time_grain_sqla"] = config.time_grain
+
+ if "time_range" in fields_set and config.time_range is not None:
+ merged["time_range"] = config.time_range
+
+ for key in ("extra_form_data", "standardizedFormData", "url_params"):
+ if merged.get(key) is None:
+ merged.pop(key, None)
+
+ # Do not discard an orphan grain here. The final-form-data validator must
+ # see and reject it consistently across immediate, preview-first, and
+ # cached update paths instead of letting the query builder silently ignore
+ # a request that cannot be represented by the frontend contract.
+ return merged
+
+
+_DATASET_BOUND_FORM_DATA_KEYS = frozenset(
Review Comment:
A cached Table rebound to another dataset still retains top-level `where` and `having`, so an old `where="region = 'EMEA'"` silently restricts the replacement when the new dataset also has `region`. Could the dataset-bound scrub remove these legacy SQL predicate sources as well?
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -90,6 +102,70 @@ class _ChartFacts(NamedTuple):
datasource_type: str | None = None
+def _is_expected_json_load_error(exc: Exception) -> bool:
+ """Recognize only the exact exceptions emitted for ordinary JSON failures."""
+ return type(exc) in {TypeError, ValueError, JSONDecodeError}
+
+
+def _requested_filter_columns(extra_form_data: dict[str, Any] | None) -> set[str]:
+ """Return simple column names explicitly requested through extra form data."""
+ if not extra_form_data:
+ return set()
+
+ columns: set[str] = set()
+ for filter_ in extra_form_data.get("filters", []):
Review Comment:
An accepted `extra_form_data={"filters": null}` request now raises `TypeError` in this post-query check and returns `InternalError` instead of chart data; the shared helper already treats null lists as absent. Could this keep that null-list handling for both `filters` and `adhoc_filters`?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1773,741 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
Review Comment:
Rebinding a saved `word_cloud` to another dataset with a complete Sunburst config still scrubs the source visualization first and raises the dataset-only rebind error because that source has no registered role contract. Could a complete cross-viz replacement start from the mapped target rather than require a source role contract?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1416 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
Review Comment:
These mandatory result limits ignore `MCP_RESPONSE_SIZE_CONFIG`, but the troubleshooting guide still tells operators to raise `max_bytes` or disable that guard to resolve size failures; even a single 65,537-character cell fails after either change. Could the guide distinguish the mandatory row/cell/aggregate limits from the configurable middleware guard?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1416 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
Review Comment:
Normalizing temporal values to strings here also changes Excel exports: a SQL DATE such as `2025-01-02` is now written as text with `General` formatting instead of an Excel date cell, breaking spreadsheet date grouping and calculations. Could validated temporal values remain typed until the format-specific export projection?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1416 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
Review Comment:
Queries containing ordinary binary cells now fail with `InvalidQueryResult` before the existing UTF-8/base64 serializer runs; for example, a returned `b"\xff"` can no longer become `"base64:/w=="`. Could exact built-in `bytes`, `bytearray`, and `memoryview` values be normalized within the safety budgets instead of rejected?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1773,741 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(existing_form_data)
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
Review Comment:
A Table-to-Sunburst replacement with explicit `filters=[]` removes only `adhoc_filters`; saved legacy `filters` survive the allowlisted merge and are reconstructed for execution, so an EMEA-only chart remains filtered after the requested clear. Could the explicit clear remove every preserved predicate representation?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1416 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = list.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
Review Comment:
Every FULL payload includes one `indexnames` entry per row, so this 4,096-item metadata limit rejects an otherwise valid 4,097-row chart result even though the data-row limit is 50,000. Could index metadata use the row budget, with coverage built from a real FULL producer payload?
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -1634,6 +1652,1045 @@ def reject_metric_style_groupby(self) -> "TreemapChartUpdateConfig":
return self
+class SunburstStandardizedControls(UnknownFieldCheckMixin):
+ """Bounded shared controls retained by Explore across viz changes."""
+
+ model_config = ConfigDict(extra="ignore")
+
+ metrics: list[JsonValue] = Field(default_factory=list)
+ columns: list[JsonValue] = Field(default_factory=list)
+
+
+class SunburstStandardizedFormData(UnknownFieldCheckMixin):
+ """Validated shape of Explore's cross-plugin UI memory."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ controls: SunburstStandardizedControls
+ memorized_form_data: list[tuple[str, dict[str, JsonValue]]] = Field(
+ default_factory=list,
+ validation_alias=AliasChoices("memorized_form_data", "memorizedFormData"),
+ serialization_alias="memorizedFormData",
+ )
+
+
+_NATIVE_COLUMN_META_MAX_DEPTH = 8
+_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS = 128
+_NATIVE_COLUMN_META_MAX_TOTAL_VALUES = 512
+_NATIVE_COLUMN_META_MAX_KEY_BYTES = 1_024
+_NATIVE_COLUMN_META_MAX_STRING_BYTES = 16 * 1_024
+_NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES = 64 * 1_024
+_NATIVE_COLUMN_META_MAX_INT_BITS = 64
+_NATIVE_METRIC_ALLOWED_KEYS = frozenset(
+ {
+ "aggregate",
+ "column",
+ "datasourceWarning",
+ "expressionType",
+ "hasCustomLabel",
+ "label",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_EXPRESSION_TYPES = frozenset({"SIMPLE", "SQL"})
+_NATIVE_METRIC_DISCRIMINATOR_KEYS = frozenset(
+ {
+ "column",
+ "datasourceWarning",
+ "hasCustomLabel",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_MISSING = object()
+_NATIVE_METRIC_INPUT_KEYS = frozenset(
+ {"metric", "metrics", "secondary_metric", "secondaryMetric"}
+)
+_NATIVE_METRIC_INVALID_VALUE = b"invalid native metric value"
+_NATIVE_METRIC_INVALID_KEY = "__invalid_native_metric_key__"
+_NATIVE_METRIC_SANITIZE_MAX_DEPTH = 16
+_NATIVE_METRIC_SANITIZE_MAX_VALUES = 1_024
+_NATIVE_METRIC_ERROR_STRING_LENGTH = 128
+
+
+def _native_column_meta_string_bytes(value: str, *, key: bool = False) -> int:
+ """Return exact UTF-8 size for a trusted built-in string."""
+ kind = "key" if key else "string"
+ try:
+ size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(
+ f"Sunburst native metric column metadata {kind} is invalid"
+ ) from ex
+ limit = (
+ _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ if key
+ else _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ )
+ if size > limit:
+ raise ValueError(f"Sunburst native metric column metadata {kind} is too long")
+ return size
+
+
+def _sanitize_native_metric_dict(
+ value: dict[Any, Any],
+ depth: int,
+ remaining: list[int],
+ ancestors: set[int],
+) -> tuple[Any, bool]:
+ """Project one exact dictionary through built-in operations only."""
+ value_id = id(value)
+ if value_id in ancestors:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ return {
+ f"invalid_{index}": _NATIVE_METRIC_INVALID_VALUE
+ for index in range(_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS + 1)
+ }, True
+
+ ancestors.add(value_id)
+ projected: dict[str, Any] = {}
+ requires_rejection = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ projected[_NATIVE_METRIC_INVALID_KEY] = _NATIVE_METRIC_INVALID_VALUE
+ requires_rejection = True
+ continue
+ try:
+ key_size = len(str.encode(key, "utf-8"))
+ except UnicodeEncodeError:
+ key_size = _NATIVE_COLUMN_META_MAX_KEY_BYTES + 1
+ if (
+ str.__len__(key) > _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ or key_size > _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ ):
+ projected["x" * (_NATIVE_COLUMN_META_MAX_KEY_BYTES + 1)] = (
+ _NATIVE_METRIC_INVALID_VALUE
+ )
+ requires_rejection = True
+ continue
+ projected_item, item_requires_rejection = _sanitize_native_metric_value(
+ item,
+ depth=depth + 1,
+ remaining=remaining,
+ ancestors=ancestors,
+ )
+ projected[key] = projected_item
+ requires_rejection = requires_rejection or item_requires_rejection
+ ancestors.remove(value_id)
+ return projected, requires_rejection
+
+
+def _sanitize_native_metric_list(
+ value: list[Any],
+ depth: int,
+ remaining: list[int],
+ ancestors: set[int],
+) -> tuple[Any, bool]:
+ """Project one exact list through built-in operations only."""
+ value_id = id(value)
+ if value_id in ancestors:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ return [_NATIVE_METRIC_INVALID_VALUE] * (
+ _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS + 1
+ ), True
+
+ ancestors.add(value_id)
+ projected_items: list[Any] = []
+ requires_rejection = False
+ for index in range(list.__len__(value)):
+ projected_item, item_requires_rejection = _sanitize_native_metric_value(
+ list.__getitem__(value, index),
+ depth=depth + 1,
+ remaining=remaining,
+ ancestors=ancestors,
+ )
+ projected_items.append(projected_item)
+ requires_rejection = requires_rejection or item_requires_rejection
+ ancestors.remove(value_id)
+ return projected_items, requires_rejection
+
+
+def _sanitize_native_metric_scalar(value: Any) -> tuple[Any, bool] | None:
+ """Project an exact scalar, or signal that ``value`` is a container."""
+ value_type = type(value)
+ if value_type is str:
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if (
+ str.__len__(value) > _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ or encoded_size > _NATIVE_COLUMN_META_MAX_STRING_BYTES * 4
+ ):
+ return "x" * (_NATIVE_COLUMN_META_MAX_STRING_BYTES + 1), True
+ return (
+ str.__getitem__(value, slice(0, _NATIVE_METRIC_ERROR_STRING_LENGTH)),
+ False,
+ )
+ if value_type is int and int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ return 1 << (_NATIVE_COLUMN_META_MAX_INT_BITS + 1), True
+ if value_type in (int, float, bool, type(None), ColumnRef):
+ return value, False
+ return None
+
+
+def _sanitize_native_metric_value(
+ value: Any,
+ *,
+ depth: int = 0,
+ remaining: list[int] | None = None,
+ ancestors: set[int] | None = None,
+) -> tuple[Any, bool]:
+ """Build a bounded exact-type projection without dispatching object hooks."""
+ if remaining is None:
+ remaining = [_NATIVE_METRIC_SANITIZE_MAX_VALUES]
+ if ancestors is None:
+ ancestors = set()
+ remaining[0] -= 1
+ if remaining[0] < 0 or depth > _NATIVE_METRIC_SANITIZE_MAX_DEPTH:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+
+ if (scalar := _sanitize_native_metric_scalar(value)) is not None:
+ return scalar
+ value_type = type(value)
+ if value_type is dict:
+ return _sanitize_native_metric_dict(value, depth, remaining, ancestors)
+ if value_type is list:
+ return _sanitize_native_metric_list(value, depth, remaining, ancestors)
+ return _NATIVE_METRIC_INVALID_VALUE, True
+
+
+def _metric_paths_in_dict_require_safe_rejection(
+ value: dict[Any, Any], depth: int, remaining: list[int], seen: set[int]
+) -> bool:
+ """Inspect metric-bearing paths of one exact dictionary."""
+ value_id = id(value)
+ if value_id in seen:
+ return False
+ seen.add(value_id)
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ return True
+ if key in _NATIVE_METRIC_INPUT_KEYS:
+ if _sanitize_native_metric_value(nested)[1]:
+ return True
+ elif _native_metric_paths_require_safe_rejection(
+ nested, depth=depth + 1, remaining=remaining, seen=seen
+ ):
+ return True
+ return False
+
+
+def _metric_paths_in_list_require_safe_rejection(
+ value: list[Any], depth: int, remaining: list[int], seen: set[int]
+) -> bool:
+ """Inspect metric-bearing paths of one exact list."""
+ value_id = id(value)
+ if value_id in seen:
+ return False
+ seen.add(value_id)
+ return any(
+ _native_metric_paths_require_safe_rejection(
+ list.__getitem__(value, index),
+ depth=depth + 1,
+ remaining=remaining,
+ seen=seen,
+ )
+ for index in range(list.__len__(value))
+ )
+
+
+def _native_metric_paths_require_safe_rejection(
+ value: Any,
+ *,
+ depth: int = 0,
+ remaining: list[int] | None = None,
+ seen: set[int] | None = None,
+) -> bool:
+ """Find unsafe metric subtrees without consuming non-exact containers."""
+ if remaining is None:
+ remaining = [_NATIVE_METRIC_SANITIZE_MAX_VALUES]
+ if seen is None:
+ seen = set()
+ remaining[0] -= 1
+ if remaining[0] < 0 or depth > _NATIVE_METRIC_SANITIZE_MAX_DEPTH:
+ return False
+ if type(value) is dict:
+ return _metric_paths_in_dict_require_safe_rejection(
+ value, depth, remaining, seen
+ )
+ if type(value) is list:
+ return _metric_paths_in_list_require_safe_rejection(
+ value, depth, remaining, seen
+ )
+ return False
+
+
+def _replace_exact_dict_contents(
+ value: dict[Any, Any], replacement: dict[str, Any]
+) -> None:
+ """Replace exact-dict input through built-in operations only."""
+ dict.clear(value)
+ for key, item in dict.items(replacement):
+ dict.__setitem__(value, key, item)
+
+
+def _sanitize_rejected_native_metric_config(
+ data: dict[str, Any], original_data: dict[str, Any] | None
+) -> dict[str, Any]:
+ """Project a rejected config and replace Pydantic's retained input."""
+ projected, _ = _sanitize_native_metric_value(data)
+ if type(projected) is not dict:
+ projected = {_NATIVE_METRIC_INVALID_KEY: _NATIVE_METRIC_INVALID_VALUE}
+ if original_data is not None:
+ _replace_exact_dict_contents(original_data, projected)
+ return projected
+
+
+def _validate_native_metric_string(
+ value: Any, field: str, max_length: int
+) -> str | None:
+ """Validate one optional exact string in the closed native metric wrapper."""
+ if value is None:
+ return None
+ if type(value) is not str:
+ raise ValueError(f"Sunburst native metric {field} must be a string")
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(f"Sunburst native metric {field} is invalid") from ex
+ if str.__len__(value) > max_length or encoded_size > max_length * 4:
+ raise ValueError(f"Sunburst native metric {field} is too long")
+ return value
+
+
+def _inspect_native_metric_expression_type(value: Any) -> tuple[object, bool]:
+ """Inspect an exact wrapper without hashing or comparing hostile objects."""
+ if type(value) is not dict:
+ return _NATIVE_METRIC_MISSING, False
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+ expression_type: object = _NATIVE_METRIC_MISSING
+ has_native_discriminator = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key in _NATIVE_METRIC_DISCRIMINATOR_KEYS:
+ has_native_discriminator = True
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError(
+ "Sunburst native metric expressionType must be a string"
+ )
+ if item not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError(
+ "Sunburst native metric expressionType must be SIMPLE or SQL"
+ )
+ expression_type = item
+ return expression_type, has_native_discriminator
+
+
+def _validate_native_metric_column_value(value: Any) -> None:
+ """Validate a wrapper column without dispatching container or scalar hooks."""
+ if type(value) is str:
+ _validate_native_metric_string(value, "column", 255)
+ elif type(value) is dict:
+ _validate_bounded_native_column_meta(value)
+ elif value is not None:
+ raise ValueError("Sunburst native metric column must be a string or object")
+
+
+def _validate_native_metric_item(key: str, item: Any) -> object:
+ """Validate one value after its wrapper key is known to be an exact string."""
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError("Sunburst native metric expressionType must be a string")
+ _validate_native_metric_string(item, "expressionType", 10)
+ return item
+ if key in ("label", "optionName"):
+ _validate_native_metric_string(item, key, 500)
+ elif key == "sqlExpression":
+ _validate_native_metric_string(item, key, 2_000)
+ elif key == "aggregate":
+ _validate_native_metric_string(item, key, 32)
+ elif key in ("datasourceWarning", "hasCustomLabel"):
+ if item is not None and type(item) is not bool:
+ raise ValueError(f"Sunburst native metric {key} must be a boolean")
+ elif key == "column":
+ _validate_native_metric_column_value(item)
+ return _NATIVE_METRIC_MISSING
+
+
+def _validate_native_metric_wrapper(value: Any) -> dict[str, Any]:
+ """Validate and copy the closed native metric wrapper without calling hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric must be a string or object")
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+
+ validated: dict[str, Any] = {}
+ expression_type: object = _NATIVE_METRIC_MISSING
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key not in _NATIVE_METRIC_ALLOWED_KEYS:
+ raise ValueError(f"Unknown Sunburst native metric field: {key}")
+ item_expression_type = _validate_native_metric_item(key, item)
+ if item_expression_type is not _NATIVE_METRIC_MISSING:
+ expression_type = item_expression_type
+ validated[key] = item
+
+ if expression_type is _NATIVE_METRIC_MISSING:
+ raise ValueError("Sunburst native metric requires expressionType")
+ if expression_type not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError("Sunburst native metric expressionType must be SIMPLE or SQL")
+ return validated
+
+
+def _validate_non_dict_native_metric(value: Any) -> ColumnRef:
+ """Accept an existing model while rejecting hook-bearing dict subclasses."""
+ if type(value) is ColumnRef:
+ return value
+ raise ValueError("Sunburst native metric must be a string or object")
+
+
+def _native_column_meta_dict_children(
+ value: dict[Any, Any], depth: int
+) -> tuple[list[tuple[Any, int]], int]:
+ """Read exact-dict children while validating bounded primitive keys."""
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata object is too large")
+ children: list[tuple[Any, int]] = []
+ string_bytes = 0
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ raise ValueError(
+ "Sunburst native metric column metadata keys must be strings"
+ )
+ string_bytes += _native_column_meta_string_bytes(key, key=True)
+ children.append((nested, depth + 1))
+ return children, string_bytes
+
+
+def _native_column_meta_list_children(
+ value: list[Any], depth: int
+) -> list[tuple[Any, int]]:
+ """Read exact-list children without dispatching subclass hooks."""
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata array is too large")
+ return [
+ (list.__getitem__(value, index), depth + 1)
+ for index in range(list.__len__(value))
+ ]
+
+
+def _validate_native_column_meta_scalar(value: Any) -> int:
+ """Validate one exact JSON scalar and return its string-byte contribution."""
+ value_type = type(value)
+ if value_type is str:
+ return _native_column_meta_string_bytes(value)
+ if value_type is int:
+ if int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ raise ValueError(
+ "Sunburst native metric column metadata integer is too large"
+ )
+ return 0
+ if value_type is float:
+ if not math.isfinite(value):
+ raise ValueError(
+ "Sunburst native metric column metadata number must be finite"
+ )
+ return 0
+ if value_type not in (bool, type(None)):
+ raise ValueError(
+ "Sunburst native metric column metadata contains an invalid value"
+ )
+ return 0
+
+
+def _validate_bounded_native_column_meta(value: Any) -> Any:
+ """Validate an extensible frontend ColumnMeta object without invoking hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric column must be an object")
+
+ total_values = 0
+ total_string_bytes = 0
+ stack: list[tuple[Any, int]] = [(value, 0)]
+ while stack:
+ item, depth = stack.pop()
+ total_values += 1
+ if total_values > _NATIVE_COLUMN_META_MAX_TOTAL_VALUES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ if depth > _NATIVE_COLUMN_META_MAX_DEPTH:
+ raise ValueError(
+ "Sunburst native metric column metadata is too deeply nested"
+ )
+
+ item_type = type(item)
+ if item_type is dict:
+ children, key_bytes = _native_column_meta_dict_children(item, depth)
+ total_string_bytes += key_bytes
+ stack.extend(children)
+ elif item_type is list:
+ stack.extend(_native_column_meta_list_children(item, depth))
+ else:
+ total_string_bytes += _validate_native_column_meta_scalar(item)
+
+ if total_string_bytes > _NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ return value
+
+
+class SunburstNativeMetricColumn(BaseModel):
+ """Owned projection of the frontend's intentionally extensible ColumnMeta."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ type: StrictStr | None = Field(None, max_length=255)
+
+ @model_validator(mode="before")
+ @classmethod
+ def validate_bounded_column_meta(cls, value: Any) -> Any:
+ """Bound the full open object before projecting the fields we own."""
+ projected, requires_safe_rejection = _sanitize_native_metric_value(value)
+ if requires_safe_rejection:
+ if type(value) is dict:
+ if type(projected) is not dict:
+ projected = {
+ _NATIVE_METRIC_INVALID_KEY: _NATIVE_METRIC_INVALID_VALUE
+ }
+ _replace_exact_dict_contents(value, projected)
+ value = projected
+ else:
+ # Let Pydantic reject the safe scalar so the original subclass
+ # is not retained as the input of an in-validator exception.
+ return projected
+ value = _validate_bounded_native_column_meta(value)
+ snake_name = dict.get(value, "column_name", _NATIVE_METRIC_MISSING)
+ camel_name = dict.get(value, "columnName", _NATIVE_METRIC_MISSING)
+ if (
+ snake_name is not _NATIVE_METRIC_MISSING
+ and camel_name is not _NATIVE_METRIC_MISSING
+ ):
+ if type(snake_name) is not str or type(camel_name) is not str:
+ raise ValueError(
+ "column_name and columnName must both be exact strings"
+ )
+ if snake_name != camel_name:
+ raise ValueError(
+ "column_name and columnName must match when both are provided"
+ )
+ return value
+
+
+class XYNativeMetricColumn(UnknownFieldCheckMixin):
+ """Closed legacy ColumnMeta projection retained for native XY axes."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ id: StrictInt | None = Field(None, ge=0)
+ type: StrictStr | None = Field(None, max_length=255)
+ type_generic: StrictInt | None = Field(None, ge=0, le=10)
+ groupby: StrictBool | None = None
+ is_dttm: StrictBool | None = None
+ filterable: StrictBool | None = None
+ verbose_name: StrictStr | None = Field(None, max_length=500)
+ description: StrictStr | None = Field(None, max_length=2000)
+ expression: StrictStr | None = Field(None, max_length=2000)
+ database_expression: StrictStr | None = Field(None, max_length=2000)
+ python_date_format: StrictStr | None = Field(None, max_length=255)
+ option_name: StrictStr | None = Field(
+ None,
+ max_length=500,
+ validation_alias=AliasChoices("option_name", "optionName"),
+ )
+ filter_by: StrictStr | None = Field(
+ None,
+ max_length=255,
+ validation_alias=AliasChoices("filter_by", "filterBy"),
+ )
+ value: StrictStr | None = Field(None, max_length=500)
+ advanced_data_type: StrictStr | None = Field(None, max_length=255)
+ uuid: StrictStr | None = Field(None, max_length=64)
+
+
+class SunburstLegacySavedFields(BaseModel):
+ """Known obsolete controls tolerated only while reading saved Sunbursts."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ country_fieldtype: StrictStr | None = Field(None, max_length=255)
+ entity: StrictStr | None = Field(None, max_length=255)
+ granularity: StrictStr | None = Field(None, max_length=255)
+ limit: StrictInt | StrictStr | None = None
+ markup_type: Literal["markdown", "html"] | None = None
+ show_bubbles: StrictBool | None = None
+
+ @field_validator("limit")
+ @classmethod
+ def validate_legacy_limit(cls, value: int | str | None) -> int | str | None:
+ """Bound the ignored legacy row limit and reject non-numeric strings."""
+ if value is None:
+ return None
+ if isinstance(value, str):
+ if not value.isascii() or not value.isdigit():
+ raise ValueError("legacy Sunburst limit must be an integer")
+ numeric_value = int(value)
+ else:
+ numeric_value = value
+ if not 1 <= numeric_value <= 50000:
+ raise ValueError("legacy Sunburst limit must be between 1 and 50000")
+ return value
+
+
+_NATIVE_FORM_DATA_MARKER = "_mcp_native_form_data"
+_SUNBURST_IGNORED_LEGACY_FIELDS = frozenset(
+ {
+ "country_fieldtype",
+ "entity",
+ "granularity",
+ "limit",
+ "markup_type",
+ "show_bubbles",
+ }
+)
+
+
+class SunburstChartConfig(BaseChartConfig):
+ """Config for the ECharts Sunburst plugin (viz_type ``sunburst_v2``)."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ chart_type: Literal["sunburst"] = "sunburst"
+ viz_type: Literal["sunburst_v2"] = Field(
+ "sunburst_v2",
+ description="Exact Superset frontend visualization tag",
+ )
+ hierarchy: List[ColumnRef] = Field(
+ ...,
+ min_length=1,
+ description=(
+ "Hierarchy dimensions in ring order, from the innermost/root level "
+ "to the outermost/leaf level"
+ ),
+ validation_alias=AliasChoices("hierarchy", "columns", "groupby"),
+ )
+ metric: ColumnRef = Field(
+ ...,
+ description=(
+ "Primary metric used to size arcs. Use aggregate for a SIMPLE "
+ "adhoc metric, saved_metric=True for a dataset metric, or "
+ "sql_expression plus label for a SQL metric."
+ ),
+ )
+ secondary_metric: ColumnRef | None = Field(
+ None,
+ description=(
+ "Optional metric whose ratio to the primary metric drives the "
+ "sequential color scale (for example, profit/revenue represents "
+ "margin). When omitted, arc colors are categorical."
+ ),
+ validation_alias=AliasChoices("secondary_metric", "secondaryMetric"),
+ )
+ filters: List[FilterConfig] | None = Field(
+ None,
+ description=(
+ "Structured WHERE filters (column/op/value/clause). An omitted list is "
+ "preserved on updates; an explicit [] clears saved filters."
+ ),
+ )
+ time_range: str | None = Field(
+ None,
+ min_length=1,
+ max_length=1000,
+ description=(
+ "Superset time range, for example 'Last year', "
+ "'2025-01-01 : 2025-12-31', or 'No filter'"
+ ),
+ )
+ time_grain: TimeGrain | None = Field(
+ None,
+ description="Optional bucket for temporal hierarchy columns",
+ validation_alias=AliasChoices("time_grain", "time_grain_sqla"),
+ )
+ sort_by_metric: bool = Field(
+ False,
+ description=(
+ "Order hierarchy rows by the primary metric descending before "
+ "applying row_limit, matching the frontend buildQuery transform"
+ ),
+ )
+ row_limit: int = Field(10000, description="Maximum hierarchy rows", ge=1, le=50000)
+ color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Categorical scheme used when secondary_metric is omitted",
+ )
+ linear_color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Sequential scheme used when secondary_metric is present",
+ )
+ show_labels: bool = False
+ show_labels_threshold: float = Field(
+ 5,
+ ge=0,
+ le=100,
+ description="Minimum arc size in percentage points for showing a label",
+ )
+ show_total: bool = False
+ show_null_values: bool = Field(
+ True,
+ description="Keep null-valued hierarchy nodes in the rendered tree",
+ )
+ label_type: Literal["key", "value", "key_value"] = "key"
+ number_format: str = Field("SMART_NUMBER", min_length=1, max_length=50)
+ date_format: str = Field("smart_date", min_length=1, max_length=50)
+ currency_format: CurrencyFormat | None = None
+ # Bounded native Explore envelope/UI state. These keys occur in real saved
+ # form_data and are safe to round-trip, but remain typed so native mode does
+ # not become an escape hatch for arbitrary plugin keys or malformed nesting.
+ annotation_layers: list[dict[str, JsonValue]] | None = None
+ dashboard_id: int | None = Field(
+ None,
+ validation_alias=AliasChoices("dashboard_id", "dashboardId"),
+ serialization_alias="dashboardId",
+ )
+ dashboards: list[int] | None = None
+ datasource: str | None = Field(None, min_length=1, max_length=255)
+ extra_form_data: dict[str, JsonValue] | None = None
+ slice_id: int | None = None
+ url_params: dict[str, JsonValue] | None = None
+ standardized_form_data: SunburstStandardizedFormData | None = Field(
+ None,
+ validation_alias=AliasChoices("standardized_form_data", "standardizedFormData"),
+ serialization_alias="standardizedFormData",
+ )
+ since: str | None = Field(None, max_length=1000)
+ until: str | None = Field(None, max_length=1000)
+ time_compare: str | list[str] | None = None
+ compare_lag: int | str | None = None
+ compare_suffix: str | None = Field(None, max_length=255)
+
+ @staticmethod
+ def _looks_like_native_form_data(data: Any) -> bool:
+ """Identify saved Explore payloads without weakening typed typo checks."""
+ if type(data) is not dict:
+ return False
+ native_marker = dict.get(data, _NATIVE_FORM_DATA_MARKER)
+ viz_type = dict.get(data, "viz_type")
+ if not (
+ (type(native_marker) is bool and native_marker)
+ or (type(viz_type) is str and viz_type == "sunburst_v2")
+ ):
+ return False
+ metric = dict.get(data, "metric")
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(metric)
+ )
+ return (
+ any(
+ key in data
+ for key in (
+ "adhoc_filters",
+ "annotation_layers",
+ "datasource",
+ "extra_form_data",
+ "since",
+ "slice_id",
+ "standardizedFormData",
+ "until",
+ )
+ )
+ or type(metric) is str
+ or expression_type is not _NATIVE_METRIC_MISSING
+ or has_native_discriminator
+ )
+
+ @staticmethod
+ def _coerce_native_metric(
+ value: Any, *, allow_extensible_column_meta: bool = True
+ ) -> Any:
+ """Accept saved metric names and native SIMPLE/SQL metric objects."""
+ if type(value) is str:
+ return {"name": value, "saved_metric": True}
+ if type(value) is dict:
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(value)
+ )
+ if expression_type is _NATIVE_METRIC_MISSING:
+ if has_native_discriminator:
+ raise ValueError("Sunburst native metric requires expressionType")
+ return value
+ value = _validate_native_metric_wrapper(value)
+ else:
+ return _validate_non_dict_native_metric(value)
+
+ expression_type = value.get("expressionType")
+ label = value.get("label")
+ has_custom_label = value.get("hasCustomLabel")
+ if expression_type == "SQL":
+ sql_expression = value.get("sqlExpression")
+ return {
+ "sql_expression": sql_expression,
+ # The frontend still assigns a result label when the label is
+ # not custom. Preserve a nonempty native label first, then use
+ # the SQL expression as the same effective fallback.
+ "label": label or sql_expression,
+ "has_custom_label": has_custom_label,
+ }
+ if expression_type == "SIMPLE":
+ column = value.get("column")
+ if type(column) is dict:
+ column_model = (
+ SunburstNativeMetricColumn
+ if allow_extensible_column_meta
+ else XYNativeMetricColumn
+ )
+ column_metadata = column_model.model_validate(column)
+ name = column_metadata.column_name
+ dtype = column_metadata.type
+ elif type(column) is str:
+ name = column
+ dtype = None
+ else:
+ raise ValueError(
+ "Sunburst SIMPLE metric column must be a column name or object"
+ )
+ return {
+ "name": name,
+ "aggregate": value.get("aggregate"),
+ "label": label,
+ "has_custom_label": has_custom_label,
+ "dtype": dtype,
+ }
+ return value
+
+ @staticmethod
+ def _coerce_native_hierarchy(data: dict[str, Any]) -> None:
+ """Normalize native ``columns``/``groupby`` dimension shortcuts."""
+ hierarchy_key = next(
+ (key for key in ("hierarchy", "columns", "groupby") if key in data),
+ None,
+ )
+ if hierarchy_key is None:
+ return
+ hierarchy = data[hierarchy_key]
+ if isinstance(hierarchy, (str, dict)):
+ hierarchy = [hierarchy]
+ if isinstance(hierarchy, list):
+ data[hierarchy_key] = [
+ {"name": item} if isinstance(item, str) else item for item in hierarchy
+ ]
+ if hierarchy_key != "groupby":
+ # Saved Sunburst payloads sometimes retain an empty groupby from a
+ # previous plugin. The explicit columns/hierarchy control wins.
+ data.pop("groupby", None)
+
+ @staticmethod
+ def _coerce_native_filter(
+ native_filter: Any, data: dict[str, Any]
+ ) -> dict[str, Any] | None:
+ """Convert one native SIMPLE filter, extracting temporal controls."""
+ if not isinstance(native_filter, dict):
+ raise ValueError("Each Sunburst native adhoc_filter must be an object")
+ allowed_keys = {
+ "clause",
+ "comparator",
+ "datasourceWarning",
+ "expressionType",
+ "filterOptionName",
+ "isExtra",
+ "operator",
+ "sqlExpression",
+ "subject",
+ }
+ if unknown := set(native_filter) - allowed_keys:
+ raise ValueError(
+ "Unknown Sunburst native adhoc_filter field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ if native_filter.get("expressionType") != "SIMPLE":
+ raise ValueError(
+ "Sunburst native adhoc_filters round-trip supports SIMPLE filters "
+ "only; express filters with the typed 'filters' field"
+ )
+ operator = native_filter.get("operator")
+ clause = native_filter.get("clause", "WHERE")
+ if not isinstance(clause, str) or clause != "WHERE":
+ raise ValueError(
+ "Sunburst native SIMPLE filter clause must be 'WHERE'; "
+ "SIMPLE HAVING is unsupported because the shared query mapper "
+ "cannot preserve HAVING semantics"
+ )
+ if operator == "TEMPORAL_RANGE":
+ if "temporal_column" not in data and "granularity_sqla" not in data:
+ data["temporal_column"] = native_filter.get("subject")
+ data.setdefault("time_range", native_filter.get("comparator"))
+ return None
Review Comment:
A native Sunburst with `order_date=Last year` and `ship_date=Last month` temporal predicates now promotes only the first to top-level controls and drops both filters, so the ship-date restriction disappears from the query and saved chart. Could multiple temporal predicates be preserved, or rejected if the typed conversion cannot represent them?
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -1634,6 +1652,1045 @@ def reject_metric_style_groupby(self) -> "TreemapChartUpdateConfig":
return self
+class SunburstStandardizedControls(UnknownFieldCheckMixin):
+ """Bounded shared controls retained by Explore across viz changes."""
+
+ model_config = ConfigDict(extra="ignore")
+
+ metrics: list[JsonValue] = Field(default_factory=list)
+ columns: list[JsonValue] = Field(default_factory=list)
+
+
+class SunburstStandardizedFormData(UnknownFieldCheckMixin):
+ """Validated shape of Explore's cross-plugin UI memory."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ controls: SunburstStandardizedControls
+ memorized_form_data: list[tuple[str, dict[str, JsonValue]]] = Field(
+ default_factory=list,
+ validation_alias=AliasChoices("memorized_form_data", "memorizedFormData"),
+ serialization_alias="memorizedFormData",
+ )
+
+
+_NATIVE_COLUMN_META_MAX_DEPTH = 8
+_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS = 128
+_NATIVE_COLUMN_META_MAX_TOTAL_VALUES = 512
+_NATIVE_COLUMN_META_MAX_KEY_BYTES = 1_024
+_NATIVE_COLUMN_META_MAX_STRING_BYTES = 16 * 1_024
+_NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES = 64 * 1_024
+_NATIVE_COLUMN_META_MAX_INT_BITS = 64
+_NATIVE_METRIC_ALLOWED_KEYS = frozenset(
+ {
+ "aggregate",
+ "column",
+ "datasourceWarning",
+ "expressionType",
+ "hasCustomLabel",
+ "label",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_EXPRESSION_TYPES = frozenset({"SIMPLE", "SQL"})
+_NATIVE_METRIC_DISCRIMINATOR_KEYS = frozenset(
+ {
+ "column",
+ "datasourceWarning",
+ "hasCustomLabel",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_MISSING = object()
+_NATIVE_METRIC_INPUT_KEYS = frozenset(
+ {"metric", "metrics", "secondary_metric", "secondaryMetric"}
+)
+_NATIVE_METRIC_INVALID_VALUE = b"invalid native metric value"
+_NATIVE_METRIC_INVALID_KEY = "__invalid_native_metric_key__"
+_NATIVE_METRIC_SANITIZE_MAX_DEPTH = 16
+_NATIVE_METRIC_SANITIZE_MAX_VALUES = 1_024
+_NATIVE_METRIC_ERROR_STRING_LENGTH = 128
+
+
+def _native_column_meta_string_bytes(value: str, *, key: bool = False) -> int:
+ """Return exact UTF-8 size for a trusted built-in string."""
+ kind = "key" if key else "string"
+ try:
+ size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(
+ f"Sunburst native metric column metadata {kind} is invalid"
+ ) from ex
+ limit = (
+ _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ if key
+ else _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ )
+ if size > limit:
+ raise ValueError(f"Sunburst native metric column metadata {kind} is too long")
+ return size
+
+
+def _sanitize_native_metric_dict(
+ value: dict[Any, Any],
+ depth: int,
+ remaining: list[int],
+ ancestors: set[int],
+) -> tuple[Any, bool]:
+ """Project one exact dictionary through built-in operations only."""
+ value_id = id(value)
+ if value_id in ancestors:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ return {
+ f"invalid_{index}": _NATIVE_METRIC_INVALID_VALUE
+ for index in range(_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS + 1)
+ }, True
+
+ ancestors.add(value_id)
+ projected: dict[str, Any] = {}
+ requires_rejection = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ projected[_NATIVE_METRIC_INVALID_KEY] = _NATIVE_METRIC_INVALID_VALUE
+ requires_rejection = True
+ continue
+ try:
+ key_size = len(str.encode(key, "utf-8"))
+ except UnicodeEncodeError:
+ key_size = _NATIVE_COLUMN_META_MAX_KEY_BYTES + 1
+ if (
+ str.__len__(key) > _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ or key_size > _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ ):
+ projected["x" * (_NATIVE_COLUMN_META_MAX_KEY_BYTES + 1)] = (
+ _NATIVE_METRIC_INVALID_VALUE
+ )
+ requires_rejection = True
+ continue
+ projected_item, item_requires_rejection = _sanitize_native_metric_value(
+ item,
+ depth=depth + 1,
+ remaining=remaining,
+ ancestors=ancestors,
+ )
+ projected[key] = projected_item
+ requires_rejection = requires_rejection or item_requires_rejection
+ ancestors.remove(value_id)
+ return projected, requires_rejection
+
+
+def _sanitize_native_metric_list(
+ value: list[Any],
+ depth: int,
+ remaining: list[int],
+ ancestors: set[int],
+) -> tuple[Any, bool]:
+ """Project one exact list through built-in operations only."""
+ value_id = id(value)
+ if value_id in ancestors:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ return [_NATIVE_METRIC_INVALID_VALUE] * (
+ _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS + 1
+ ), True
+
+ ancestors.add(value_id)
+ projected_items: list[Any] = []
+ requires_rejection = False
+ for index in range(list.__len__(value)):
+ projected_item, item_requires_rejection = _sanitize_native_metric_value(
+ list.__getitem__(value, index),
+ depth=depth + 1,
+ remaining=remaining,
+ ancestors=ancestors,
+ )
+ projected_items.append(projected_item)
+ requires_rejection = requires_rejection or item_requires_rejection
+ ancestors.remove(value_id)
+ return projected_items, requires_rejection
+
+
+def _sanitize_native_metric_scalar(value: Any) -> tuple[Any, bool] | None:
+ """Project an exact scalar, or signal that ``value`` is a container."""
+ value_type = type(value)
+ if value_type is str:
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if (
+ str.__len__(value) > _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ or encoded_size > _NATIVE_COLUMN_META_MAX_STRING_BYTES * 4
+ ):
+ return "x" * (_NATIVE_COLUMN_META_MAX_STRING_BYTES + 1), True
+ return (
+ str.__getitem__(value, slice(0, _NATIVE_METRIC_ERROR_STRING_LENGTH)),
+ False,
+ )
+ if value_type is int and int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ return 1 << (_NATIVE_COLUMN_META_MAX_INT_BITS + 1), True
+ if value_type in (int, float, bool, type(None), ColumnRef):
+ return value, False
+ return None
+
+
+def _sanitize_native_metric_value(
+ value: Any,
+ *,
+ depth: int = 0,
+ remaining: list[int] | None = None,
+ ancestors: set[int] | None = None,
+) -> tuple[Any, bool]:
+ """Build a bounded exact-type projection without dispatching object hooks."""
+ if remaining is None:
+ remaining = [_NATIVE_METRIC_SANITIZE_MAX_VALUES]
+ if ancestors is None:
+ ancestors = set()
+ remaining[0] -= 1
+ if remaining[0] < 0 or depth > _NATIVE_METRIC_SANITIZE_MAX_DEPTH:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+
+ if (scalar := _sanitize_native_metric_scalar(value)) is not None:
+ return scalar
+ value_type = type(value)
+ if value_type is dict:
+ return _sanitize_native_metric_dict(value, depth, remaining, ancestors)
+ if value_type is list:
+ return _sanitize_native_metric_list(value, depth, remaining, ancestors)
+ return _NATIVE_METRIC_INVALID_VALUE, True
+
+
+def _metric_paths_in_dict_require_safe_rejection(
+ value: dict[Any, Any], depth: int, remaining: list[int], seen: set[int]
+) -> bool:
+ """Inspect metric-bearing paths of one exact dictionary."""
+ value_id = id(value)
+ if value_id in seen:
+ return False
+ seen.add(value_id)
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ return True
+ if key in _NATIVE_METRIC_INPUT_KEYS:
+ if _sanitize_native_metric_value(nested)[1]:
+ return True
+ elif _native_metric_paths_require_safe_rejection(
+ nested, depth=depth + 1, remaining=remaining, seen=seen
+ ):
+ return True
+ return False
+
+
+def _metric_paths_in_list_require_safe_rejection(
+ value: list[Any], depth: int, remaining: list[int], seen: set[int]
+) -> bool:
+ """Inspect metric-bearing paths of one exact list."""
+ value_id = id(value)
+ if value_id in seen:
+ return False
+ seen.add(value_id)
+ return any(
+ _native_metric_paths_require_safe_rejection(
+ list.__getitem__(value, index),
+ depth=depth + 1,
+ remaining=remaining,
+ seen=seen,
+ )
+ for index in range(list.__len__(value))
+ )
+
+
+def _native_metric_paths_require_safe_rejection(
+ value: Any,
+ *,
+ depth: int = 0,
+ remaining: list[int] | None = None,
+ seen: set[int] | None = None,
+) -> bool:
+ """Find unsafe metric subtrees without consuming non-exact containers."""
+ if remaining is None:
+ remaining = [_NATIVE_METRIC_SANITIZE_MAX_VALUES]
+ if seen is None:
+ seen = set()
+ remaining[0] -= 1
+ if remaining[0] < 0 or depth > _NATIVE_METRIC_SANITIZE_MAX_DEPTH:
+ return False
+ if type(value) is dict:
+ return _metric_paths_in_dict_require_safe_rejection(
+ value, depth, remaining, seen
+ )
+ if type(value) is list:
+ return _metric_paths_in_list_require_safe_rejection(
+ value, depth, remaining, seen
+ )
+ return False
+
+
+def _replace_exact_dict_contents(
+ value: dict[Any, Any], replacement: dict[str, Any]
+) -> None:
+ """Replace exact-dict input through built-in operations only."""
+ dict.clear(value)
+ for key, item in dict.items(replacement):
+ dict.__setitem__(value, key, item)
+
+
+def _sanitize_rejected_native_metric_config(
+ data: dict[str, Any], original_data: dict[str, Any] | None
+) -> dict[str, Any]:
+ """Project a rejected config and replace Pydantic's retained input."""
+ projected, _ = _sanitize_native_metric_value(data)
+ if type(projected) is not dict:
+ projected = {_NATIVE_METRIC_INVALID_KEY: _NATIVE_METRIC_INVALID_VALUE}
+ if original_data is not None:
+ _replace_exact_dict_contents(original_data, projected)
+ return projected
+
+
+def _validate_native_metric_string(
+ value: Any, field: str, max_length: int
+) -> str | None:
+ """Validate one optional exact string in the closed native metric wrapper."""
+ if value is None:
+ return None
+ if type(value) is not str:
+ raise ValueError(f"Sunburst native metric {field} must be a string")
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(f"Sunburst native metric {field} is invalid") from ex
+ if str.__len__(value) > max_length or encoded_size > max_length * 4:
+ raise ValueError(f"Sunburst native metric {field} is too long")
+ return value
+
+
+def _inspect_native_metric_expression_type(value: Any) -> tuple[object, bool]:
+ """Inspect an exact wrapper without hashing or comparing hostile objects."""
+ if type(value) is not dict:
+ return _NATIVE_METRIC_MISSING, False
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+ expression_type: object = _NATIVE_METRIC_MISSING
+ has_native_discriminator = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key in _NATIVE_METRIC_DISCRIMINATOR_KEYS:
+ has_native_discriminator = True
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError(
+ "Sunburst native metric expressionType must be a string"
+ )
+ if item not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError(
+ "Sunburst native metric expressionType must be SIMPLE or SQL"
+ )
+ expression_type = item
+ return expression_type, has_native_discriminator
+
+
+def _validate_native_metric_column_value(value: Any) -> None:
+ """Validate a wrapper column without dispatching container or scalar hooks."""
+ if type(value) is str:
+ _validate_native_metric_string(value, "column", 255)
+ elif type(value) is dict:
+ _validate_bounded_native_column_meta(value)
+ elif value is not None:
+ raise ValueError("Sunburst native metric column must be a string or object")
+
+
+def _validate_native_metric_item(key: str, item: Any) -> object:
+ """Validate one value after its wrapper key is known to be an exact string."""
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError("Sunburst native metric expressionType must be a string")
+ _validate_native_metric_string(item, "expressionType", 10)
+ return item
+ if key in ("label", "optionName"):
+ _validate_native_metric_string(item, key, 500)
+ elif key == "sqlExpression":
+ _validate_native_metric_string(item, key, 2_000)
+ elif key == "aggregate":
+ _validate_native_metric_string(item, key, 32)
+ elif key in ("datasourceWarning", "hasCustomLabel"):
+ if item is not None and type(item) is not bool:
+ raise ValueError(f"Sunburst native metric {key} must be a boolean")
+ elif key == "column":
+ _validate_native_metric_column_value(item)
+ return _NATIVE_METRIC_MISSING
+
+
+def _validate_native_metric_wrapper(value: Any) -> dict[str, Any]:
+ """Validate and copy the closed native metric wrapper without calling hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric must be a string or object")
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+
+ validated: dict[str, Any] = {}
+ expression_type: object = _NATIVE_METRIC_MISSING
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key not in _NATIVE_METRIC_ALLOWED_KEYS:
+ raise ValueError(f"Unknown Sunburst native metric field: {key}")
+ item_expression_type = _validate_native_metric_item(key, item)
+ if item_expression_type is not _NATIVE_METRIC_MISSING:
+ expression_type = item_expression_type
+ validated[key] = item
+
+ if expression_type is _NATIVE_METRIC_MISSING:
+ raise ValueError("Sunburst native metric requires expressionType")
+ if expression_type not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError("Sunburst native metric expressionType must be SIMPLE or SQL")
+ return validated
+
+
+def _validate_non_dict_native_metric(value: Any) -> ColumnRef:
+ """Accept an existing model while rejecting hook-bearing dict subclasses."""
+ if type(value) is ColumnRef:
+ return value
+ raise ValueError("Sunburst native metric must be a string or object")
+
+
+def _native_column_meta_dict_children(
+ value: dict[Any, Any], depth: int
+) -> tuple[list[tuple[Any, int]], int]:
+ """Read exact-dict children while validating bounded primitive keys."""
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata object is too large")
+ children: list[tuple[Any, int]] = []
+ string_bytes = 0
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ raise ValueError(
+ "Sunburst native metric column metadata keys must be strings"
+ )
+ string_bytes += _native_column_meta_string_bytes(key, key=True)
+ children.append((nested, depth + 1))
+ return children, string_bytes
+
+
+def _native_column_meta_list_children(
+ value: list[Any], depth: int
+) -> list[tuple[Any, int]]:
+ """Read exact-list children without dispatching subclass hooks."""
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata array is too large")
+ return [
+ (list.__getitem__(value, index), depth + 1)
+ for index in range(list.__len__(value))
+ ]
+
+
+def _validate_native_column_meta_scalar(value: Any) -> int:
+ """Validate one exact JSON scalar and return its string-byte contribution."""
+ value_type = type(value)
+ if value_type is str:
+ return _native_column_meta_string_bytes(value)
+ if value_type is int:
+ if int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ raise ValueError(
+ "Sunburst native metric column metadata integer is too large"
+ )
+ return 0
+ if value_type is float:
+ if not math.isfinite(value):
+ raise ValueError(
+ "Sunburst native metric column metadata number must be finite"
+ )
+ return 0
+ if value_type not in (bool, type(None)):
+ raise ValueError(
+ "Sunburst native metric column metadata contains an invalid value"
+ )
+ return 0
+
+
+def _validate_bounded_native_column_meta(value: Any) -> Any:
+ """Validate an extensible frontend ColumnMeta object without invoking hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric column must be an object")
+
+ total_values = 0
+ total_string_bytes = 0
+ stack: list[tuple[Any, int]] = [(value, 0)]
+ while stack:
+ item, depth = stack.pop()
+ total_values += 1
+ if total_values > _NATIVE_COLUMN_META_MAX_TOTAL_VALUES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ if depth > _NATIVE_COLUMN_META_MAX_DEPTH:
+ raise ValueError(
+ "Sunburst native metric column metadata is too deeply nested"
+ )
+
+ item_type = type(item)
+ if item_type is dict:
+ children, key_bytes = _native_column_meta_dict_children(item, depth)
+ total_string_bytes += key_bytes
+ stack.extend(children)
+ elif item_type is list:
+ stack.extend(_native_column_meta_list_children(item, depth))
+ else:
+ total_string_bytes += _validate_native_column_meta_scalar(item)
+
+ if total_string_bytes > _NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ return value
+
+
+class SunburstNativeMetricColumn(BaseModel):
+ """Owned projection of the frontend's intentionally extensible ColumnMeta."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ type: StrictStr | None = Field(None, max_length=255)
+
+ @model_validator(mode="before")
+ @classmethod
+ def validate_bounded_column_meta(cls, value: Any) -> Any:
+ """Bound the full open object before projecting the fields we own."""
+ projected, requires_safe_rejection = _sanitize_native_metric_value(value)
+ if requires_safe_rejection:
+ if type(value) is dict:
+ if type(projected) is not dict:
+ projected = {
+ _NATIVE_METRIC_INVALID_KEY: _NATIVE_METRIC_INVALID_VALUE
+ }
+ _replace_exact_dict_contents(value, projected)
+ value = projected
+ else:
+ # Let Pydantic reject the safe scalar so the original subclass
+ # is not retained as the input of an in-validator exception.
+ return projected
+ value = _validate_bounded_native_column_meta(value)
+ snake_name = dict.get(value, "column_name", _NATIVE_METRIC_MISSING)
+ camel_name = dict.get(value, "columnName", _NATIVE_METRIC_MISSING)
+ if (
+ snake_name is not _NATIVE_METRIC_MISSING
+ and camel_name is not _NATIVE_METRIC_MISSING
+ ):
+ if type(snake_name) is not str or type(camel_name) is not str:
+ raise ValueError(
+ "column_name and columnName must both be exact strings"
+ )
+ if snake_name != camel_name:
+ raise ValueError(
+ "column_name and columnName must match when both are provided"
+ )
+ return value
+
+
+class XYNativeMetricColumn(UnknownFieldCheckMixin):
+ """Closed legacy ColumnMeta projection retained for native XY axes."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ id: StrictInt | None = Field(None, ge=0)
+ type: StrictStr | None = Field(None, max_length=255)
+ type_generic: StrictInt | None = Field(None, ge=0, le=10)
+ groupby: StrictBool | None = None
+ is_dttm: StrictBool | None = None
+ filterable: StrictBool | None = None
+ verbose_name: StrictStr | None = Field(None, max_length=500)
+ description: StrictStr | None = Field(None, max_length=2000)
+ expression: StrictStr | None = Field(None, max_length=2000)
+ database_expression: StrictStr | None = Field(None, max_length=2000)
+ python_date_format: StrictStr | None = Field(None, max_length=255)
+ option_name: StrictStr | None = Field(
+ None,
+ max_length=500,
+ validation_alias=AliasChoices("option_name", "optionName"),
+ )
+ filter_by: StrictStr | None = Field(
+ None,
+ max_length=255,
+ validation_alias=AliasChoices("filter_by", "filterBy"),
+ )
+ value: StrictStr | None = Field(None, max_length=500)
+ advanced_data_type: StrictStr | None = Field(None, max_length=255)
+ uuid: StrictStr | None = Field(None, max_length=64)
+
+
+class SunburstLegacySavedFields(BaseModel):
+ """Known obsolete controls tolerated only while reading saved Sunbursts."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ country_fieldtype: StrictStr | None = Field(None, max_length=255)
+ entity: StrictStr | None = Field(None, max_length=255)
+ granularity: StrictStr | None = Field(None, max_length=255)
+ limit: StrictInt | StrictStr | None = None
+ markup_type: Literal["markdown", "html"] | None = None
+ show_bubbles: StrictBool | None = None
+
+ @field_validator("limit")
+ @classmethod
+ def validate_legacy_limit(cls, value: int | str | None) -> int | str | None:
+ """Bound the ignored legacy row limit and reject non-numeric strings."""
+ if value is None:
+ return None
+ if isinstance(value, str):
+ if not value.isascii() or not value.isdigit():
+ raise ValueError("legacy Sunburst limit must be an integer")
+ numeric_value = int(value)
+ else:
+ numeric_value = value
+ if not 1 <= numeric_value <= 50000:
+ raise ValueError("legacy Sunburst limit must be between 1 and 50000")
+ return value
+
+
+_NATIVE_FORM_DATA_MARKER = "_mcp_native_form_data"
+_SUNBURST_IGNORED_LEGACY_FIELDS = frozenset(
+ {
+ "country_fieldtype",
+ "entity",
+ "granularity",
+ "limit",
+ "markup_type",
+ "show_bubbles",
+ }
+)
+
+
+class SunburstChartConfig(BaseChartConfig):
+ """Config for the ECharts Sunburst plugin (viz_type ``sunburst_v2``)."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ chart_type: Literal["sunburst"] = "sunburst"
+ viz_type: Literal["sunburst_v2"] = Field(
+ "sunburst_v2",
+ description="Exact Superset frontend visualization tag",
+ )
+ hierarchy: List[ColumnRef] = Field(
+ ...,
+ min_length=1,
+ description=(
+ "Hierarchy dimensions in ring order, from the innermost/root level "
+ "to the outermost/leaf level"
+ ),
+ validation_alias=AliasChoices("hierarchy", "columns", "groupby"),
+ )
+ metric: ColumnRef = Field(
+ ...,
+ description=(
+ "Primary metric used to size arcs. Use aggregate for a SIMPLE "
+ "adhoc metric, saved_metric=True for a dataset metric, or "
+ "sql_expression plus label for a SQL metric."
+ ),
+ )
+ secondary_metric: ColumnRef | None = Field(
+ None,
+ description=(
+ "Optional metric whose ratio to the primary metric drives the "
+ "sequential color scale (for example, profit/revenue represents "
+ "margin). When omitted, arc colors are categorical."
+ ),
+ validation_alias=AliasChoices("secondary_metric", "secondaryMetric"),
+ )
+ filters: List[FilterConfig] | None = Field(
+ None,
+ description=(
+ "Structured WHERE filters (column/op/value/clause). An omitted list is "
+ "preserved on updates; an explicit [] clears saved filters."
+ ),
+ )
+ time_range: str | None = Field(
+ None,
+ min_length=1,
+ max_length=1000,
+ description=(
+ "Superset time range, for example 'Last year', "
+ "'2025-01-01 : 2025-12-31', or 'No filter'"
+ ),
+ )
+ time_grain: TimeGrain | None = Field(
+ None,
+ description="Optional bucket for temporal hierarchy columns",
+ validation_alias=AliasChoices("time_grain", "time_grain_sqla"),
+ )
+ sort_by_metric: bool = Field(
+ False,
+ description=(
+ "Order hierarchy rows by the primary metric descending before "
+ "applying row_limit, matching the frontend buildQuery transform"
+ ),
+ )
+ row_limit: int = Field(10000, description="Maximum hierarchy rows", ge=1, le=50000)
+ color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Categorical scheme used when secondary_metric is omitted",
+ )
+ linear_color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Sequential scheme used when secondary_metric is present",
+ )
+ show_labels: bool = False
+ show_labels_threshold: float = Field(
+ 5,
+ ge=0,
+ le=100,
+ description="Minimum arc size in percentage points for showing a label",
+ )
+ show_total: bool = False
+ show_null_values: bool = Field(
+ True,
+ description="Keep null-valued hierarchy nodes in the rendered tree",
+ )
+ label_type: Literal["key", "value", "key_value"] = "key"
+ number_format: str = Field("SMART_NUMBER", min_length=1, max_length=50)
+ date_format: str = Field("smart_date", min_length=1, max_length=50)
+ currency_format: CurrencyFormat | None = None
+ # Bounded native Explore envelope/UI state. These keys occur in real saved
+ # form_data and are safe to round-trip, but remain typed so native mode does
+ # not become an escape hatch for arbitrary plugin keys or malformed nesting.
+ annotation_layers: list[dict[str, JsonValue]] | None = None
+ dashboard_id: int | None = Field(
+ None,
+ validation_alias=AliasChoices("dashboard_id", "dashboardId"),
+ serialization_alias="dashboardId",
+ )
+ dashboards: list[int] | None = None
+ datasource: str | None = Field(None, min_length=1, max_length=255)
+ extra_form_data: dict[str, JsonValue] | None = None
+ slice_id: int | None = None
+ url_params: dict[str, JsonValue] | None = None
+ standardized_form_data: SunburstStandardizedFormData | None = Field(
+ None,
+ validation_alias=AliasChoices("standardized_form_data", "standardizedFormData"),
+ serialization_alias="standardizedFormData",
+ )
+ since: str | None = Field(None, max_length=1000)
+ until: str | None = Field(None, max_length=1000)
+ time_compare: str | list[str] | None = None
+ compare_lag: int | str | None = None
+ compare_suffix: str | None = Field(None, max_length=255)
+
+ @staticmethod
+ def _looks_like_native_form_data(data: Any) -> bool:
+ """Identify saved Explore payloads without weakening typed typo checks."""
+ if type(data) is not dict:
+ return False
+ native_marker = dict.get(data, _NATIVE_FORM_DATA_MARKER)
+ viz_type = dict.get(data, "viz_type")
+ if not (
+ (type(native_marker) is bool and native_marker)
+ or (type(viz_type) is str and viz_type == "sunburst_v2")
+ ):
+ return False
+ metric = dict.get(data, "metric")
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(metric)
+ )
+ return (
+ any(
+ key in data
+ for key in (
+ "adhoc_filters",
+ "annotation_layers",
+ "datasource",
+ "extra_form_data",
+ "since",
+ "slice_id",
+ "standardizedFormData",
+ "until",
+ )
+ )
+ or type(metric) is str
+ or expression_type is not _NATIVE_METRIC_MISSING
+ or has_native_discriminator
+ )
+
+ @staticmethod
+ def _coerce_native_metric(
+ value: Any, *, allow_extensible_column_meta: bool = True
+ ) -> Any:
+ """Accept saved metric names and native SIMPLE/SQL metric objects."""
+ if type(value) is str:
+ return {"name": value, "saved_metric": True}
+ if type(value) is dict:
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(value)
+ )
+ if expression_type is _NATIVE_METRIC_MISSING:
+ if has_native_discriminator:
+ raise ValueError("Sunburst native metric requires expressionType")
+ return value
+ value = _validate_native_metric_wrapper(value)
+ else:
+ return _validate_non_dict_native_metric(value)
+
+ expression_type = value.get("expressionType")
+ label = value.get("label")
+ has_custom_label = value.get("hasCustomLabel")
+ if expression_type == "SQL":
+ sql_expression = value.get("sqlExpression")
+ return {
+ "sql_expression": sql_expression,
+ # The frontend still assigns a result label when the label is
+ # not custom. Preserve a nonempty native label first, then use
+ # the SQL expression as the same effective fallback.
+ "label": label or sql_expression,
+ "has_custom_label": has_custom_label,
+ }
+ if expression_type == "SIMPLE":
+ column = value.get("column")
+ if type(column) is dict:
+ column_model = (
+ SunburstNativeMetricColumn
+ if allow_extensible_column_meta
+ else XYNativeMetricColumn
+ )
+ column_metadata = column_model.model_validate(column)
+ name = column_metadata.column_name
+ dtype = column_metadata.type
+ elif type(column) is str:
+ name = column
+ dtype = None
+ else:
+ raise ValueError(
+ "Sunburst SIMPLE metric column must be a column name or object"
+ )
+ return {
+ "name": name,
+ "aggregate": value.get("aggregate"),
+ "label": label,
+ "has_custom_label": has_custom_label,
+ "dtype": dtype,
+ }
+ return value
+
+ @staticmethod
+ def _coerce_native_hierarchy(data: dict[str, Any]) -> None:
+ """Normalize native ``columns``/``groupby`` dimension shortcuts."""
+ hierarchy_key = next(
+ (key for key in ("hierarchy", "columns", "groupby") if key in data),
+ None,
+ )
+ if hierarchy_key is None:
+ return
+ hierarchy = data[hierarchy_key]
+ if isinstance(hierarchy, (str, dict)):
+ hierarchy = [hierarchy]
+ if isinstance(hierarchy, list):
+ data[hierarchy_key] = [
+ {"name": item} if isinstance(item, str) else item for item in hierarchy
+ ]
+ if hierarchy_key != "groupby":
+ # Saved Sunburst payloads sometimes retain an empty groupby from a
+ # previous plugin. The explicit columns/hierarchy control wins.
+ data.pop("groupby", None)
+
+ @staticmethod
+ def _coerce_native_filter(
+ native_filter: Any, data: dict[str, Any]
+ ) -> dict[str, Any] | None:
+ """Convert one native SIMPLE filter, extracting temporal controls."""
+ if not isinstance(native_filter, dict):
+ raise ValueError("Each Sunburst native adhoc_filter must be an object")
+ allowed_keys = {
+ "clause",
+ "comparator",
+ "datasourceWarning",
+ "expressionType",
+ "filterOptionName",
+ "isExtra",
+ "operator",
+ "sqlExpression",
+ "subject",
+ }
+ if unknown := set(native_filter) - allowed_keys:
+ raise ValueError(
+ "Unknown Sunburst native adhoc_filter field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ if native_filter.get("expressionType") != "SIMPLE":
+ raise ValueError(
+ "Sunburst native adhoc_filters round-trip supports SIMPLE filters "
+ "only; express filters with the typed 'filters' field"
+ )
+ operator = native_filter.get("operator")
+ clause = native_filter.get("clause", "WHERE")
+ if not isinstance(clause, str) or clause != "WHERE":
+ raise ValueError(
+ "Sunburst native SIMPLE filter clause must be 'WHERE'; "
+ "SIMPLE HAVING is unsupported because the shared query mapper "
+ "cannot preserve HAVING semantics"
+ )
+ if operator == "TEMPORAL_RANGE":
+ if "temporal_column" not in data and "granularity_sqla" not in data:
+ data["temporal_column"] = native_filter.get("subject")
+ data.setdefault("time_range", native_filter.get("comparator"))
+ return None
+ mapped_operator = "=" if operator == "==" else operator
+ comparator = native_filter.get("comparator")
+ if mapped_operator in {"IS NULL", "IS NOT NULL"}:
+ comparator = None
+ return {
+ "column": native_filter.get("subject"),
+ "clause": clause,
+ "op": mapped_operator,
+ "value": comparator,
+ }
+
+ @staticmethod
+ def _discard_legacy_saved_fields(data: dict[str, Any]) -> None:
+ """Validate and ignore only the known obsolete saved-chart controls."""
+ legacy = {
+ key: data[key] for key in _SUNBURST_IGNORED_LEGACY_FIELDS if key in data
+ }
+ if legacy:
+ SunburstLegacySavedFields.model_validate(legacy)
+ for key in legacy:
+ data.pop(key, None)
+
+ @classmethod
+ def _coerce_native_filters(cls, data: dict[str, Any]) -> None:
+ """Round-trip native saved filters into the typed filter contract."""
+ native_filters = data.pop("adhoc_filters", None)
+ if native_filters is None or "filters" in data:
+ return
+ if not isinstance(native_filters, list):
+ raise ValueError("Sunburst native adhoc_filters must be a list")
+ filters = [
+ converted
+ for native_filter in native_filters or []
+ if (converted := cls._coerce_native_filter(native_filter, data)) is not None
+ ]
+ data["filters"] = filters
+
+ @model_validator(mode="before")
+ @classmethod
+ def accept_native_form_data(cls, data: Any) -> Any:
+ """Translate native saved Sunburst form_data into the typed contract."""
+ if not isinstance(data, dict):
+ return data
+ original_data = data if type(data) is dict else None
+ data = dict(data)
+ if _native_metric_paths_require_safe_rejection(data):
+ data = _sanitize_rejected_native_metric_config(data, original_data)
+ is_native_form_data = cls._looks_like_native_form_data(data)
+ data.pop(_NATIVE_FORM_DATA_MARKER, None)
Review Comment:
Sunburst form data generated with a temporal binding contains `_mcp_dashboard_time_filter_subject`, but this adapter leaves it in place and the unknown-field check rejects it when the returned or saved config is reused. Could the native round-trip adapter handle its own generated marker, as the single-metric adapter does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4203365590
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -3867,6 +5073,134 @@ def _normalize_chart_request_input(data: Any) -> Any:
config.pop("viz_type", None)
elif config.get("chart_type") != "table":
config.pop("viz_type", None)
+
+ if config.get("chart_type") == "xy":
+ # Native Timeseries form data uses ``x_axis`` for the semantic
+ # query column, while the typed MCP contract uses the same spelling
+ # for AxisConfig presentation state. Disambiguate before the
+ # discriminated union runs: scalars and native query-column objects
+ # become ``x``; title/format/scale mappings remain ``x_axis``.
+ if "x_axis" in config and config["x_axis"] is not None:
+ native_x_axis = config["x_axis"]
+ is_native_column = not isinstance(native_x_axis, dict) or bool(
+ {
+ "column",
+ "column_name",
+ "columnName",
+ "expressionType",
+ "sqlExpression",
+ }
+ & set(native_x_axis)
+ )
+ if is_native_column:
+ if "x" in config:
+ raise ValueError(
+ "XY config cannot provide both semantic x and native "
+ "x_axis query columns"
+ )
+ if isinstance(native_x_axis, dict):
+ allowed_x_axis_keys = {
+ "column",
+ "column_name",
+ "columnName",
+ "columnType",
+ "expressionType",
+ "label",
+ "sqlExpression",
+ "timeGrain",
+ }
+ if unknown := set(native_x_axis) - allowed_x_axis_keys:
+ raise ValueError(
+ "Unknown XY native x_axis field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ expression_type = native_x_axis.get("expressionType")
+ if expression_type == "SIMPLE":
+ column = native_x_axis.get("column")
+ if isinstance(column, dict):
+ column = XYNativeMetricColumn.model_validate(
+ column
+ ).column_name
+ native_x_axis = {
+ "name": native_x_axis.get("column_name")
+ or native_x_axis.get("columnName")
+ or column,
+ }
+ elif (
+ expression_type == "SQL"
+ and native_x_axis.get("columnType") == "BASE_AXIS"
+ ):
+ native_x_axis = {
+ "name": native_x_axis.get("sqlExpression"),
Review Comment:
Fixed in 2732afbb1c85ca572d8f7f09185f3e0a4a88db7c.
The BASE_AXIS normalizer carries timeGrain into config.time_grain when neither panel-level time_grain nor time_grain_sqla was supplied. Explicit panel values take precedence. test_native_xy_base_axis_grain_reaches_generated_and_updated_queries checks monthly bucketing in the actual query column for generate, update, and preview requests, plus panel overrides. The three monthly cases failed before and pass after.
Validation: 9,113 passed, 4 skipped across the full MCP/common unit suites; pre-commit passed on all 82 branch-changed files, including mypy.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,790 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
Review Comment:
Fixed in 2732afbb1c85ca572d8f7f09185f3e0a4a88db7c.
Omitted group_by now preserves saved groupby on same-viz XY updates; explicit group_by: [] clears it, and supplied grouping replaces it. Cross-viz updates do not inherit source grouping. test_xy_update_preserves_grouping_only_when_omitted exercises a saved region-grouped chart with the same x/y and stacked: true, checks the resulting query series, and covers omission, clear, replacement, native alias, and cross-viz behavior for all four XY kinds. The four omission cases failed before and pass after. The behavior is documented in the MCP docs.
Validation: 9,113 passed, 4 skipped across the full MCP/common unit suites; pre-commit passed on all 82 branch-changed files, including mypy.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +427,1309 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ extra_metric = sort_metric
+ else:
+ extra_metric = None
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ sort_metric=extra_metric,
+ )
+ if extra_metric is not None:
+ query.setdefault("metrics", []).append(extra_metric)
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = (
+ _label(explicit_x_axis)
+ if explicit_x_axis
+ else _x_axis_label(form_data, frontend_strict=True)
+ )
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
+ extras = query.get("extras")
+ if isinstance(extras, dict):
+ extras.pop("time_grain_sqla", None)
+ if not extras:
+ query.pop("extras", None)
+ elif not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
query["post_processing"] = post_processing
- # ``granularity`` does two jobs downstream: it names the temporal column the
- # time range filters on, and it is the column ``time_grain_sqla`` buckets
- # (``models/helpers.py`` swaps a selected column for its timestamp expression
- # when that column equals ``granularity``). Only the first job depends on
- # there being an active range, so set it whenever form data carries one โ
- # matching ``extractExtras.ts``, which sets it unconditionally. Gating it on
- # ``time_range`` dropped the bucketing, so an ordinary "all-time totals by
- # month" chart exported one row per raw timestamp instead of one per month.
- if granularity := form_data.get("granularity") or form_data.get("granularity_sqla"):
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
+ page_size = int(form_data.get("server_page_length") or 0)
+ configured_limit = int(form_data.get("row_limit") or 0)
+ query["row_limit"] = (
+ min(page_size, configured_limit) if configured_limit else page_size
Review Comment:
Fixed in 2732afbb1c85ca572d8f7f09185f3e0a4a88db7c.
Pagination takes the effective query row limit (including the explicit builder/compile cap) and applies min only when page_size is positive. A zero or missing page size therefore keeps row_limit=2 instead of falling back to the deployment default. test_table_pagination_respects_explicit_limit_without_positive_page_size and test_table_pagination_keeps_query_builder_caller_cap cover Table and AG Grid Table, zero/missing/positive page sizes, and a saved row_limit=1000 overridden by the caller's two-row cap. Eight regression cases failed before and pass after.
Validation: 9,113 passed, 4 skipped across the full MCP/common unit suites; pre-commit passed on all 82 branch-changed files, including mypy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4190031875
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -1634,6 +1652,1045 @@ def reject_metric_style_groupby(self) -> "TreemapChartUpdateConfig":
return self
+class SunburstStandardizedControls(UnknownFieldCheckMixin):
+ """Bounded shared controls retained by Explore across viz changes."""
+
+ model_config = ConfigDict(extra="ignore")
+
+ metrics: list[JsonValue] = Field(default_factory=list)
+ columns: list[JsonValue] = Field(default_factory=list)
+
+
+class SunburstStandardizedFormData(UnknownFieldCheckMixin):
+ """Validated shape of Explore's cross-plugin UI memory."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ controls: SunburstStandardizedControls
+ memorized_form_data: list[tuple[str, dict[str, JsonValue]]] = Field(
+ default_factory=list,
+ validation_alias=AliasChoices("memorized_form_data", "memorizedFormData"),
+ serialization_alias="memorizedFormData",
+ )
+
+
+_NATIVE_COLUMN_META_MAX_DEPTH = 8
+_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS = 128
+_NATIVE_COLUMN_META_MAX_TOTAL_VALUES = 512
+_NATIVE_COLUMN_META_MAX_KEY_BYTES = 1_024
+_NATIVE_COLUMN_META_MAX_STRING_BYTES = 16 * 1_024
+_NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES = 64 * 1_024
+_NATIVE_COLUMN_META_MAX_INT_BITS = 64
+_NATIVE_METRIC_ALLOWED_KEYS = frozenset(
+ {
+ "aggregate",
+ "column",
+ "datasourceWarning",
+ "expressionType",
+ "hasCustomLabel",
+ "label",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_EXPRESSION_TYPES = frozenset({"SIMPLE", "SQL"})
+_NATIVE_METRIC_DISCRIMINATOR_KEYS = frozenset(
+ {
+ "column",
+ "datasourceWarning",
+ "hasCustomLabel",
+ "optionName",
+ "sqlExpression",
+ }
+)
+_NATIVE_METRIC_MISSING = object()
+_NATIVE_METRIC_INPUT_KEYS = frozenset(
+ {"metric", "metrics", "secondary_metric", "secondaryMetric"}
+)
+_NATIVE_METRIC_INVALID_VALUE = b"invalid native metric value"
+_NATIVE_METRIC_INVALID_KEY = "__invalid_native_metric_key__"
+_NATIVE_METRIC_SANITIZE_MAX_DEPTH = 16
+_NATIVE_METRIC_SANITIZE_MAX_VALUES = 1_024
+_NATIVE_METRIC_ERROR_STRING_LENGTH = 128
+
+
+def _native_column_meta_string_bytes(value: str, *, key: bool = False) -> int:
+ """Return exact UTF-8 size for a trusted built-in string."""
+ kind = "key" if key else "string"
+ try:
+ size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(
+ f"Sunburst native metric column metadata {kind} is invalid"
+ ) from ex
+ limit = (
+ _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ if key
+ else _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ )
+ if size > limit:
+ raise ValueError(f"Sunburst native metric column metadata {kind} is too long")
+ return size
+
+
+def _sanitize_native_metric_dict(
+ value: dict[Any, Any],
+ depth: int,
+ remaining: list[int],
+ ancestors: set[int],
+) -> tuple[Any, bool]:
+ """Project one exact dictionary through built-in operations only."""
+ value_id = id(value)
+ if value_id in ancestors:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ return {
+ f"invalid_{index}": _NATIVE_METRIC_INVALID_VALUE
+ for index in range(_NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS + 1)
+ }, True
+
+ ancestors.add(value_id)
+ projected: dict[str, Any] = {}
+ requires_rejection = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ projected[_NATIVE_METRIC_INVALID_KEY] = _NATIVE_METRIC_INVALID_VALUE
+ requires_rejection = True
+ continue
+ try:
+ key_size = len(str.encode(key, "utf-8"))
+ except UnicodeEncodeError:
+ key_size = _NATIVE_COLUMN_META_MAX_KEY_BYTES + 1
+ if (
+ str.__len__(key) > _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ or key_size > _NATIVE_COLUMN_META_MAX_KEY_BYTES
+ ):
+ projected["x" * (_NATIVE_COLUMN_META_MAX_KEY_BYTES + 1)] = (
+ _NATIVE_METRIC_INVALID_VALUE
+ )
+ requires_rejection = True
+ continue
+ projected_item, item_requires_rejection = _sanitize_native_metric_value(
+ item,
+ depth=depth + 1,
+ remaining=remaining,
+ ancestors=ancestors,
+ )
+ projected[key] = projected_item
+ requires_rejection = requires_rejection or item_requires_rejection
+ ancestors.remove(value_id)
+ return projected, requires_rejection
+
+
+def _sanitize_native_metric_list(
+ value: list[Any],
+ depth: int,
+ remaining: list[int],
+ ancestors: set[int],
+) -> tuple[Any, bool]:
+ """Project one exact list through built-in operations only."""
+ value_id = id(value)
+ if value_id in ancestors:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ return [_NATIVE_METRIC_INVALID_VALUE] * (
+ _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS + 1
+ ), True
+
+ ancestors.add(value_id)
+ projected_items: list[Any] = []
+ requires_rejection = False
+ for index in range(list.__len__(value)):
+ projected_item, item_requires_rejection = _sanitize_native_metric_value(
+ list.__getitem__(value, index),
+ depth=depth + 1,
+ remaining=remaining,
+ ancestors=ancestors,
+ )
+ projected_items.append(projected_item)
+ requires_rejection = requires_rejection or item_requires_rejection
+ ancestors.remove(value_id)
+ return projected_items, requires_rejection
+
+
+def _sanitize_native_metric_scalar(value: Any) -> tuple[Any, bool] | None:
+ """Project an exact scalar, or signal that ``value`` is a container."""
+ value_type = type(value)
+ if value_type is str:
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+ if (
+ str.__len__(value) > _NATIVE_COLUMN_META_MAX_STRING_BYTES
+ or encoded_size > _NATIVE_COLUMN_META_MAX_STRING_BYTES * 4
+ ):
+ return "x" * (_NATIVE_COLUMN_META_MAX_STRING_BYTES + 1), True
+ return (
+ str.__getitem__(value, slice(0, _NATIVE_METRIC_ERROR_STRING_LENGTH)),
+ False,
+ )
+ if value_type is int and int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ return 1 << (_NATIVE_COLUMN_META_MAX_INT_BITS + 1), True
+ if value_type in (int, float, bool, type(None), ColumnRef):
+ return value, False
+ return None
+
+
+def _sanitize_native_metric_value(
+ value: Any,
+ *,
+ depth: int = 0,
+ remaining: list[int] | None = None,
+ ancestors: set[int] | None = None,
+) -> tuple[Any, bool]:
+ """Build a bounded exact-type projection without dispatching object hooks."""
+ if remaining is None:
+ remaining = [_NATIVE_METRIC_SANITIZE_MAX_VALUES]
+ if ancestors is None:
+ ancestors = set()
+ remaining[0] -= 1
+ if remaining[0] < 0 or depth > _NATIVE_METRIC_SANITIZE_MAX_DEPTH:
+ return _NATIVE_METRIC_INVALID_VALUE, True
+
+ if (scalar := _sanitize_native_metric_scalar(value)) is not None:
+ return scalar
+ value_type = type(value)
+ if value_type is dict:
+ return _sanitize_native_metric_dict(value, depth, remaining, ancestors)
+ if value_type is list:
+ return _sanitize_native_metric_list(value, depth, remaining, ancestors)
+ return _NATIVE_METRIC_INVALID_VALUE, True
+
+
+def _metric_paths_in_dict_require_safe_rejection(
+ value: dict[Any, Any], depth: int, remaining: list[int], seen: set[int]
+) -> bool:
+ """Inspect metric-bearing paths of one exact dictionary."""
+ value_id = id(value)
+ if value_id in seen:
+ return False
+ seen.add(value_id)
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ return True
+ if key in _NATIVE_METRIC_INPUT_KEYS:
+ if _sanitize_native_metric_value(nested)[1]:
+ return True
+ elif _native_metric_paths_require_safe_rejection(
+ nested, depth=depth + 1, remaining=remaining, seen=seen
+ ):
+ return True
+ return False
+
+
+def _metric_paths_in_list_require_safe_rejection(
+ value: list[Any], depth: int, remaining: list[int], seen: set[int]
+) -> bool:
+ """Inspect metric-bearing paths of one exact list."""
+ value_id = id(value)
+ if value_id in seen:
+ return False
+ seen.add(value_id)
+ return any(
+ _native_metric_paths_require_safe_rejection(
+ list.__getitem__(value, index),
+ depth=depth + 1,
+ remaining=remaining,
+ seen=seen,
+ )
+ for index in range(list.__len__(value))
+ )
+
+
+def _native_metric_paths_require_safe_rejection(
+ value: Any,
+ *,
+ depth: int = 0,
+ remaining: list[int] | None = None,
+ seen: set[int] | None = None,
+) -> bool:
+ """Find unsafe metric subtrees without consuming non-exact containers."""
+ if remaining is None:
+ remaining = [_NATIVE_METRIC_SANITIZE_MAX_VALUES]
+ if seen is None:
+ seen = set()
+ remaining[0] -= 1
+ if remaining[0] < 0 or depth > _NATIVE_METRIC_SANITIZE_MAX_DEPTH:
+ return False
+ if type(value) is dict:
+ return _metric_paths_in_dict_require_safe_rejection(
+ value, depth, remaining, seen
+ )
+ if type(value) is list:
+ return _metric_paths_in_list_require_safe_rejection(
+ value, depth, remaining, seen
+ )
+ return False
+
+
+def _replace_exact_dict_contents(
+ value: dict[Any, Any], replacement: dict[str, Any]
+) -> None:
+ """Replace exact-dict input through built-in operations only."""
+ dict.clear(value)
+ for key, item in dict.items(replacement):
+ dict.__setitem__(value, key, item)
+
+
+def _sanitize_rejected_native_metric_config(
+ data: dict[str, Any], original_data: dict[str, Any] | None
+) -> dict[str, Any]:
+ """Project a rejected config and replace Pydantic's retained input."""
+ projected, _ = _sanitize_native_metric_value(data)
+ if type(projected) is not dict:
+ projected = {_NATIVE_METRIC_INVALID_KEY: _NATIVE_METRIC_INVALID_VALUE}
+ if original_data is not None:
+ _replace_exact_dict_contents(original_data, projected)
+ return projected
+
+
+def _validate_native_metric_string(
+ value: Any, field: str, max_length: int
+) -> str | None:
+ """Validate one optional exact string in the closed native metric wrapper."""
+ if value is None:
+ return None
+ if type(value) is not str:
+ raise ValueError(f"Sunburst native metric {field} must be a string")
+ try:
+ encoded_size = len(str.encode(value, "utf-8"))
+ except UnicodeEncodeError as ex:
+ raise ValueError(f"Sunburst native metric {field} is invalid") from ex
+ if str.__len__(value) > max_length or encoded_size > max_length * 4:
+ raise ValueError(f"Sunburst native metric {field} is too long")
+ return value
+
+
+def _inspect_native_metric_expression_type(value: Any) -> tuple[object, bool]:
+ """Inspect an exact wrapper without hashing or comparing hostile objects."""
+ if type(value) is not dict:
+ return _NATIVE_METRIC_MISSING, False
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+ expression_type: object = _NATIVE_METRIC_MISSING
+ has_native_discriminator = False
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key in _NATIVE_METRIC_DISCRIMINATOR_KEYS:
+ has_native_discriminator = True
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError(
+ "Sunburst native metric expressionType must be a string"
+ )
+ if item not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError(
+ "Sunburst native metric expressionType must be SIMPLE or SQL"
+ )
+ expression_type = item
+ return expression_type, has_native_discriminator
+
+
+def _validate_native_metric_column_value(value: Any) -> None:
+ """Validate a wrapper column without dispatching container or scalar hooks."""
+ if type(value) is str:
+ _validate_native_metric_string(value, "column", 255)
+ elif type(value) is dict:
+ _validate_bounded_native_column_meta(value)
+ elif value is not None:
+ raise ValueError("Sunburst native metric column must be a string or object")
+
+
+def _validate_native_metric_item(key: str, item: Any) -> object:
+ """Validate one value after its wrapper key is known to be an exact string."""
+ if key == "expressionType":
+ if type(item) is not str:
+ raise ValueError("Sunburst native metric expressionType must be a string")
+ _validate_native_metric_string(item, "expressionType", 10)
+ return item
+ if key in ("label", "optionName"):
+ _validate_native_metric_string(item, key, 500)
+ elif key == "sqlExpression":
+ _validate_native_metric_string(item, key, 2_000)
+ elif key == "aggregate":
+ _validate_native_metric_string(item, key, 32)
+ elif key in ("datasourceWarning", "hasCustomLabel"):
+ if item is not None and type(item) is not bool:
+ raise ValueError(f"Sunburst native metric {key} must be a boolean")
+ elif key == "column":
+ _validate_native_metric_column_value(item)
+ return _NATIVE_METRIC_MISSING
+
+
+def _validate_native_metric_wrapper(value: Any) -> dict[str, Any]:
+ """Validate and copy the closed native metric wrapper without calling hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric must be a string or object")
+ if dict.__len__(value) > len(_NATIVE_METRIC_ALLOWED_KEYS):
+ raise ValueError("Sunburst native metric has too many fields")
+
+ validated: dict[str, Any] = {}
+ expression_type: object = _NATIVE_METRIC_MISSING
+ for key, item in dict.items(value):
+ if type(key) is not str:
+ raise ValueError("Sunburst native metric keys must be strings")
+ _validate_native_metric_string(key, "field name", 255)
+ if key not in _NATIVE_METRIC_ALLOWED_KEYS:
+ raise ValueError(f"Unknown Sunburst native metric field: {key}")
+ item_expression_type = _validate_native_metric_item(key, item)
+ if item_expression_type is not _NATIVE_METRIC_MISSING:
+ expression_type = item_expression_type
+ validated[key] = item
+
+ if expression_type is _NATIVE_METRIC_MISSING:
+ raise ValueError("Sunburst native metric requires expressionType")
+ if expression_type not in _NATIVE_METRIC_EXPRESSION_TYPES:
+ raise ValueError("Sunburst native metric expressionType must be SIMPLE or SQL")
+ return validated
+
+
+def _validate_non_dict_native_metric(value: Any) -> ColumnRef:
+ """Accept an existing model while rejecting hook-bearing dict subclasses."""
+ if type(value) is ColumnRef:
+ return value
+ raise ValueError("Sunburst native metric must be a string or object")
+
+
+def _native_column_meta_dict_children(
+ value: dict[Any, Any], depth: int
+) -> tuple[list[tuple[Any, int]], int]:
+ """Read exact-dict children while validating bounded primitive keys."""
+ if dict.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata object is too large")
+ children: list[tuple[Any, int]] = []
+ string_bytes = 0
+ for key, nested in dict.items(value):
+ if type(key) is not str:
+ raise ValueError(
+ "Sunburst native metric column metadata keys must be strings"
+ )
+ string_bytes += _native_column_meta_string_bytes(key, key=True)
+ children.append((nested, depth + 1))
+ return children, string_bytes
+
+
+def _native_column_meta_list_children(
+ value: list[Any], depth: int
+) -> list[tuple[Any, int]]:
+ """Read exact-list children without dispatching subclass hooks."""
+ if list.__len__(value) > _NATIVE_COLUMN_META_MAX_CONTAINER_ITEMS:
+ raise ValueError("Sunburst native metric column metadata array is too large")
+ return [
+ (list.__getitem__(value, index), depth + 1)
+ for index in range(list.__len__(value))
+ ]
+
+
+def _validate_native_column_meta_scalar(value: Any) -> int:
+ """Validate one exact JSON scalar and return its string-byte contribution."""
+ value_type = type(value)
+ if value_type is str:
+ return _native_column_meta_string_bytes(value)
+ if value_type is int:
+ if int.bit_length(value) > _NATIVE_COLUMN_META_MAX_INT_BITS:
+ raise ValueError(
+ "Sunburst native metric column metadata integer is too large"
+ )
+ return 0
+ if value_type is float:
+ if not math.isfinite(value):
+ raise ValueError(
+ "Sunburst native metric column metadata number must be finite"
+ )
+ return 0
+ if value_type not in (bool, type(None)):
+ raise ValueError(
+ "Sunburst native metric column metadata contains an invalid value"
+ )
+ return 0
+
+
+def _validate_bounded_native_column_meta(value: Any) -> Any:
+ """Validate an extensible frontend ColumnMeta object without invoking hooks."""
+ if type(value) is not dict:
+ raise ValueError("Sunburst native metric column must be an object")
+
+ total_values = 0
+ total_string_bytes = 0
+ stack: list[tuple[Any, int]] = [(value, 0)]
+ while stack:
+ item, depth = stack.pop()
+ total_values += 1
+ if total_values > _NATIVE_COLUMN_META_MAX_TOTAL_VALUES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ if depth > _NATIVE_COLUMN_META_MAX_DEPTH:
+ raise ValueError(
+ "Sunburst native metric column metadata is too deeply nested"
+ )
+
+ item_type = type(item)
+ if item_type is dict:
+ children, key_bytes = _native_column_meta_dict_children(item, depth)
+ total_string_bytes += key_bytes
+ stack.extend(children)
+ elif item_type is list:
+ stack.extend(_native_column_meta_list_children(item, depth))
+ else:
+ total_string_bytes += _validate_native_column_meta_scalar(item)
+
+ if total_string_bytes > _NATIVE_COLUMN_META_MAX_TOTAL_STRING_BYTES:
+ raise ValueError("Sunburst native metric column metadata is too large")
+ return value
+
+
+class SunburstNativeMetricColumn(BaseModel):
+ """Owned projection of the frontend's intentionally extensible ColumnMeta."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ type: StrictStr | None = Field(None, max_length=255)
+
+ @model_validator(mode="before")
+ @classmethod
+ def validate_bounded_column_meta(cls, value: Any) -> Any:
+ """Bound the full open object before projecting the fields we own."""
+ projected, requires_safe_rejection = _sanitize_native_metric_value(value)
+ if requires_safe_rejection:
+ if type(value) is dict:
+ if type(projected) is not dict:
+ projected = {
+ _NATIVE_METRIC_INVALID_KEY: _NATIVE_METRIC_INVALID_VALUE
+ }
+ _replace_exact_dict_contents(value, projected)
+ value = projected
+ else:
+ # Let Pydantic reject the safe scalar so the original subclass
+ # is not retained as the input of an in-validator exception.
+ return projected
+ value = _validate_bounded_native_column_meta(value)
+ snake_name = dict.get(value, "column_name", _NATIVE_METRIC_MISSING)
+ camel_name = dict.get(value, "columnName", _NATIVE_METRIC_MISSING)
+ if (
+ snake_name is not _NATIVE_METRIC_MISSING
+ and camel_name is not _NATIVE_METRIC_MISSING
+ ):
+ if type(snake_name) is not str or type(camel_name) is not str:
+ raise ValueError(
+ "column_name and columnName must both be exact strings"
+ )
+ if snake_name != camel_name:
+ raise ValueError(
+ "column_name and columnName must match when both are provided"
+ )
+ return value
+
+
+class XYNativeMetricColumn(UnknownFieldCheckMixin):
+ """Closed legacy ColumnMeta projection retained for native XY axes."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ column_name: StrictStr = Field(
+ ...,
+ min_length=1,
+ max_length=255,
+ validation_alias=AliasChoices("column_name", "columnName"),
+ )
+ id: StrictInt | None = Field(None, ge=0)
+ type: StrictStr | None = Field(None, max_length=255)
+ type_generic: StrictInt | None = Field(None, ge=0, le=10)
+ groupby: StrictBool | None = None
+ is_dttm: StrictBool | None = None
+ filterable: StrictBool | None = None
+ verbose_name: StrictStr | None = Field(None, max_length=500)
+ description: StrictStr | None = Field(None, max_length=2000)
+ expression: StrictStr | None = Field(None, max_length=2000)
+ database_expression: StrictStr | None = Field(None, max_length=2000)
+ python_date_format: StrictStr | None = Field(None, max_length=255)
+ option_name: StrictStr | None = Field(
+ None,
+ max_length=500,
+ validation_alias=AliasChoices("option_name", "optionName"),
+ )
+ filter_by: StrictStr | None = Field(
+ None,
+ max_length=255,
+ validation_alias=AliasChoices("filter_by", "filterBy"),
+ )
+ value: StrictStr | None = Field(None, max_length=500)
+ advanced_data_type: StrictStr | None = Field(None, max_length=255)
+ uuid: StrictStr | None = Field(None, max_length=64)
+
+
+class SunburstLegacySavedFields(BaseModel):
+ """Known obsolete controls tolerated only while reading saved Sunbursts."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ country_fieldtype: StrictStr | None = Field(None, max_length=255)
+ entity: StrictStr | None = Field(None, max_length=255)
+ granularity: StrictStr | None = Field(None, max_length=255)
+ limit: StrictInt | StrictStr | None = None
+ markup_type: Literal["markdown", "html"] | None = None
+ show_bubbles: StrictBool | None = None
+
+ @field_validator("limit")
+ @classmethod
+ def validate_legacy_limit(cls, value: int | str | None) -> int | str | None:
+ """Bound the ignored legacy row limit and reject non-numeric strings."""
+ if value is None:
+ return None
+ if isinstance(value, str):
+ if not value.isascii() or not value.isdigit():
+ raise ValueError("legacy Sunburst limit must be an integer")
+ numeric_value = int(value)
+ else:
+ numeric_value = value
+ if not 1 <= numeric_value <= 50000:
+ raise ValueError("legacy Sunburst limit must be between 1 and 50000")
+ return value
+
+
+_NATIVE_FORM_DATA_MARKER = "_mcp_native_form_data"
+_SUNBURST_IGNORED_LEGACY_FIELDS = frozenset(
+ {
+ "country_fieldtype",
+ "entity",
+ "granularity",
+ "limit",
+ "markup_type",
+ "show_bubbles",
+ }
+)
+
+
+class SunburstChartConfig(BaseChartConfig):
+ """Config for the ECharts Sunburst plugin (viz_type ``sunburst_v2``)."""
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ chart_type: Literal["sunburst"] = "sunburst"
+ viz_type: Literal["sunburst_v2"] = Field(
+ "sunburst_v2",
+ description="Exact Superset frontend visualization tag",
+ )
+ hierarchy: List[ColumnRef] = Field(
+ ...,
+ min_length=1,
+ description=(
+ "Hierarchy dimensions in ring order, from the innermost/root level "
+ "to the outermost/leaf level"
+ ),
+ validation_alias=AliasChoices("hierarchy", "columns", "groupby"),
+ )
+ metric: ColumnRef = Field(
+ ...,
+ description=(
+ "Primary metric used to size arcs. Use aggregate for a SIMPLE "
+ "adhoc metric, saved_metric=True for a dataset metric, or "
+ "sql_expression plus label for a SQL metric."
+ ),
+ )
+ secondary_metric: ColumnRef | None = Field(
+ None,
+ description=(
+ "Optional metric whose ratio to the primary metric drives the "
+ "sequential color scale (for example, profit/revenue represents "
+ "margin). When omitted, arc colors are categorical."
+ ),
+ validation_alias=AliasChoices("secondary_metric", "secondaryMetric"),
+ )
+ filters: List[FilterConfig] | None = Field(
+ None,
+ description=(
+ "Structured WHERE filters (column/op/value/clause). An omitted list is "
+ "preserved on updates; an explicit [] clears saved filters."
+ ),
+ )
+ time_range: str | None = Field(
+ None,
+ min_length=1,
+ max_length=1000,
+ description=(
+ "Superset time range, for example 'Last year', "
+ "'2025-01-01 : 2025-12-31', or 'No filter'"
+ ),
+ )
+ time_grain: TimeGrain | None = Field(
+ None,
+ description="Optional bucket for temporal hierarchy columns",
+ validation_alias=AliasChoices("time_grain", "time_grain_sqla"),
+ )
+ sort_by_metric: bool = Field(
+ False,
+ description=(
+ "Order hierarchy rows by the primary metric descending before "
+ "applying row_limit, matching the frontend buildQuery transform"
+ ),
+ )
+ row_limit: int = Field(10000, description="Maximum hierarchy rows", ge=1, le=50000)
+ color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Categorical scheme used when secondary_metric is omitted",
+ )
+ linear_color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Sequential scheme used when secondary_metric is present",
+ )
+ show_labels: bool = False
+ show_labels_threshold: float = Field(
+ 5,
+ ge=0,
+ le=100,
+ description="Minimum arc size in percentage points for showing a label",
+ )
+ show_total: bool = False
+ show_null_values: bool = Field(
+ True,
+ description="Keep null-valued hierarchy nodes in the rendered tree",
+ )
+ label_type: Literal["key", "value", "key_value"] = "key"
+ number_format: str = Field("SMART_NUMBER", min_length=1, max_length=50)
+ date_format: str = Field("smart_date", min_length=1, max_length=50)
+ currency_format: CurrencyFormat | None = None
+ # Bounded native Explore envelope/UI state. These keys occur in real saved
+ # form_data and are safe to round-trip, but remain typed so native mode does
+ # not become an escape hatch for arbitrary plugin keys or malformed nesting.
+ annotation_layers: list[dict[str, JsonValue]] | None = None
+ dashboard_id: int | None = Field(
+ None,
+ validation_alias=AliasChoices("dashboard_id", "dashboardId"),
+ serialization_alias="dashboardId",
+ )
+ dashboards: list[int] | None = None
+ datasource: str | None = Field(None, min_length=1, max_length=255)
+ extra_form_data: dict[str, JsonValue] | None = None
+ slice_id: int | None = None
+ url_params: dict[str, JsonValue] | None = None
+ standardized_form_data: SunburstStandardizedFormData | None = Field(
+ None,
+ validation_alias=AliasChoices("standardized_form_data", "standardizedFormData"),
+ serialization_alias="standardizedFormData",
+ )
+ since: str | None = Field(None, max_length=1000)
+ until: str | None = Field(None, max_length=1000)
+ time_compare: str | list[str] | None = None
+ compare_lag: int | str | None = None
+ compare_suffix: str | None = Field(None, max_length=255)
+
+ @staticmethod
+ def _looks_like_native_form_data(data: Any) -> bool:
+ """Identify saved Explore payloads without weakening typed typo checks."""
+ if type(data) is not dict:
+ return False
+ native_marker = dict.get(data, _NATIVE_FORM_DATA_MARKER)
+ viz_type = dict.get(data, "viz_type")
+ if not (
+ (type(native_marker) is bool and native_marker)
+ or (type(viz_type) is str and viz_type == "sunburst_v2")
+ ):
+ return False
+ metric = dict.get(data, "metric")
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(metric)
+ )
+ return (
+ any(
+ key in data
+ for key in (
+ "adhoc_filters",
+ "annotation_layers",
+ "datasource",
+ "extra_form_data",
+ "since",
+ "slice_id",
+ "standardizedFormData",
+ "until",
+ )
+ )
+ or type(metric) is str
+ or expression_type is not _NATIVE_METRIC_MISSING
+ or has_native_discriminator
+ )
+
+ @staticmethod
+ def _coerce_native_metric(
+ value: Any, *, allow_extensible_column_meta: bool = True
+ ) -> Any:
+ """Accept saved metric names and native SIMPLE/SQL metric objects."""
+ if type(value) is str:
+ return {"name": value, "saved_metric": True}
+ if type(value) is dict:
+ expression_type, has_native_discriminator = (
+ _inspect_native_metric_expression_type(value)
+ )
+ if expression_type is _NATIVE_METRIC_MISSING:
+ if has_native_discriminator:
+ raise ValueError("Sunburst native metric requires expressionType")
+ return value
+ value = _validate_native_metric_wrapper(value)
+ else:
+ return _validate_non_dict_native_metric(value)
+
+ expression_type = value.get("expressionType")
+ label = value.get("label")
+ has_custom_label = value.get("hasCustomLabel")
+ if expression_type == "SQL":
+ sql_expression = value.get("sqlExpression")
+ return {
+ "sql_expression": sql_expression,
+ # The frontend still assigns a result label when the label is
+ # not custom. Preserve a nonempty native label first, then use
+ # the SQL expression as the same effective fallback.
+ "label": label or sql_expression,
+ "has_custom_label": has_custom_label,
+ }
+ if expression_type == "SIMPLE":
+ column = value.get("column")
+ if type(column) is dict:
+ column_model = (
+ SunburstNativeMetricColumn
+ if allow_extensible_column_meta
+ else XYNativeMetricColumn
+ )
+ column_metadata = column_model.model_validate(column)
+ name = column_metadata.column_name
+ dtype = column_metadata.type
+ elif type(column) is str:
+ name = column
+ dtype = None
+ else:
+ raise ValueError(
+ "Sunburst SIMPLE metric column must be a column name or object"
+ )
+ return {
+ "name": name,
+ "aggregate": value.get("aggregate"),
+ "label": label,
+ "has_custom_label": has_custom_label,
+ "dtype": dtype,
+ }
+ return value
+
+ @staticmethod
+ def _coerce_native_hierarchy(data: dict[str, Any]) -> None:
+ """Normalize native ``columns``/``groupby`` dimension shortcuts."""
+ hierarchy_key = next(
+ (key for key in ("hierarchy", "columns", "groupby") if key in data),
+ None,
+ )
+ if hierarchy_key is None:
+ return
+ hierarchy = data[hierarchy_key]
+ if isinstance(hierarchy, (str, dict)):
+ hierarchy = [hierarchy]
+ if isinstance(hierarchy, list):
+ data[hierarchy_key] = [
+ {"name": item} if isinstance(item, str) else item for item in hierarchy
+ ]
+ if hierarchy_key != "groupby":
+ # Saved Sunburst payloads sometimes retain an empty groupby from a
+ # previous plugin. The explicit columns/hierarchy control wins.
+ data.pop("groupby", None)
+
+ @staticmethod
+ def _coerce_native_filter(
+ native_filter: Any, data: dict[str, Any]
+ ) -> dict[str, Any] | None:
+ """Convert one native SIMPLE filter, extracting temporal controls."""
+ if not isinstance(native_filter, dict):
+ raise ValueError("Each Sunburst native adhoc_filter must be an object")
+ allowed_keys = {
+ "clause",
+ "comparator",
+ "datasourceWarning",
+ "expressionType",
+ "filterOptionName",
+ "isExtra",
+ "operator",
+ "sqlExpression",
+ "subject",
+ }
+ if unknown := set(native_filter) - allowed_keys:
+ raise ValueError(
+ "Unknown Sunburst native adhoc_filter field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ if native_filter.get("expressionType") != "SIMPLE":
+ raise ValueError(
+ "Sunburst native adhoc_filters round-trip supports SIMPLE filters "
+ "only; express filters with the typed 'filters' field"
+ )
+ operator = native_filter.get("operator")
+ clause = native_filter.get("clause", "WHERE")
+ if not isinstance(clause, str) or clause != "WHERE":
+ raise ValueError(
+ "Sunburst native SIMPLE filter clause must be 'WHERE'; "
+ "SIMPLE HAVING is unsupported because the shared query mapper "
+ "cannot preserve HAVING semantics"
+ )
+ if operator == "TEMPORAL_RANGE":
+ if "temporal_column" not in data and "granularity_sqla" not in data:
+ data["temporal_column"] = native_filter.get("subject")
+ data.setdefault("time_range", native_filter.get("comparator"))
+ return None
+ mapped_operator = "=" if operator == "==" else operator
+ comparator = native_filter.get("comparator")
+ if mapped_operator in {"IS NULL", "IS NOT NULL"}:
+ comparator = None
+ return {
+ "column": native_filter.get("subject"),
+ "clause": clause,
+ "op": mapped_operator,
+ "value": comparator,
+ }
+
+ @staticmethod
+ def _discard_legacy_saved_fields(data: dict[str, Any]) -> None:
+ """Validate and ignore only the known obsolete saved-chart controls."""
+ legacy = {
+ key: data[key] for key in _SUNBURST_IGNORED_LEGACY_FIELDS if key in data
+ }
+ if legacy:
+ SunburstLegacySavedFields.model_validate(legacy)
+ for key in legacy:
+ data.pop(key, None)
+
+ @classmethod
+ def _coerce_native_filters(cls, data: dict[str, Any]) -> None:
+ """Round-trip native saved filters into the typed filter contract."""
+ native_filters = data.pop("adhoc_filters", None)
+ if native_filters is None or "filters" in data:
+ return
+ if not isinstance(native_filters, list):
+ raise ValueError("Sunburst native adhoc_filters must be a list")
+ filters = [
+ converted
+ for native_filter in native_filters or []
+ if (converted := cls._coerce_native_filter(native_filter, data)) is not None
+ ]
+ data["filters"] = filters
+
+ @model_validator(mode="before")
+ @classmethod
+ def accept_native_form_data(cls, data: Any) -> Any:
+ """Translate native saved Sunburst form_data into the typed contract."""
+ if not isinstance(data, dict):
+ return data
+ original_data = data if type(data) is dict else None
+ data = dict(data)
+ if _native_metric_paths_require_safe_rejection(data):
+ data = _sanitize_rejected_native_metric_config(data, original_data)
+ is_native_form_data = cls._looks_like_native_form_data(data)
+ data.pop(_NATIVE_FORM_DATA_MARKER, None)
Review Comment:
Fixed in 6cf22d8bb1. The Sunburst native adapter now consumes its own generated `_mcp_dashboard_time_filter_subject` marker. Like the Gantt adapter, it requires a non-empty column name, so mapper output can be reused as a config. The marker key constant now lives in `schemas.py`, and `chart_utils` imports it from there, so the two cannot drift. Tests: `test_generated_temporal_binding_marker_round_trips_as_native_config` (uses real mapper output with the marker) and `test_native_time_binding_marker_must_name_a_column`.
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1416 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
Review Comment:
Fixed in 29c3c589bc. Validated temporal cells stay canonical ISO text for the JSON projection. The Excel export now gets the temporal columns from the result's `coltypes` and writes those cells back as `date`/`datetime`/`time` values. Offsets are dropped, keeping wall-clock time, as Superset's own Excel export does. This covers openpyxl and the xlsxwriter fallback, which gets explicit date formats. A SQL DATE is a date cell again, not General text. Tests: `test_excel_export_keeps_sql_dates_as_excel_dates` (end to end through `get_chart_data`, format=excel) and `test_excel_export_writes_validated_temporal_cells_as_dates` (both writers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4190029086
##########
superset/mcp_service/chart/chart_helpers.py:
##########
@@ -853,55 +1206,28 @@ def build_query_dicts_from_form_data(
if plugin_queries is not None:
return plugin_queries
- metrics, groupby = resolve_metrics_and_groupby(form_data, chart)
-
# Deck.gl charts use spatial column configs rather than the standard
# metrics / groupby fields. Extract columns from the spatial controls.
if viz_type.startswith("deck_"):
- deck_columns = resolve_deck_gl_columns(form_data)
- deck_metrics = _resolve_deck_gl_metrics(form_data, viz_type)
- qd = build_single_query_dict(
- form_data,
- deck_columns,
- deck_metrics,
- row_limit=row_limit,
- order_desc=order_desc,
- )
- if deck_metrics:
- # Mirror BaseDeckGLViz.query_obj(): order by first metric descending
- qd["orderby"] = [(deck_metrics[0], not form_data.get("order_desc", True))]
- if viz_type in _DECK_TIMESERIES_VIZ_TYPES and (
- time_grain := form_data.get("time_grain_sqla")
- ):
- qd["is_timeseries"] = True
- qd["granularity"] = form_data.get("granularity_sqla")
- qd.setdefault("extras", {})["time_grain_sqla"] = time_grain
- if form_data.get("filter_nulls", True):
- null_filters = _deck_gl_null_filters(form_data)
- if null_filters:
- qd["filters"] = [*(qd.get("filters") or []), *null_filters]
- return [qd]
-
- if viz_type.startswith("echarts_timeseries"):
- groupby = with_x_axis_column(form_data, groupby)
-
- return [
- build_single_query_dict(
+ from superset.common.form_data_query_context import normalize_time_column
+
+ qd = build_query_objects_from_form_data(
form_data,
- groupby,
- metrics,
+ viz_type=viz_type,
row_limit=row_limit,
order_desc=order_desc,
- )
- ]
-
-
-def with_x_axis_column(form_data: dict[str, Any], groupby: list[Any]) -> list[Any]:
- """Prepend a time-series chart's x-axis column to its query columns."""
- x_axis_col = extract_x_axis_col(form_data)
- if x_axis_col and x_axis_col not in groupby:
- return [x_axis_col, *groupby]
- return groupby
+ filters_prepared=True,
+ )[0]
+ qd = _deck_query_adapter(form_data, qd, viz_type)
+ return [normalize_time_column(form_data, qd)]
+ return build_query_objects_from_form_data(
Review Comment:
Fixed in 0d1196e052. `build_query_objects_from_form_data` now has a World/Country Map adapter that mirrors both plugins' `buildQuery`: `entity` is the only grouping column, Country Map selects `metric`, and World Map adds a distinct `secondary_metric` and orders by the metric only when `sort_by_metric` is set. A rebuild without `query_context` returns per-entity rows again. Regression test: `test_build_query_dicts_groups_entity_maps_by_entity`.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1773,741 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(existing_form_data)
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ if getattr(config, "filters", None) == []:
+ merged.pop("adhoc_filters", None)
+ elif getattr(config, "filters", None) is None:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool(
+ {"temporal_column", "time_grain", "time_range"} & fields_set
+ ),
+ )
+ if filters is not None:
+ merged["adhoc_filters"] = filters
+ return merged
+
+
+def retain_mixed_timeseries_secondary_update_state(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Return saved query-B controls a same-viz Mixed Timeseries update keeps.
+
+ Query B inherits unsuffixed controls only when the suffixed key is absent.
+ Preserve explicit native clears for controls the typed mapper did not
+ replace, so []/None never turns into accidental inheritance from query A.
+ Valid comparison state is also retained; malformed/stale dataset roles
+ remain fail-closed and are dropped.
+ """
+ from superset.common.form_data_query_context import (
+ MIXED_TIMESERIES_SECONDARY_QUERY_KEYS,
+ )
+
+ retained: Dict[str, Any] = {}
+ for key in MIXED_TIMESERIES_SECONDARY_QUERY_KEYS:
+ if key in new_form_data or key not in existing_form_data:
+ continue
+ value = existing_form_data[key]
+ is_explicit_clear = value is None or value in ([], {}, "")
+ is_valid_comparison = key == "comparison_type_b" and value in {
+ "values",
+ "difference",
+ "percentage",
+ "ratio",
+ }
+ is_valid_list_state = key in {
+ "adhoc_filters_b",
+ "annotation_layers_b",
+ "time_compare_b",
+ } and isinstance(value, list)
+ if is_explicit_clear or is_valid_comparison or is_valid_list_state:
+ retained[key] = value
+ return retained
+
+
+def finalize_sunburst_update_form_data( # noqa: C901
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ merged: Dict[str, Any],
+ config: SunburstChartConfig,
+) -> Dict[str, Any]:
+ """Apply Sunburst omission, explicit-clear and temporal-pair semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ temporal_fields = {"time_grain", "temporal_column"}
+ for field_name, form_key in _SUNBURST_UPDATE_FIELD_KEYS.items():
+ if field_name in temporal_fields:
+ continue
+ if field_name not in fields_set:
+ if same_viz:
+ if form_key in existing_form_data:
+ merged[form_key] = existing_form_data[form_key]
+ else:
+ merged.pop(form_key, None)
+ continue
+
+ value = getattr(config, field_name)
+ if value is None:
+ merged.pop(form_key, None)
+ if field_name == "time_range":
+ # The generic query-context mapper reconstructs time_range from
+ # these legacy keys. A clear must remove all three sources.
+ merged.pop("since", None)
+ merged.pop("until", None)
+
+ # A null temporal control is an atomic clear. Apply it after every merge so
+ # cached/native aliases and extra-form-data overrides cannot recreate time
+ # state in QueryContextFactory.
+ explicit_temporal_clear = any(
+ field_name in fields_set and getattr(config, field_name) is None
+ for field_name in ("temporal_column", "time_grain", "time_range")
+ )
+ if explicit_temporal_clear:
+ merged = _scrub_temporal_form_data(merged)
+
+ # Merge the temporal subject and grain as one final-state control pair.
+ temporal_set = "temporal_column" in fields_set
+ grain_set = "time_grain" in fields_set
+ if temporal_set and config.temporal_column is not None:
+ merged["granularity_sqla"] = config.temporal_column
+
+ if grain_set and config.time_grain is not None:
+ merged["time_grain_sqla"] = config.time_grain
+
+ if "time_range" in fields_set and config.time_range is not None:
+ merged["time_range"] = config.time_range
+
+ for key in ("extra_form_data", "standardizedFormData", "url_params"):
+ if merged.get(key) is None:
+ merged.pop(key, None)
+
+ # Do not discard an orphan grain here. The final-form-data validator must
+ # see and reject it consistently across immediate, preview-first, and
+ # cached update paths instead of letting the query builder silently ignore
+ # a request that cannot be represented by the frontend contract.
+ return merged
+
+
+_DATASET_BOUND_FORM_DATA_KEYS = frozenset(
Review Comment:
Fixed in 6fe5fcbb8b. `where` and `having` are now in `_DATASET_BOUND_FORM_DATA_KEYS`, so a dataset rebind scrubs them along with `adhoc_filters`/`filters`. `prepare_form_data_for_query` can no longer rebuild them into the new dataset's query. Regression test: `test_cached_table_rebind_drops_legacy_sql_predicates`.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1773,741 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
Review Comment:
Fixed in 6fe5fcbb8b. `scrub_dataset_bound_form_data` takes a `target_viz_type`. When it differs from the source viz, the replacement starts from the mapped target config and keeps only the shared preservation-registry keys that are not dataset-bound. No source role contract is needed, because the cross-viz merge never inherits source query roles. Both `merge_form_data_for_update` and the cached `update_chart_preview` path pass it. A dataset-only rebind of an unregistered viz still fails closed. Tests: `test_cross_viz_rebind_from_unregistered_viz_uses_mapped_target`, `test_cached_cross_viz_rebind_from_unregistered_viz_uses_target_config`.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1773,741 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(existing_form_data)
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
Review Comment:
Fixed in 6fe5fcbb8b. An explicit `filters=[]` now pops every saved predicate source: `adhoc_filters`, `filters`, `extra_filters`, `where` and `having`. The legacy keys can no longer survive the allowlisted merge and be rebuilt at query time. Regression test: `test_explicit_empty_filters_clear_legacy_predicates_on_cross_viz_update`, which also builds the query and asserts no EMEA predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3923271152
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -419,15 +447,16 @@ async def get_chart_data( # noqa: C901
)
try:
cached_form_data_dict = utils_json.loads(cached_form_data)
- except (TypeError, ValueError) as e:
+ except (TypeError, ValueError) as exc:
+ if type(exc) not in {TypeError, ValueError}:
+ raise
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Exact-type guard breaks parse fallback</b></div>
<div id="fix">
`utils_json.loads` delegates to `simplejson.loads`, which raises `JSONDecodeError` (a `ValueError` subclass) on invalid JSON. The exact-type guard `type(exc) not in {TypeError, ValueError}` re-raises it, so the graceful fallback/`ParseError` path never runs. Use `isinstance(exc, (TypeError, ValueError))` instead. Same pattern at lines 558-560, 603-605, 686-688.
</div>
</div>
<small><i>Code Review Run #bb24a2</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3915661024
##########
superset/mcp_service/utils/response_utils.py:
##########
@@ -167,53 +171,208 @@ def build(self) -> Dict[str, str]:
STATS_ROW_CAP: int = 5000
+STATS_SAMPLE_VALUE_COUNT: int = 3
+STATS_TOTAL_WORK_CAP: int = 100_000
+
+_GENERIC_DATA_TYPE_NAMES: dict[int, str] = {
+ GenericDataType.NUMERIC: "numeric",
+ GenericDataType.STRING: "string",
+ GenericDataType.TEMPORAL: "temporal",
+ GenericDataType.BOOLEAN: "boolean",
+}
+_MAX_PROFILE_INTEGER_BITS = 4_096
+_MAX_PROFILE_STRING_LENGTH = 65_536
+
+
+@dataclass
+class _ColumnStatsBudget:
+ """One nested-node budget shared by all result columns."""
+
+ nodes: int = 0
+
+
+def data_column_stats_row_limit(row_count: int, column_count: int) -> int:
+ """Return a row sample whose aggregate top-level cell work is bounded."""
+ if row_count <= 0 or column_count <= 0:
+ return 0
+ return min(row_count, STATS_ROW_CAP, STATS_TOTAL_WORK_CAP // column_count)
+
+
+def _profile_value_identity( # noqa: C901
+ value: Any, budget: _ColumnStatsBudget
+) -> tuple[Any, ...] | None:
+ """Build a hook-free identity under the shared iterative node budget."""
+ tokens: list[Any] = []
+ stack: list[tuple[str, Any]] = [("value", value)]
+ active_containers: set[int] = set()
+ while stack:
+ action, item = stack.pop()
+ budget.nodes += 1
+ if budget.nodes > STATS_TOTAL_WORK_CAP:
+ return None
+ if action == "token":
+ tokens.append(item)
+ continue
+ if action == "leave":
+ active_containers.remove(id(item))
+ continue
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ tokens.append(("cyclic_list",))
+ continue
+ active_containers.add(identity)
+ width = list.__len__(item)
+ tokens.append(("list", width))
+ stack.append(("leave", item))
+ stack.append(("token", "list_end"))
+ stack.extend(
+ ("value", list.__getitem__(item, index))
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ tokens.append(("cyclic_dict",))
+ continue
+ active_containers.add(identity)
+ entries = list(dict.items(item))
+ tokens.append(("dict", list.__len__(entries)))
+ stack.append(("leave", item))
+ stack.append(("token", "dict_end"))
+ for key, child in reversed(entries):
+ key_token = (
+ ("key", key)
+ if type(key) is str
+ and str.__len__(key) <= _MAX_PROFILE_STRING_LENGTH
+ else ("opaque_key", id(type(key)), id(key))
+ )
+ stack.append(("value", child))
+ stack.append(("token", key_token))
+ continue
+
+ value_type = type(item)
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>bool/int unique_count collision</b></div>
<div id="fix">
`_profile_value_identity` tokenizes bool as `("number", int(item), 1)`, which collides with the int token `("number", item, 1)`. For a column mixing booleans and integers (e.g. `[True, 1, False, 0]`), `unique_count` drops from 4 (old `str(val)` behavior) to 2. Use a distinct `"boolean"` tag to preserve type distinction.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
````suggestion
elif value_type is bool:
tokens.append(("boolean", item))
elif value_type is int:
````
</div>
</details>
</div>
<small><i>Code Review Run #4d446c</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -167,6 +175,110 @@ def _rejected_requested_filter_columns(
}
_MAX_RECOMMENDATIONS = 4
+_MAX_COLUMN_PROFILE_CELLS = 100_000
+
+
+@dataclass
+class _ProfileBudget:
+ """One iterative node budget shared by every profiled column."""
+
+ nodes: int = 0
+
+
+def _canonical_profile_key(
+ value: Any, budget: _ProfileBudget
+) -> tuple[Any, ...] | None:
+ """Build a bounded identity from canonical JSON primitives, iteratively."""
+ tokens: list[Any] = []
+ stack: list[tuple[str, Any]] = [("value", value)]
+ while stack:
+ action, item = stack.pop()
+ budget.nodes += 1
+ if budget.nodes > _MAX_COLUMN_PROFILE_CELLS:
+ return None
+ if action == "end":
+ tokens.append(item)
+ continue
+ if type(item) is list:
+ tokens.append(("list", len(item)))
+ stack.append(("end", "list_end"))
+ stack.extend(
+ ("value", list.__getitem__(item, index))
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if type(item) is dict:
+ entries = sorted(dict.items(item))
+ tokens.append(("dict", len(entries)))
+ stack.append(("end", "dict_end"))
+ for key, child in reversed(entries):
+ stack.append(("value", child))
+ stack.append(("end", ("key", key)))
+ continue
+ value_type = type(item)
+ if value_type not in {type(None), bool, int, float, str}: # noqa: E721
+ raise TypeError("query result value was not canonicalized")
+ tokens.append((value_type.__name__, item))
+ return tuple(tokens)
+
+
+def _build_data_columns( # noqa: C901
+ data: list[dict[str, Any]],
+ raw_columns: list[str],
+ coltypes: list[Any] | None = None,
+) -> list[DataColumn]:
+ """Profile canonical rows under one nested-node budget."""
+ column_count = max(len(raw_columns), 1)
+ profile_row_limit = max(3, _MAX_COLUMN_PROFILE_CELLS // column_count)
+ budget = _ProfileBudget()
+ profiled_values: dict[str, list[Any]] = {column: [] for column in raw_columns}
+ profiled_keys: dict[str, set[tuple[Any, ...]]] = {
+ column: set() for column in raw_columns
+ }
+ sampled_rows = 0
+ for row_offset in range(min(len(data), profile_row_limit)):
+ row = list.__getitem__(data, row_offset)
+ row_values: list[tuple[str, Any, tuple[Any, ...]]] = []
+ for col_name in raw_columns:
+ value = dict.get(row, col_name)
+ identity = _canonical_profile_key(value, budget)
+ if identity is None:
+ break
+ row_values.append((col_name, value, identity))
+ if len(row_values) != len(raw_columns):
+ break
+ for col_name, value, identity in row_values:
+ profiled_values[col_name].append(value)
+ profiled_keys[col_name].add(identity)
+ sampled_rows += 1
+
+ columns: list[DataColumn] = []
+ for idx, col_name in enumerate(raw_columns):
+ values = profiled_values[col_name]
+ sample_values = [value for value in values[:3] if value is not None]
+ data_type = "string"
+ if coltypes:
+ data_type = _GENERIC_TYPE_MAP.get(coltypes[idx], "string")
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Decimal type detection removed</b></div>
<div id="fix">
Removed `Decimal` from numeric type detection at line 261. Original code checked for `{int, float, Decimal}`; now only checks `{int, float}`. This may cause `Decimal` values to be misclassified as "string" instead of "numeric".
</div>
</div>
<small><i>Code Review Run #4d446c</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5690126469
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5505583059
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5508371831
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3944693734
##########
superset/mcp_service/chart/sunburst.py:
##########
@@ -0,0 +1,329 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Shared native Sunburst result-role resolution and validation."""
+
+import math
+from collections.abc import Mapping
+from dataclasses import dataclass
+from decimal import Decimal
+from typing import Any
+
+from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.common.error_schemas import DatasetContext
+
+
+@dataclass(frozen=True)
+class SunburstResultRoles:
+ """Resolved query-output fields required to render a Sunburst."""
+
+ hierarchy: tuple[str, ...]
+ primary_metric: str
+ secondary_metric: str | None = None
+
+
+def _column_result_label(column: Any) -> str | None:
+ """Resolve a bounded native column reference to its query output label."""
+ if isinstance(column, str) and column:
+ return column
+ if not isinstance(column, Mapping) or not 0 < len(column) <= 20:
+ return None
+ for key in ("label", "column_name", "columnName", "sqlExpression"):
+ value = column.get(key)
+ if isinstance(value, str) and value:
+ return value
+ if value not in (None, ""):
+ return None
+ return None
+
+
+def _metric_result_label(metric: Any) -> str | None:
+ """Resolve saved, SIMPLE, and SQL metric aliases like the frontend."""
+ if isinstance(metric, str) and metric:
+ return metric
+ if not isinstance(metric, Mapping) or not 0 < len(metric) <= 20:
+ return None
+
+ label = metric.get("label")
+ if isinstance(label, str) and label:
+ return label
+ if label not in (None, ""):
+ return None
+
+ expression_type = metric.get("expressionType")
+ if expression_type == "SIMPLE":
+ aggregate = metric.get("aggregate")
+ column = metric.get("column")
+ if (
+ not isinstance(aggregate, str)
+ or not aggregate
+ or len(aggregate) > 100
+ or not isinstance(column, Mapping)
+ or not 0 < len(column) <= 50
+ ):
+ return None
+ column_name = column.get("column_name") or column.get("columnName")
+ if not isinstance(column_name, str) or not column_name:
+ return None
+ return f"{aggregate}({column_name})"
+ if expression_type == "SQL":
+ sql_expression = metric.get("sqlExpression")
+ if (
+ isinstance(sql_expression, str)
+ and sql_expression
+ and len(sql_expression) <= 2000
+ ):
+ return sql_expression
+ return None
+
+
+def _metric_query_identity(metric: Any) -> tuple[str, ...] | None:
+ """Return a hook-free identity for one exact native metric value."""
+ if type(metric) is str and metric:
+ return ("saved", metric)
+ if type(metric) is not dict:
+ return None
+ expression_type = dict.get(metric, "expressionType")
+ if expression_type == "SQL":
+ expression = dict.get(metric, "sqlExpression")
+ return ("sql", expression) if type(expression) is str and expression else None
+ if expression_type == "SIMPLE":
+ aggregate = dict.get(metric, "aggregate")
+ column = dict.get(metric, "column")
+ if type(aggregate) is not str:
+ return None
+ column_name: Any
+ if type(column) is str:
+ column_name = column
+ elif type(column) is dict:
+ column_name = dict.get(column, "column_name")
+ if column_name is None:
+ column_name = dict.get(column, "columnName")
+ else:
+ return None
+ if type(column_name) is str and column_name:
+ return ("simple", aggregate, column_name)
+ return None
+
+
+def resolve_sunburst_result_roles(
+ form_data: Mapping[str, Any],
+) -> tuple[SunburstResultRoles | None, ChartError | None]:
+ """Resolve and validate all native Sunburst query-result roles."""
+ columns = form_data.get("columns")
+ if not isinstance(columns, list) or not columns:
+ return None, ChartError(
+ error="Sunburst form data requires one or more hierarchy columns.",
+ error_type="InvalidSunburstFormData",
+ )
+ hierarchy: list[str] = []
+ for index, column in enumerate(columns):
+ label = _column_result_label(column)
+ if label is None:
+ return None, ChartError(
+ error=f"Sunburst hierarchy column {index + 1} is malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ hierarchy.append(label)
+
+ primary = _metric_result_label(form_data.get("metric"))
+ if primary is None:
+ return None, ChartError(
+ error="Sunburst primary metric is missing or malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ primary_metric = form_data.get("metric")
+ secondary: str | None = None
+ if (secondary_metric := form_data.get("secondary_metric")) is not None:
+ secondary = _metric_result_label(secondary_metric)
+ if secondary is None:
+ return None, ChartError(
+ error="Sunburst secondary metric is malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ if (
+ secondary.casefold() == primary.casefold()
+ and _metric_query_identity(primary_metric)
+ == _metric_query_identity(secondary_metric)
+ and _metric_query_identity(primary_metric) is not None
+ ):
+ # The frontend interprets a repeated primary metric as categorical
+ # color mode and the query has one physical metric output.
+ secondary = None
+
+ labels = [*hierarchy, primary, *([secondary] if secondary else [])]
+ seen: dict[str, str] = {}
+ for label in labels:
+ folded = label.casefold()
+ if previous := seen.get(folded):
+ return None, ChartError(
+ error=(
+ f"Sunburst result label {label!r} is ambiguous with {previous!r}."
+ ),
+ error_type="InvalidSunburstFormData",
+ )
+ seen[folded] = label
+
+ return SunburstResultRoles(tuple(hierarchy), primary, secondary), None
+
+
+def _finite_number(value: Any) -> bool:
+ """Return whether a normalized database value is exactly numeric and finite."""
+ value_type = type(value)
+ if value_type is int:
+ return True
+ if value_type is float:
+ return math.isfinite(value)
+ if value_type is Decimal:
+ return Decimal.is_finite(value)
+ return False
+
+
+def _valid_hierarchy_value(value: Any) -> bool:
+ """Reject nested/container values that cannot form a stable node label."""
+ return not isinstance(value, (Mapping, list, tuple, set))
+
+
+def validate_sunburst_result_data(
+ data: Any, form_data: Mapping[str, Any]
+) -> tuple[SunburstResultRoles | None, ChartError | None]:
+ """Validate every Sunburst row and all resolved result aliases."""
+ roles, error = resolve_sunburst_result_roles(form_data)
+ if error is not None:
+ return None, error
+ assert roles is not None
+ if not isinstance(data, list):
+ return None, ChartError(
+ error="Sunburst query result data is not an array of rows.",
+ error_type="InvalidSunburstResult",
+ )
+
+ required_fields = [
+ *roles.hierarchy,
+ roles.primary_metric,
+ *([roles.secondary_metric] if roles.secondary_metric else []),
+ ]
+ for index, row in enumerate(data, start=1):
+ if not isinstance(row, Mapping):
+ return None, ChartError(
+ error=f"Sunburst result row {index} is not an object.",
+ error_type="InvalidSunburstResult",
+ )
+ missing = [field for field in required_fields if field not in row]
+ if missing:
+ return None, ChartError(
+ error=(
+ f"Sunburst result row {index} is missing required field(s): "
+ f"{', '.join(str(field) for field in missing)}."
+ ),
+ error_type="InvalidSunburstResult",
+ )
+ for field in roles.hierarchy:
+ if not _valid_hierarchy_value(row[field]):
+ return None, ChartError(
+ error=(
+ f"Sunburst result row {index} has a malformed hierarchy "
+ f"value for {field!r}."
+ ),
+ error_type="InvalidSunburstResult",
+ )
+ for field in (
+ roles.primary_metric,
+ *([roles.secondary_metric] if roles.secondary_metric else []),
+ ):
+ assert field is not None
+ if not _finite_number(row[field]):
Review Comment:
Addressed in bc8ac02f9a0dd3da8bdc3909079de1a99459efad. After generic producer normalization, the shared Sunburst validator converts NULL metric cells to zero, matching `Sunburst/transformProps.ts` โ `utils/treeBuilder.ts:getMetricValue`. This applies before compile/preview/data/export consumers; absent result columns, strings, booleans, and infinity remain invalid. NULL hierarchy cells are not changed.
Passing regressions: `test_compile_proves_numeric_metric_by_resolved_alias` covers real producer NULL/NaN output with saved/SIMPLE/SQL aliases; `test_saved_preview_is_sunburst_faithful_and_vega_is_unsupported` covers NULL saved previews; `test_null_sunburst_metrics_normalize_without_changing_null_hierarchy` covers either metric role; `test_decimal_sunburst_get_data_is_numeric_and_json_safe` exercises the real DataFrame/QueryContextProcessor/ChartDataCommand producer through registered FastMCP saved, unsaved, and cached-update JSON/CSV calls. Focused suite: **776 passed**. Final full-suite and exact-head CI verification are underway; no new review requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3944693734
##########
superset/mcp_service/chart/sunburst.py:
##########
@@ -0,0 +1,329 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Shared native Sunburst result-role resolution and validation."""
+
+import math
+from collections.abc import Mapping
+from dataclasses import dataclass
+from decimal import Decimal
+from typing import Any
+
+from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.common.error_schemas import DatasetContext
+
+
+@dataclass(frozen=True)
+class SunburstResultRoles:
+ """Resolved query-output fields required to render a Sunburst."""
+
+ hierarchy: tuple[str, ...]
+ primary_metric: str
+ secondary_metric: str | None = None
+
+
+def _column_result_label(column: Any) -> str | None:
+ """Resolve a bounded native column reference to its query output label."""
+ if isinstance(column, str) and column:
+ return column
+ if not isinstance(column, Mapping) or not 0 < len(column) <= 20:
+ return None
+ for key in ("label", "column_name", "columnName", "sqlExpression"):
+ value = column.get(key)
+ if isinstance(value, str) and value:
+ return value
+ if value not in (None, ""):
+ return None
+ return None
+
+
+def _metric_result_label(metric: Any) -> str | None:
+ """Resolve saved, SIMPLE, and SQL metric aliases like the frontend."""
+ if isinstance(metric, str) and metric:
+ return metric
+ if not isinstance(metric, Mapping) or not 0 < len(metric) <= 20:
+ return None
+
+ label = metric.get("label")
+ if isinstance(label, str) and label:
+ return label
+ if label not in (None, ""):
+ return None
+
+ expression_type = metric.get("expressionType")
+ if expression_type == "SIMPLE":
+ aggregate = metric.get("aggregate")
+ column = metric.get("column")
+ if (
+ not isinstance(aggregate, str)
+ or not aggregate
+ or len(aggregate) > 100
+ or not isinstance(column, Mapping)
+ or not 0 < len(column) <= 50
+ ):
+ return None
+ column_name = column.get("column_name") or column.get("columnName")
+ if not isinstance(column_name, str) or not column_name:
+ return None
+ return f"{aggregate}({column_name})"
+ if expression_type == "SQL":
+ sql_expression = metric.get("sqlExpression")
+ if (
+ isinstance(sql_expression, str)
+ and sql_expression
+ and len(sql_expression) <= 2000
+ ):
+ return sql_expression
+ return None
+
+
+def _metric_query_identity(metric: Any) -> tuple[str, ...] | None:
+ """Return a hook-free identity for one exact native metric value."""
+ if type(metric) is str and metric:
+ return ("saved", metric)
+ if type(metric) is not dict:
+ return None
+ expression_type = dict.get(metric, "expressionType")
+ if expression_type == "SQL":
+ expression = dict.get(metric, "sqlExpression")
+ return ("sql", expression) if type(expression) is str and expression else None
+ if expression_type == "SIMPLE":
+ aggregate = dict.get(metric, "aggregate")
+ column = dict.get(metric, "column")
+ if type(aggregate) is not str:
+ return None
+ column_name: Any
+ if type(column) is str:
+ column_name = column
+ elif type(column) is dict:
+ column_name = dict.get(column, "column_name")
+ if column_name is None:
+ column_name = dict.get(column, "columnName")
+ else:
+ return None
+ if type(column_name) is str and column_name:
+ return ("simple", aggregate, column_name)
+ return None
+
+
+def resolve_sunburst_result_roles(
+ form_data: Mapping[str, Any],
+) -> tuple[SunburstResultRoles | None, ChartError | None]:
+ """Resolve and validate all native Sunburst query-result roles."""
+ columns = form_data.get("columns")
+ if not isinstance(columns, list) or not columns:
+ return None, ChartError(
+ error="Sunburst form data requires one or more hierarchy columns.",
+ error_type="InvalidSunburstFormData",
+ )
+ hierarchy: list[str] = []
+ for index, column in enumerate(columns):
+ label = _column_result_label(column)
+ if label is None:
+ return None, ChartError(
+ error=f"Sunburst hierarchy column {index + 1} is malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ hierarchy.append(label)
+
+ primary = _metric_result_label(form_data.get("metric"))
+ if primary is None:
+ return None, ChartError(
+ error="Sunburst primary metric is missing or malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ primary_metric = form_data.get("metric")
+ secondary: str | None = None
+ if (secondary_metric := form_data.get("secondary_metric")) is not None:
+ secondary = _metric_result_label(secondary_metric)
+ if secondary is None:
+ return None, ChartError(
+ error="Sunburst secondary metric is malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ if (
+ secondary.casefold() == primary.casefold()
+ and _metric_query_identity(primary_metric)
+ == _metric_query_identity(secondary_metric)
+ and _metric_query_identity(primary_metric) is not None
+ ):
+ # The frontend interprets a repeated primary metric as categorical
+ # color mode and the query has one physical metric output.
+ secondary = None
+
+ labels = [*hierarchy, primary, *([secondary] if secondary else [])]
+ seen: dict[str, str] = {}
+ for label in labels:
+ folded = label.casefold()
+ if previous := seen.get(folded):
+ return None, ChartError(
+ error=(
+ f"Sunburst result label {label!r} is ambiguous with {previous!r}."
+ ),
+ error_type="InvalidSunburstFormData",
+ )
+ seen[folded] = label
+
+ return SunburstResultRoles(tuple(hierarchy), primary, secondary), None
+
+
+def _finite_number(value: Any) -> bool:
+ """Return whether a normalized database value is exactly numeric and finite."""
+ value_type = type(value)
+ if value_type is int:
+ return True
+ if value_type is float:
+ return math.isfinite(value)
+ if value_type is Decimal:
+ return Decimal.is_finite(value)
+ return False
+
+
+def _valid_hierarchy_value(value: Any) -> bool:
+ """Reject nested/container values that cannot form a stable node label."""
+ return not isinstance(value, (Mapping, list, tuple, set))
+
+
+def validate_sunburst_result_data(
+ data: Any, form_data: Mapping[str, Any]
+) -> tuple[SunburstResultRoles | None, ChartError | None]:
+ """Validate every Sunburst row and all resolved result aliases."""
+ roles, error = resolve_sunburst_result_roles(form_data)
+ if error is not None:
+ return None, error
+ assert roles is not None
+ if not isinstance(data, list):
+ return None, ChartError(
+ error="Sunburst query result data is not an array of rows.",
+ error_type="InvalidSunburstResult",
+ )
+
+ required_fields = [
+ *roles.hierarchy,
+ roles.primary_metric,
+ *([roles.secondary_metric] if roles.secondary_metric else []),
+ ]
+ for index, row in enumerate(data, start=1):
+ if not isinstance(row, Mapping):
+ return None, ChartError(
+ error=f"Sunburst result row {index} is not an object.",
+ error_type="InvalidSunburstResult",
+ )
+ missing = [field for field in required_fields if field not in row]
+ if missing:
+ return None, ChartError(
+ error=(
+ f"Sunburst result row {index} is missing required field(s): "
+ f"{', '.join(str(field) for field in missing)}."
+ ),
+ error_type="InvalidSunburstResult",
+ )
+ for field in roles.hierarchy:
+ if not _valid_hierarchy_value(row[field]):
+ return None, ChartError(
+ error=(
+ f"Sunburst result row {index} has a malformed hierarchy "
+ f"value for {field!r}."
+ ),
+ error_type="InvalidSunburstResult",
+ )
+ for field in (
+ roles.primary_metric,
+ *([roles.secondary_metric] if roles.secondary_metric else []),
+ ):
+ assert field is not None
+ if not _finite_number(row[field]):
Review Comment:
Addressed in bc8ac02f9a0dd3da8bdc3909079de1a99459efad. After generic producer normalization, the shared Sunburst validator converts NULL metric cells to zero, matching `Sunburst/transformProps.ts` โ `utils/treeBuilder.ts:getMetricValue`. This applies before compile/preview/data/export consumers; absent result columns, strings, booleans, and infinity remain invalid. NULL hierarchy cells are not changed.
Passing regressions: `test_compile_proves_numeric_metric_by_resolved_alias` covers real producer NULL/NaN output with saved/SIMPLE/SQL aliases; `test_saved_preview_is_sunburst_faithful_and_vega_is_unsupported` covers NULL saved previews; `test_null_sunburst_metrics_normalize_without_changing_null_hierarchy` covers either metric role; `test_decimal_sunburst_get_data_is_numeric_and_json_safe` exercises the real DataFrame/QueryContextProcessor/ChartDataCommand producer through registered FastMCP saved, unsaved, and cached-update JSON/CSV calls. Focused suite: **776 passed**. Final exact-head verification: **4,940 full affected tests passed**, all **13 required CI checks passed**. The CI main Python suite passed 15,454 tests. No new review requested.
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -0,0 +1,1520 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+
+import math
+import time as system_time
+from bisect import bisect_right
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
+from decimal import Decimal
+from enum import Enum
+from typing import Any
+from uuid import UUID
+from zoneinfo import ZoneInfo
+
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
+
+FAILED_QUERY_STATUSES = frozenset(
+ {"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
+)
+
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
+ return None
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
+
+
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = list.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (list.__getitem__(item, index), item, index, depth + 1, False)
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ children: list[tuple[Any, dict[str, Any], str, int, bool]] = []
+ for key, child in dict.items(item):
+ if type(key) is not str:
+ return None, "a non-string object key"
+ if reason := _charge_text(key, budget, key=True, metadata=metadata):
+ return None, reason
+ children.append((child, item, key, depth + 1, False))
+ stack.append((item, None, None, depth, True))
+ stack.extend(reversed(children))
+ continue
+
+ source_item = item
+ if type(item) in enum_types:
+ normalized = _safe_enum_value(item, enum_types)
+ if normalized is None:
+ return None, "an unsupported enum"
+ item = normalized
+ elif any(base is Enum for base in _type_mro(type(item))):
+ return None, "an enum outside its expected metadata slot"
+
+ normalized, reason = _normalize_scalar(item)
+ if reason is not None:
+ return None, reason
+ max_string_bytes = (
+ MAX_QUERY_RESULT_METADATA_BYTES if metadata else MAX_RESULT_STRING_LENGTH
+ )
+ if type(normalized) is str:
+ scalar_size = _json_string_size(normalized, max_string_bytes)
+ if scalar_size is None:
+ return None, "invalid text data"
+ else:
+ scalar_size = _normalized_scalar_json_size(normalized)
+ if reason := _charge_json_bytes(
+ budget,
+ scalar_size,
+ metadata=metadata,
+ ):
+ return None, reason
+ if parent is None:
+ root = normalized
+ elif normalized is not source_item:
+ if type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized)
+ return root, None
+
+
+def _normalize_metadata_value(
+ payload: dict[str, Any], key: str, budget: _ResultBudget
+) -> str | None:
+ enum_slots: dict[str, frozenset[type[Any]]] = {
+ "status": frozenset({QueryStatus}),
+ "result_format": frozenset({ChartDataResultFormat}),
+ "coltypes": frozenset({GenericDataType}),
+ "applied_filters": frozenset({ExtraFiltersTimeColumnType}),
+ "rejected_filters": frozenset(
+ {ExtraFiltersReasonType, ExtraFiltersTimeColumnType}
+ ),
+ }
+ value = dict.__getitem__(payload, key)
+ normalized, reason = _normalize_value(
+ value,
+ budget,
+ enum_types=enum_slots.get(key, frozenset()),
+ metadata=True,
+ )
+ if reason is None and normalized is not value:
+ dict.__setitem__(payload, key, normalized)
+ return reason
+
+
+def _error_text(value: Any) -> str | None:
+ """Extract a bounded error from an already validated primitive tree."""
+ stack: list[Any] = [value]
+ parts: list[str] = []
+ used = 0
+ while stack and len(parts) < 3 and used < _MAX_ERROR_TEXT_BYTES:
+ item = stack.pop()
+ if type(item) is dict:
+ stack.extend(
+ reversed(
+ [dict.__getitem__(item, key) for key in _ERROR_KEYS if key in item]
+ )
+ )
+ continue
+ if type(item) is list:
+ stack.extend(
+ list.__getitem__(item, index)
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if item is None or item is False:
+ continue
+ if type(item) is str:
+ remaining = _MAX_ERROR_TEXT_BYTES - used - (2 if parts else 0)
+ encoded = str.encode(item, "utf-8")[:remaining]
+ text = bytes.decode(encoded, "utf-8", errors="ignore")
+ if text:
+ parts.append(text)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ return "; ".join(parts) or None
+
+
+def _failure_for_payload(payload: dict[str, Any], label: str) -> ChartError | None:
+ for key in ("error", "errors", "error_message"):
+ if key in payload and (message := _error_text(dict.__getitem__(payload, key))):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+
+ raw_status = dict.get(payload, "status")
+ status = raw_status if type(raw_status) is str else ""
+ normalized_status = status.strip().casefold().replace("-", "_").replace(" ", "_")
+ if normalized_status in FAILED_QUERY_STATUSES:
+ message = (
+ _error_text(dict.get(payload, "message"))
+ or _error_text(dict.get(payload, "error_message"))
+ or normalized_status
+ )
+ return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if dict.get(payload, "success") is False:
+ message = _error_text(dict.get(payload, "message")) or "request failed"
+ return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if raw_status is None and "data" not in payload and "queries" not in payload:
+ if message := _error_text(dict.get(payload, "message")):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+ return None
+
+
+def _metadata_shape_error( # noqa: C901
+ payload: dict[str, Any], label: str, budget: _ResultBudget
+) -> ChartError | None:
+ if "success" in payload and type(dict.__getitem__(payload, "success")) is not bool:
+ return _invalid_metadata(label)
+ if "status" in payload and type(dict.__getitem__(payload, "status")) is not str:
+ return _invalid_metadata(label)
+ if (
+ "result_format" in payload
+ and dict.__getitem__(payload, "result_format") not in _RESULT_FORMAT_VALUES
+ ):
+ return _invalid_metadata(label)
+ for count_key in ("rowcount", "sql_rowcount", "total_rows"):
+ if count_key in payload:
+ count = dict.__getitem__(payload, count_key)
+ if count is not None and not (
+ type(count) is int and 0 <= count <= MAX_QUERY_RESULT_ROWCOUNT
+ ):
+ return _invalid_metadata(label)
+ if "is_cached" in payload:
+ cached = dict.__getitem__(payload, "is_cached")
+ if cached is None:
+ if reason := _charge_json_bytes(
+ budget, 1, metadata=True
+ ): # ``false`` vs ``null``
+ return _invalid_result(reason)
+ dict.__setitem__(payload, "is_cached", False)
+ elif type(cached) is not bool:
+ return _invalid_metadata(label)
+ if "cache_timeout" in payload:
+ timeout = dict.__getitem__(payload, "cache_timeout")
+ if timeout is not None and not (
Review Comment:
Addressed in bc8ac02f9a0dd3da8bdc3909079de1a99459efad. The result-envelope validator imports and accepts Superset's exact integer `CACHE_DISABLED_TIMEOUT` sentinel, retaining the non-negative bounded range for ordinary timeouts. It preserves -1 without rewriting it and still rejects -2, boolean/float/string lookalikes, integer subclasses, and out-of-range values.
Passing regressions: `test_producer_cache_timeout_preserves_disabled_and_bounded_values`, `test_cache_timeout_rejects_noncanonical_sentinel_lookalikes`, plus producer-to-registered-FastMCP `test_decimal_sunburst_get_data_is_numeric_and_json_safe` (saved/unsaved/cached-update, JSON/CSV) and compile/saved-preview matrices at cache timeouts -1, 0, and 300. Focused suite: **776 passed**; staged and full branch-diff pre-commit/MyPy/Ruff/Pylint checks pass. Final exact-head verification: **4,940 full affected tests passed**, all **13 required CI checks passed**. The CI main Python suite passed 15,454 tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3989729693
##########
docs/docs/using-superset/using-ai-with-superset.mdx:
##########
@@ -341,3 +341,20 @@ Ask your admin for the MCP server URL and any authentication tokens you need.
- Verify the connection in your AI client (e.g., the hammer icon in Claude Desktop)
- Ask the AI "What Superset tools are available?" to confirm the connection
- Restart your AI client if you recently changed the configuration
+
+
+### Sunburst chart data
+
+MCP chart tools support ordered Sunburst hierarchies with a primary metric and
+an optional secondary color metric. Use `chart_type: "sunburst"` in typed requests;
Review Comment:
Addressed in fe5d57b096. The public Sunburst docs and schema now state that the secondary-to-primary ratio drives the sequential color scale (for example, profit/revenue represents margin), while omitting `secondary_metric` uses categorical arc colors. I aligned the MCP capability text, generated schema description, plugin validation detail, chart-config guidance, and `generate_chart` tool description, and added `test_secondary_metric_schema_explains_ratio_color_semantics` to lock the ratio/sequential/categorical wording. Focused Sunburst + Gauge tests pass: 458 passed, 1 skipped; staged-file pre-commit also passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5805227505
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4193070679
##########
superset/mcp_service/chart/plugins/table.py:
##########
@@ -44,6 +46,12 @@ class TableChartPlugin(BaseChartPlugin):
}
supports_column_append = True
+ def prepare_query_form_data(self, form_data: dict[str, Any]) -> None:
+ """Resolve inherited offsets before the filter merge removes their source."""
+ inherited = (form_data.get("extra_form_data") or {}).get("time_compare")
+ if inherited and inherited not in as_list(form_data.get("time_compare") or []):
Review Comment:
Fixed in 1eaa25ec3d7d0d4a94a0d59f95b25676487ebaf3. The Table hook uses the shared _table_time_offsets resolver before filter preparation consumes extra_form_data, so custom is resolved before inherited-offset membership is checked. Your example retains both 1 year ago and 2 weeks ago; a nonmatching request-level offset still replaces the selection. test_table_inherited_custom_offset_preserves_other_comparisons checks time_offsets and comparison columns through build_query_dicts_from_form_data for Table/AG Grid, including saved-chart viz fallback and request override precedence. The matching cases failed before and pass after. Full MCP suite: 8,146 passed, 4 skipped; pre-commit passed on all branch-changed files.
##########
superset/mcp_service/chart/plugins/xy.py:
##########
@@ -53,6 +58,17 @@ class XYChartPlugin(BaseChartPlugin):
"echarts_area": "Area Chart",
"echarts_timeseries_scatter": "Scatter Plot",
}
+ query_role_keys = BaseChartPlugin.query_role_keys | {"x_axis"}
+
+ def vega_lite_preview(
Review Comment:
Fixed in 1eaa25ec3d7d0d4a94a0d59f95b25676487ebaf3. XY sets resizes_saved_preview=True, preserving the saved-preview width, height, and chart-name description framing. test_saved_xy_preview_honors_dimensions_and_description exercises VegaLitePreviewStrategy.generate for all four XY native types with default dimensions and width=800/height=200; it also checks all wide series remain present and unsaved previews retain responsive sizing. All eight cases failed before and pass after. Full MCP suite: 8,146 passed, 4 skipped; pre-commit passed on all branch-changed files.
##########
superset/mcp_service/chart/preview_utils.py:
##########
@@ -1452,33 +1671,35 @@ def _generate_vega_lite_preview_from_data( # noqa: C901
mark = viz_to_mark.get(viz_type, "bar")
# Basic Vega-Lite spec
- spec = {
+ preview_data = _bounded_vega_data(data)
+ spec: dict[str, Any] = {
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
- "data": {"values": data},
+ "data": {"values": preview_data},
"mark": mark,
}
# Get x_axis and metrics from form_data
- x_axis = form_data.get("x_axis")
+ axis = form_data.get("x_axis")
+ x_axis = get_column_name(axis) if axis else "__timestamp"
Review Comment:
Addressed in 1eaa25ec3d7d0d4a94a0d59f95b25676487ebaf3. test_grouped_timeseries_preview_resolves_legacy_and_sql_axes renders grouped wide rows through the plugin renderer for both a legacy chart with no x_axis and __timestamp output, and an SQL adhoc axis labelled Month. It asserts encoding.x.field and the complete series fold/y/color encodings. Temporarily reverting axis normalization made both cases fail (missing x encoding and unhashable-dict error); restoring it makes both pass. No renderer change was needed. Full MCP suite: 8,146 passed, 4 skipped; pre-commit passed on all branch-changed files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4202060799
##########
superset/common/form_data_query_context.py:
##########
@@ -23,30 +23,188 @@
carry no ``query_context``, so server-side consumers that need to run the query
(e.g. the dashboard Excel export) have nothing to execute.
-This module rebuilds a best-effort query context from the form data โ columns,
-metrics, filters (including free-form SQL and the time range), ordering and time
-grain โ mirroring the shared parts of the viz plugins' ``buildQuery``. Beyond
-that shared core it reproduces exactly one piece of plugin post-processing, Pie's
-unconditional ``contribution`` operator (see
-:func:`_pie_contribution_post_processing`). It does **not** reproduce any other
-post-processing (pivot, percent-metric transforms, rolling/forecast) or
-multi-query fan-out, so callers must restrict it to viz types whose data maps
-faithfully to a single query.
+This module rebuilds query contexts from form data โ columns, metrics, filters,
+ordering, temporal state, limits, query fan-out, and registered visualization
+post-processing โ mirroring the shared frontend extractor and explicit plugin
+``buildQuery`` adapters. MCP compile, SQL, preview, and data fallback products
+consume the same QueryObject contract as the legacy common fallback.
The mirrored logic lives on the frontend in
-``superset-frontend/plugins/plugin-chart-table/src/buildQuery.ts`` (query mode,
-ordering), ``superset-frontend/packages/superset-ui-core/src/query/`` (field
-extraction, ``processFilters``). There is no automated tripwire tying the two
-across the language boundary; the per-helper pointers below must be kept in sync
-when that frontend logic changes.
+``superset-frontend/plugins/*/buildQuery.ts`` and
+``superset-frontend/packages/superset-ui-core/src/query/`` (field extraction,
+``processFilters``). The adapter tests pin the Python QueryObject dictionaries;
+the per-helper pointers below must still be kept in sync when frontend contracts
+change.
"""
from __future__ import annotations
+from collections.abc import Mapping
from typing import Any
from superset.utils import json
-from superset.utils.core import as_list, get_metric_name
+from superset.utils.core import as_list, DTTM_ALIAS, get_column_name, get_metric_name
+
+# Keep this mapping identical to ``queryFieldAliases`` in
+# ``superset-ui-core/src/query/extractQueryFields.ts``. It is the shared
+# frontend contract for every form-data key that can contribute a metric,
+# column, or ordering expression to a QueryObject. Server-side consumers and
+# MCP replacement cleanup import this contract rather than maintaining partial
+# chart-specific copies.
+FORM_DATA_QUERY_FIELD_ALIASES: dict[str, str] = {
+ "metric": "metrics",
+ "metric_2": "metrics",
+ "secondary_metric": "metrics",
+ "left_metric": "metrics",
+ "right_metric": "metrics",
Review Comment:
Fixed in f2acaab6defc45cd7680662feced6b74dae95d02.
Added open, close, high, and low to the shared metric alias mapping, matching extractQueryFields.ts. Regression: test_candlestick_fallback_selects_all_ohlc_metrics verifies all four metrics reach the fallback QueryObject.
Validation: 851 tests passed across the 11 touched/related pytest modules; pre-commit passed for all changed files. The regression coverage fails against the previous implementation and passes with this fix.
##########
superset/mcp_service/chart/chart_helpers.py:
##########
@@ -815,6 +1152,14 @@ def build_mixed_timeseries_secondary(
)
+def with_x_axis_column(form_data: dict[str, Any], groupby: list[Any]) -> list[Any]:
Review Comment:
Fixed in f2acaab6defc45cd7680662feced6b74dae95d02.
Confirmed there were no remaining callers and removed with_x_axis_column, build_mixed_timeseries_secondary, _DECK_TIMESERIES_VIZ_TYPES, and _columns_and_metrics. Query construction continues through the shared adapter. Regression: test_unused_query_builders_are_removed guards against reintroducing the duplicate builders.
Validation: 851 tests passed across the 11 touched/related pytest modules; pre-commit passed for all changed files. The regression coverage fails against the previous implementation and passes with this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5865528602
Fixed the two Excel-export regressions in ba472d5bdb, merged without rewriting history at 8949a938c6.
Both tests reproduce on the previous PR head (34de8de3dd) and pass on master (ffe9a00fc4). This is a branch integration regression, not a regression in #44082: this branch's expanded query builder applies interactive table pagination during the Excel fallback rebuild. Aggregate mode adds a rowcount query; raw mode retains the saved page offset. Changing the top-level output to JSON/FULL happens after that rebuild and cannot undo it.
The workbook fallback now clears table pagination and resets the offset before query planning, preserving the full row limit for both queued exports and the synchronous budget. Four new tests cover aggregate/raw modes at and above EXCEL_EXPORT_SYNC_MAX_ROWS with EXPORT_STORAGE={}; all four fail before the fix. Saved/custom query contexts and the MCP inventory/schema budgets are unchanged.
Validation: 568 tests passed across Excel tasks, export helpers, common query-context building, and Sunburst; staged pre-commit passed, including mypy and pylint. CI is running again. The existing rusackas review request is retained; the separate tool-inventory size failures are not waived by this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4221572306
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1784,801 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Required query roles are absent:
+# a replacement config owns those through the plugin contract. Optional grouping
+# and ordering retain saved state unless their modeled fields are supplied.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "series_limit_metric": (("series_limit_metric",), ("timeseries_limit_metric",)),
+ "timeseries_limit_metric": (
+ ("series_limit_metric",),
+ ("timeseries_limit_metric",),
+ ),
+ "groupby": (("group_by",),),
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "groupby": (("group_by",),),
+ "groupby_b": (("group_by_secondary",),),
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ requested_filters = getattr(config, "filters", None)
+ if requested_filters is not None:
+ # Legacy ``filters``/``where``/``having`` are reconstructed into adhoc
+ # filters at query time, so an explicit clear or replacement removes
+ # every saved source the new form data does not set itself.
+ for key in _SAVED_PREDICATE_FORM_DATA_KEYS - new_form_data.keys():
+ merged.pop(key, None)
+ else:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool(
Review Comment:
Fixed in 0b37c7ce5caf763845a7b5ec9e28e966dbd17bc8.
`overlay_update_form_data` no longer drops saved TEMPORAL_RANGE filters when only time_grain is supplied. Explicit time_range or temporal_column changes retain their replacement semantics. `test_xy_grain_only_update_keeps_saved_temporal_range` covers P1M and an explicit grain clear, asserting both the preserved Last year filter and the resulting query predicate. Both cases fail against the previous head and pass with this fix.
Validation: touched modules 456 passed; MCP/common unit suites 9,228 passed, 4 skipped; pre-commit passed on changed files and all 83 branch-changed files, including mypy.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +427,1295 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are normalized for old server/native payloads.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Normalize saved physical references before the frontend column predicate."""
+ x_axis = _x_axis_column(form_data)
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
Review Comment:
Fixed in 0b37c7ce5caf763845a7b5ec9e28e966dbd17bc8.
Compile strips rolling_type, rolling_type_b, and forecastEnabled only from its deep-copied query form data. The bounded base-query sample therefore does not try to validate history-dependent analytics; saved controls and full preview/data pipelines remain unchanged. This limitation is documented.
`test_compile_sample_skips_rolling_and_forecast` executes the real post-processing operators on two rows. Against the previous head it raises Forecast requires at least 2 data points; with the fix compile succeeds with both rows. It also verifies the input is unchanged and full-query construction still includes rolling and prophet.
Validation: touched modules 456 passed; MCP/common unit suites 9,228 passed, 4 skipped; pre-commit passed on changed files and all 83 branch-changed files, including mypy.
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -1480,19 +1488,89 @@ def _write_excel_headers(ws: Any, columns: List[str]) -> None:
ws.cell(row=1, column=idx, value=col)
-def _write_excel_data(ws: Any, data: List[Dict[str, Any]], columns: List[str]) -> None:
+def _export_scalar(value: Any) -> Any:
+ """Project a validated result scalar to CSV/Excel writer-safe values."""
+ if value is None or (isinstance(value, float) and not math.isfinite(value)):
Review Comment:
Fixed the documentation mismatch in 0b37c7ce5caf763845a7b5ec9e28e966dbd17bc8; runtime normalization is intentionally unchanged.
The Gauge section explicitly states that non-finite floats become null before export, JSON returns null, and CSV/openpyxl/xlsxwriter write blank cells indistinguishable from SQL NULL. It warns that overflowing aggregates can appear as missing values and no longer promises nan/inf/-inf text.
`test_gauge_exports_and_docs_describe_nonfinite_as_blank` checks real producer normalization plus CSV and both XLSX writers for NULL, NaN, positive/negative infinity, and a finite value. All source groups remain present. Each case fails against the old documentation and passes with the correction.
Validation: touched modules 456 passed; MCP/common unit suites 9,228 passed, 4 skipped; pre-commit passed on changed files and all 83 branch-changed files, including mypy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4188278269
##########
superset/mcp_service/chart/compile.py:
##########
@@ -237,23 +281,47 @@ def _validate_adhoc_filter_columns(
# (column, clause) pairs: the clause decides whether a saved metric is a
# legal reference, and so whether metrics belong in the suggestions.
invalid: list[tuple[str, str]] = []
+ has_simple_having = False
for f in adhoc_filters:
# SIMPLE filters expose the column via "subject"; SQL-expression
# filters carry a free-form ``sqlExpression`` we can't safely parse,
# so skip those.
if f.get("expressionType") and f.get("expressionType") != "SIMPLE":
continue
+ clause = f.get("clause", "WHERE")
+ if not isinstance(clause, str) or clause not in {"WHERE", "HAVING"}:
+ return ChartGenerationError(
+ error_type="invalid_filter_clause",
+ message="SIMPLE filter clause must be 'WHERE' or 'HAVING'",
+ details=(
+ f"A SIMPLE filter has malformed clause {clause!r}; the clause "
+ "is never coerced or defaulted when explicitly set."
+ ),
+ suggestions=["Use clause='WHERE'"],
+ error_code="INVALID_FILTER_CLAUSE",
+ )
+ has_simple_having = has_simple_having or clause == "HAVING"
Review Comment:
P3: Deferring the SIMPLE HAVING rejection until after the column check keeps master's `test_metric_suggestions_follow_the_filter_clause` green, but on this branch its premise no longer holds. A HAVING filter with a near-miss subject (e.g. `sum_boy`) gets `invalid_column` with `sum_boys` suggested (`include_metrics=clause == "HAVING"`). The caller applies that fix and then hits `UNSUPPORTED_FILTER_CLAUSE`, so the suggestion leads to a dead end. Suggest returning the unsupported-clause error first whenever any SIMPLE HAVING filter is present (that is still order-independent), dropping the HAVING metric suggestions, and updating that master test to expect `UNSUPPORTED_FILTER_CLAUSE` for HAVING.
##########
superset/mcp_service/chart/validation/dataset_validator.py:
##########
@@ -252,13 +332,43 @@ def validate_against_dataset(
# handlebars / big number slip through ``SUM(non_numeric)`` patterns
# for the fast-path tools that skip Tier 2.
aggregation_errors = DatasetValidator._validate_aggregations(
- column_refs, dataset_context
+ column_refs,
+ dataset_context,
+ require_numeric_metrics=isinstance(config, SunburstChartConfig),
Review Comment:
P3: Master's new `_extract_metric_references` (field tuple `("y", "y_secondary", "metric", "metrics", "size")`) does not include Sunburst's `secondary_metric`. A near-miss saved-metric name there falls through to "No matching columns found" with no saved-metric hint, while the same typo in `metric` gets one. Adding `"secondary_metric"` to that tuple, plus a small test, would bring Sunburst into the merged hinting behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4203089075
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -3867,6 +5073,134 @@ def _normalize_chart_request_input(data: Any) -> Any:
config.pop("viz_type", None)
elif config.get("chart_type") != "table":
config.pop("viz_type", None)
+
+ if config.get("chart_type") == "xy":
+ # Native Timeseries form data uses ``x_axis`` for the semantic
+ # query column, while the typed MCP contract uses the same spelling
+ # for AxisConfig presentation state. Disambiguate before the
+ # discriminated union runs: scalars and native query-column objects
+ # become ``x``; title/format/scale mappings remain ``x_axis``.
+ if "x_axis" in config and config["x_axis"] is not None:
+ native_x_axis = config["x_axis"]
+ is_native_column = not isinstance(native_x_axis, dict) or bool(
+ {
+ "column",
+ "column_name",
+ "columnName",
+ "expressionType",
+ "sqlExpression",
+ }
+ & set(native_x_axis)
+ )
+ if is_native_column:
+ if "x" in config:
+ raise ValueError(
+ "XY config cannot provide both semantic x and native "
+ "x_axis query columns"
+ )
+ if isinstance(native_x_axis, dict):
+ allowed_x_axis_keys = {
+ "column",
+ "column_name",
+ "columnName",
+ "columnType",
+ "expressionType",
+ "label",
+ "sqlExpression",
+ "timeGrain",
+ }
+ if unknown := set(native_x_axis) - allowed_x_axis_keys:
+ raise ValueError(
+ "Unknown XY native x_axis field(s): "
+ + ", ".join(sorted(unknown))
+ )
+ expression_type = native_x_axis.get("expressionType")
+ if expression_type == "SIMPLE":
+ column = native_x_axis.get("column")
+ if isinstance(column, dict):
+ column = XYNativeMetricColumn.model_validate(
+ column
+ ).column_name
+ native_x_axis = {
+ "name": native_x_axis.get("column_name")
+ or native_x_axis.get("columnName")
+ or column,
+ }
+ elif (
+ expression_type == "SQL"
+ and native_x_axis.get("columnType") == "BASE_AXIS"
+ ):
+ native_x_axis = {
+ "name": native_x_axis.get("sqlExpression"),
Review Comment:
The XY native-axis normalizer accepts `timeGrain` for a `BASE_AXIS` SQL x-axis (line 5110) but rebuilds the axis as only `name`/`label`, so the grain is dropped. A native line-chart config with `x_axis={"expressionType":"SQL","columnType":"BASE_AXIS","sqlExpression":"ds","label":"ds","timeGrain":"P1M"}` and no panel-level `time_grain` validates, but the generated/updated chart queries an unbucketed `ds` instead of monthly aggregates. Should the grain be carried over to `config.time_grain` here, or rejected as unsupported?
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,790 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "order_by_cols": (("sort_by",),),
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
Review Comment:
On a same-viz update every key in the plugin's `query_role_keys` is dropped from the saved form data before the mapper output is overlaid, and `groupby` for XY charts is not one of the modeled controls that gets restored. A saved line chart grouped by `region` that receives `update_chart` with the same `x`/`y` and `stacked: true` but no `group_by` will have its regional series collapsed into one aggregate series. Is dropping the saved grouping on omission intended (with only an explicit `group_by: []` meant to clear it)? If so it seems worth calling out in UPDATING.md; otherwise should `groupby` be preserved when `group_by` is unset?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +427,1309 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+ sort_metric: Any = None,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ sort_metric_label = (
+ _label(sort_metric, metric=True) if sort_metric is not None else None
+ )
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else list(metric_labels)
+ )
+ if not offset_map and sort_metric_label is not None:
+ aggregate_labels.append(sort_metric_label)
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ sort_metric_label or "",
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ extra_metric = sort_metric
+ else:
+ extra_metric = None
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ sort_metric=extra_metric,
+ )
+ if extra_metric is not None:
+ query.setdefault("metrics", []).append(extra_metric)
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ # Saved/native Big Number payloads can carry the temporal binding as a
+ # ``{"column_name": ...}`` mapping, which the strict frontend predicate does
+ # not recognize; keep grouping by it rather than falling back to a total.
+ frontend_x_axis = _frontend_x_axis_column(form_data)
+ explicit_x_axis = frontend_x_axis or _x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = (
+ _label(explicit_x_axis)
+ if explicit_x_axis
+ else _x_axis_label(form_data, frontend_strict=True)
+ )
+ query["columns"] = time_column
+ if time_column and frontend_x_axis is None:
+ # A native ``{"column_name": ...}`` axis groups by its temporal column
+ # but is not rewritten by normalize_time_column, so drop the legacy
+ # granularity binding here rather than bucketing the same dimension
+ # twice.
+ query.pop("granularity", None)
+ extras = query.get("extras")
+ if isinstance(extras, dict):
+ extras.pop("time_grain_sqla", None)
+ if not extras:
+ query.pop("extras", None)
+ elif not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
query["post_processing"] = post_processing
- # ``granularity`` does two jobs downstream: it names the temporal column the
- # time range filters on, and it is the column ``time_grain_sqla`` buckets
- # (``models/helpers.py`` swaps a selected column for its timestamp expression
- # when that column equals ``granularity``). Only the first job depends on
- # there being an active range, so set it whenever form data carries one โ
- # matching ``extractExtras.ts``, which sets it unconditionally. Gating it on
- # ``time_range`` dropped the bucketing, so an ordinary "all-time totals by
- # month" chart exported one row per raw timestamp instead of one per month.
- if granularity := form_data.get("granularity") or form_data.get("granularity_sqla"):
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
+ page_size = int(form_data.get("server_page_length") or 0)
+ configured_limit = int(form_data.get("row_limit") or 0)
+ query["row_limit"] = (
+ min(page_size, configured_limit) if configured_limit else page_size
Review Comment:
The preserved native pagination overrides the explicit compile/preview row cap here. With a saved Table that has `server_pagination=true` and `server_page_length=0`, a requested `row_limit` of 2 gives `min(0, 2)` = 0, so `QueryObjectFactory` falls back to the deployment default row limit and the two-row compile sample becomes a full-sized query. Should the explicit caller limit win when the page length is 0 (e.g. only apply the `min` when `page_size` is positive)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-6011922318
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4194654425
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,67 +15,1487 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+import base64
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
from decimal import Decimal
+from enum import Enum
from numbers import Real
from typing import Any, cast
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.utils.serialization import BINARY_PREFIX
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _canonical_binary(
+ value: bytes | bytearray | memoryview,
+) -> tuple[str | None, str | None]:
+ """Render exact binary cells as UTF-8 text or ``base64:``-prefixed text.
+
+ Matches the MCP response serializer: valid UTF-8 is returned as text and
+ anything else is losslessly base64-encoded. The raw size is bounded before
+ copying so the rendered string stays within the per-cell text budget.
+ """
+ size = value.nbytes if type(value) is memoryview else len(value)
+ if size > MAX_RESULT_STRING_LENGTH:
+ return None, "an oversized binary value"
+ try:
+ raw = bytes(value)
+ except (BufferError, TypeError, ValueError):
+ return None, "an invalid binary value"
+ try:
+ text = raw.decode("utf-8")
+ except UnicodeDecodeError:
+ text = BINARY_PREFIX + base64.b64encode(raw).decode("ascii")
+ if _bounded_utf8_length(text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized binary value"
+ return text, None
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+ if value_type is bytes or value_type is bytearray or value_type is memoryview:
+ return _canonical_binary(value)
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = list.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (list.__getitem__(item, index), item, index, depth + 1, False)
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ children: list[tuple[Any, dict[str, Any], str, int, bool]] = []
+ for key, child in dict.items(item):
+ if type(key) is not str:
+ return None, "a non-string object key"
+ if reason := _charge_text(key, budget, key=True, metadata=metadata):
+ return None, reason
+ children.append((child, item, key, depth + 1, False))
+ stack.append((item, None, None, depth, True))
+ stack.extend(reversed(children))
+ continue
+
+ source_item = item
+ if type(item) in enum_types:
+ normalized = _safe_enum_value(item, enum_types)
+ if normalized is None:
+ return None, "an unsupported enum"
+ item = normalized
+ elif any(base is Enum for base in _type_mro(type(item))):
+ return None, "an enum outside its expected metadata slot"
+
+ normalized, reason = _normalize_scalar(item)
+ if reason is not None:
+ return None, reason
+ max_string_bytes = (
+ MAX_QUERY_RESULT_METADATA_BYTES if metadata else MAX_RESULT_STRING_LENGTH
+ )
+ if type(normalized) is str:
+ scalar_size = _json_string_size(normalized, max_string_bytes)
+ if scalar_size is None:
+ return None, "invalid text data"
+ else:
+ scalar_size = _normalized_scalar_json_size(normalized)
+ if reason := _charge_json_bytes(
+ budget,
+ scalar_size,
+ metadata=metadata,
+ ):
+ return None, reason
+ if parent is None:
+ root = normalized
+ elif normalized is not source_item:
+ if type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized)
+ return root, None
+
+
+def _normalize_metadata_value(
+ payload: dict[str, Any], key: str, budget: _ResultBudget
+) -> str | None:
+ enum_slots: dict[str, frozenset[type[Any]]] = {
+ "status": frozenset({QueryStatus}),
+ "result_format": frozenset({ChartDataResultFormat}),
+ "coltypes": frozenset({GenericDataType}),
+ "applied_filters": frozenset({ExtraFiltersTimeColumnType}),
+ "rejected_filters": frozenset(
+ {ExtraFiltersReasonType, ExtraFiltersTimeColumnType}
+ ),
+ }
+ value = dict.__getitem__(payload, key)
+ normalized, reason = _normalize_value(
+ value,
+ budget,
+ enum_types=enum_slots.get(key, frozenset()),
+ metadata=True,
+ )
+ if reason is None and normalized is not value:
+ dict.__setitem__(payload, key, normalized)
+ return reason
+
+
+def _error_text(value: Any) -> str | None:
+ """Extract a bounded error from an already validated primitive tree."""
+ stack: list[Any] = [value]
+ parts: list[str] = []
+ used = 0
+ while stack and len(parts) < 3 and used < _MAX_ERROR_TEXT_BYTES:
+ item = stack.pop()
+ if type(item) is dict:
+ stack.extend(
+ reversed(
+ [dict.__getitem__(item, key) for key in _ERROR_KEYS if key in item]
+ )
+ )
+ continue
+ if type(item) is list:
+ stack.extend(
+ list.__getitem__(item, index)
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if item is None or item is False:
+ continue
+ if type(item) is str:
+ remaining = _MAX_ERROR_TEXT_BYTES - used - (2 if parts else 0)
+ encoded = str.encode(item, "utf-8")[:remaining]
+ text = bytes.decode(encoded, "utf-8", errors="ignore")
+ if text:
+ parts.append(text)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ return "; ".join(parts) or None
+
+
+def _failure_for_payload(payload: dict[str, Any], label: str) -> ChartError | None:
for key in ("error", "errors", "error_message"):
- if message := _query_error_text(payload.get(key)):
+ if key in payload and (message := _error_text(dict.__getitem__(payload, key))):
return ChartError(
error=f"{label} failed: {message}", error_type="QueryError"
)
- raw_status = payload.get("status")
- status = str(getattr(raw_status, "value", raw_status) or "")
+ raw_status = dict.get(payload, "status")
+ status = raw_status if type(raw_status) is str else ""
normalized_status = status.strip().casefold().replace("-", "_").replace(" ", "_")
if normalized_status in FAILED_QUERY_STATUSES:
message = (
- _query_error_text(payload.get("message"))
- or _query_error_text(payload.get("error_message"))
+ _error_text(dict.get(payload, "message"))
+ or _error_text(dict.get(payload, "error_message"))
or normalized_status
)
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
- if payload.get("success") is False:
- message = _query_error_text(payload.get("message")) or "request failed"
+ if dict.get(payload, "success") is False:
+ message = _error_text(dict.get(payload, "message")) or "request failed"
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if raw_status is None and "data" not in payload and "queries" not in payload:
+ if message := _error_text(dict.get(payload, "message")):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+ return None
+
+
+def _metadata_shape_error( # noqa: C901
+ payload: dict[str, Any], label: str, budget: _ResultBudget
+) -> ChartError | None:
+ if "success" in payload and type(dict.__getitem__(payload, "success")) is not bool:
+ return _invalid_metadata(label)
+ if "status" in payload and type(dict.__getitem__(payload, "status")) is not str:
+ return _invalid_metadata(label)
if (
- raw_status is None
- and "data" not in payload
- and "queries" not in payload
- and (message := _query_error_text(payload.get("message")))
+ "result_format" in payload
+ and dict.__getitem__(payload, "result_format") not in _RESULT_FORMAT_VALUES
):
- return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ return _invalid_metadata(label)
+ for count_key in ("rowcount", "sql_rowcount", "total_rows"):
+ if count_key in payload:
+ count = dict.__getitem__(payload, count_key)
+ if count is not None and not (
+ type(count) is int and 0 <= count <= MAX_QUERY_RESULT_ROWCOUNT
+ ):
+ return _invalid_metadata(label)
+ if "is_cached" in payload:
+ cached = dict.__getitem__(payload, "is_cached")
+ if cached is None:
+ if reason := _charge_json_bytes(
+ budget, 1, metadata=True
+ ): # ``false`` vs ``null``
+ return _invalid_result(reason)
+ dict.__setitem__(payload, "is_cached", False)
+ elif type(cached) is not bool:
+ return _invalid_metadata(label)
+ if "cache_timeout" in payload:
+ timeout = dict.__getitem__(payload, "cache_timeout")
+ if timeout is not None and not (
+ type(timeout) is int
+ and (
+ timeout == CACHE_DISABLED_TIMEOUT
+ or 0 <= timeout <= MAX_QUERY_RESULT_CACHE_TIMEOUT
+ )
+ ):
+ return _invalid_metadata(label)
+ if "cache_key" in payload:
+ cache_key = dict.__getitem__(payload, "cache_key")
+ if cache_key is not None and (type(cache_key) is not str or not cache_key):
+ return _invalid_metadata(label)
+ for timestamp_key in ("cached_dttm", "cache_dttm", "queried_dttm"):
+ if timestamp_key not in payload:
+ continue
+ timestamp = dict.__getitem__(payload, timestamp_key)
+ if timestamp is None:
+ continue
+ if (
+ type(timestamp) is not str
+ or not timestamp
+ or len(timestamp) > MAX_QUERY_RESULT_TIMESTAMP_LENGTH
+ ):
+ return _invalid_metadata(label)
+ normalized = f"{timestamp[:-1]}+00:00" if timestamp.endswith("Z") else timestamp
+ try:
+ parsed = datetime.fromisoformat(normalized)
+ except ValueError:
+ return _invalid_metadata(label)
+ if parsed.tzinfo is None:
+ parsed = parsed.replace(tzinfo=timezone.utc)
+ if parsed.utcoffset() != timedelta(0):
+ return _invalid_metadata(label)
+ canonical_timestamp = parsed.astimezone(timezone.utc).isoformat()
+ original_size = _json_string_size(timestamp, MAX_RESULT_STRING_LENGTH)
+ canonical_size = _json_string_size(
+ canonical_timestamp, MAX_RESULT_STRING_LENGTH
+ )
+ assert original_size is not None
+ assert canonical_size is not None
+ if canonical_size > original_size and (
+ reason := _charge_json_bytes(
+ budget, canonical_size - original_size, metadata=True
+ )
+ ):
+ return _invalid_result(reason)
+ dict.__setitem__(payload, timestamp_key, canonical_timestamp)
+ return None
+
+
+def _filter_metadata_is_valid(value: Any, *, rejected: bool) -> bool:
+ if type(value) is not list or list.__len__(value) > MAX_RESULT_VALUE_ITEMS:
+ return False
+ expected = {"column", "reason"} if rejected else {"column"}
+ for entry in value:
+ if type(entry) is not dict or set(dict.keys(entry)) != expected:
+ return False
+ if type(dict.__getitem__(entry, "column")) is not str:
+ return False
+ if rejected and type(dict.__getitem__(entry, "reason")) is not str:
+ return False
+ return True
+
+
+def _validate_query_metadata(query: dict[str, Any], index: int) -> ChartError | None:
+ label = f"Chart query {index}"
+ colnames_present = "colnames" in query
+ coltypes_present = "coltypes" in query
+ colnames = dict.get(query, "colnames", [])
+ coltypes = dict.get(query, "coltypes", [])
+ if (
+ type(colnames) is not list
+ or list.__len__(colnames) > MAX_QUERY_RESULT_COLUMNS
+ or any(type(column) is not str or not column for column in colnames)
+ ):
+ return _invalid_result(f"malformed column metadata for query {index}")
+ if (
+ type(coltypes) is not list
+ or list.__len__(coltypes) > MAX_QUERY_RESULT_COLUMNS
+ or any(
+ type(value) is not int or value not in _COLTYPE_VALUES for value in coltypes
+ )
+ ):
+ return _invalid_result(f"malformed column type metadata for query {index}")
+ if (colnames_present or coltypes_present) and (
+ not colnames_present
+ or not coltypes_present
+ or list.__len__(colnames) != list.__len__(coltypes)
+ ):
+ return _invalid_result(f"misaligned column metadata for query {index}")
+ if "applied_filters" in query and not _filter_metadata_is_valid(
+ dict.__getitem__(query, "applied_filters"), rejected=False
+ ):
+ return _invalid_metadata(label)
+ if "rejected_filters" in query and not _filter_metadata_is_valid(
+ dict.__getitem__(query, "rejected_filters"), rejected=True
+ ):
+ return _invalid_metadata(label)
+ if "rejected_filter_columns" in query:
+ rejected_columns = dict.__getitem__(query, "rejected_filter_columns")
+ if type(rejected_columns) is not list or any(
+ type(column) is not str for column in rejected_columns
+ ):
+ return _invalid_metadata(label)
+ return None
+
+
+def validate_query_result_envelope( # noqa: C901
+ result: Any, *, none_as_empty: bool = False
+) -> ChartError | None:
+ """Canonicalize and strictly validate one real command result in place.
+
+ ``ChartDataCommand.run`` adds a live ``query_context`` sidecar. It is
+ intentionally exempted without access or traversal: MCP consumers only use
+ the wire query payload, and inspecting the sidecar would cross arbitrary
+ datasource/query hooks. Every wire value is charged against one aggregate
+ row/node/UTF-8 budget shared by all queries.
+ """
+ if type(result) is not dict:
+ return _invalid_result("a malformed result envelope")
+
+ budget = _ResultBudget()
+ if reason := _charge_value(budget, metadata=True):
+ return _invalid_result(reason)
+ top_level_keys = list(dict.keys(result))
+ if any(type(key) is not str for key in top_level_keys):
+ return _invalid_metadata("Chart query")
+ has_query_context = any(key == "query_context" for key in top_level_keys)
+ wire_item_count = dict.__len__(result) - has_query_context
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(wire_item_count, mapping=True),
+ metadata=True,
+ ):
+ return _invalid_result(reason)
+ for key in top_level_keys:
+ if key == "query_context":
+ continue
+ if reason := _charge_text(key, budget, key=True, metadata=True):
+ return _invalid_result(reason)
+ if key == "queries":
+ continue
+ if reason := _normalize_metadata_value(result, key, budget):
+ return _invalid_metadata("Chart query")
+
+ if error := _metadata_shape_error(result, "Chart query", budget):
+ return error
+ if failure := _failure_for_payload(result, "Chart query"):
+ return failure
+ if not dict.__contains__(result, "queries"):
+ return _invalid_result("no query result envelope")
+ queries = dict.__getitem__(result, "queries")
+ if type(queries) is not list or not queries:
+ return _invalid_result("no query result envelope")
+ if list.__len__(queries) > MAX_QUERY_RESULTS:
+ return _invalid_result("too many query result envelopes")
+ if reason := _charge_value(budget):
+ return _invalid_result(reason)
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(list.__len__(queries), mapping=False),
+ ):
+ return _invalid_result(reason)
+
+ for offset in range(list.__len__(queries)):
+ index = offset + 1
+ query = list.__getitem__(queries, offset)
+ if type(query) is not dict:
+ return _invalid_result(f"a malformed query {index} result envelope")
+ if reason := _charge_value(budget):
+ return _invalid_result(reason)
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(dict.__len__(query), mapping=True),
+ metadata=True,
+ ):
+ return _invalid_result(reason)
+ for key in list(dict.keys(query)):
+ if type(key) is not str:
+ return _invalid_metadata(f"Chart query {index}")
+ if reason := _charge_text(key, budget, key=True, metadata=True):
+ return _invalid_result(reason)
+ if key in {"data", "indexnames"}:
+ continue
+ if reason := _normalize_metadata_value(query, key, budget):
+ return _invalid_metadata(f"Chart query {index}")
+
+ if error := _metadata_shape_error(query, f"Chart query {index}", budget):
+ return error
+ if failure := _failure_for_payload(query, f"Chart query {index}"):
+ return failure
+ if not dict.__contains__(query, "data"):
+ return _invalid_result(f"query {index} result data is not an array of rows")
+ data = dict.__getitem__(query, "data")
+ if data is None and none_as_empty:
+ data = []
+ dict.__setitem__(query, "data", data)
+ if type(data) is not list:
+ return _invalid_result(f"query {index} result data is not an array of rows")
+ data_length = list.__len__(data)
+ if data_length > MAX_QUERY_RESULT_ROWS_PER_QUERY:
+ return _invalid_result(f"too many rows for query {index}")
+ budget.rows += data_length
+ if budget.rows > MAX_QUERY_RESULT_ROWS:
+ return _invalid_result("too many aggregate rows")
+ if reason := _charge_value(budget):
+ return _invalid_result(reason)
+ if reason := _charge_json_bytes(
+ budget, _container_json_syntax_size(data_length, mapping=False)
+ ):
+ return _invalid_result(reason)
+
+ if error := _validate_query_metadata(query, index):
+ return error
+ declared_columns = (
+ dict.__getitem__(query, "colnames") if "colnames" in query else None
+ )
+ declared_row_columns = (
+ list(dict.fromkeys(declared_columns))
+ if declared_columns is not None
+ else None
+ )
+
+ # Check declared order in the same capped pass that normalizes cells.
+ # A width mismatch fails before comparing names, so empty rows cannot
+ # multiply work by the maximum declared column count.
+ for row_offset in range(data_length):
+ row = list.__getitem__(data, row_offset)
+ if type(row) is not dict or dict.__len__(row) > MAX_QUERY_RESULT_COLUMNS:
+ return _invalid_result(f"a malformed data row for query {index}")
+ if reason := _charge_value(budget):
+ return _invalid_result(reason)
+ row_items = list(dict.items(row))
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(list.__len__(row_items), mapping=True),
+ ):
+ return _invalid_result(reason)
+ if declared_row_columns is not None and list.__len__(
+ row_items
+ ) != list.__len__(declared_row_columns):
+ return ChartError(
+ error=(
+ f"Chart query {index} returned rows that do not match the "
+ "declared column order."
+ ),
+ error_type="InvalidQueryResult",
+ )
+ for column_offset, (column, value) in enumerate(row_items):
+ if type(column) is not str:
+ return _invalid_result(
+ f"hostile or oversized row data for query {index}"
+ )
+ if declared_row_columns is not None and column != list.__getitem__(
+ declared_row_columns, column_offset
+ ):
+ return ChartError(
+ error=(
+ f"Chart query {index} returned rows that do not match "
+ "the declared column order."
+ ),
+ error_type="InvalidQueryResult",
+ )
+ if reason := _charge_text(column, budget, key=True):
+ return _invalid_result(reason)
+ normalized, reason = _normalize_value(value, budget)
Review Comment:
Fixed in 6a6f8d51884404ba979ded002fe961c3e64a68ce. Exact tuple and NumPy-array cells are converted to JSON lists within the existing depth, item, and aggregate byte budgets. Original container identities are tracked for cycle detection; subclasses remain rejected without invoking their conversion hooks. test_sequence_cells_are_canonical_json_arrays covers tuples, Arrow-style arrays, nested values, and multidimensional arrays; all four cases failed before and pass after. Additional tests cover width/byte limits, object-array cycles, and subclass hooks.
##########
superset/common/form_data_query_context.py:
##########
@@ -288,71 +461,1293 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
-def build_query_context_from_form_data(
- form_data: dict[str, Any],
- datasource: dict[str, Any],
- viz_type: str | None = None,
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
) -> dict[str, Any]:
- """
- Build a query-context payload (the JSON shape ``ChartDataQueryContextSchema``
- loads) from a chart's form data and datasource reference.
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
- :param form_data: The chart's saved ``params`` parsed to a dict.
- :param datasource: ``{"id": <int>, "type": "table"}`` datasource reference.
- :param viz_type: The chart's viz type, used for viz-specific handling.
- :returns: A single-query query-context dict.
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
"""
- columns, metrics = _columns_and_metrics(form_data, viz_type)
-
- # SIMPLE adhoc filters (+ legacy top-level ``filters``) become query filters;
- # free-form SQL predicates go into ``extras``. Only ``WHERE``-clause SIMPLE
- # filters are applied (matching the chart), so the export never filters on a
- # ``HAVING`` clause the chart itself ignores.
- filters = adhoc_filters_to_query_filters(
- form_data.get("adhoc_filters", []), where_only=True
- )
- for flt in form_data.get("filters") or []:
- if isinstance(flt, dict) and flt.get("col") is not None:
- filters.append(flt)
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
- extras = freeform_where_having(form_data)
- if form_data.get("time_grain_sqla"):
- extras["time_grain_sqla"] = form_data["time_grain_sqla"]
- # Prefer the modern ``time_range``; fall back to the legacy ``since``/``until``
- # pair (older charts store the range that way) before defaulting to no filter.
- time_range = form_data.get("time_range")
- if not time_range and (form_data.get("since") or form_data.get("until")):
- time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
- time_range = time_range or "No filter"
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
query: dict[str, Any] = {
"columns": columns,
"metrics": metrics,
- "orderby": orderby_from_form_data(form_data, metrics, viz_type),
- "filters": filters,
- "time_range": time_range,
}
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
if extras:
query["extras"] = extras
- if viz_type == "pie" and (
- post_processing := _pie_contribution_post_processing(metrics)
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ **(
+ {"fill_time_range": True}
+ if form_data.get("resample_fill_time_range")
+ else {}
+ ),
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ x_axis = _x_axis_column(form_data)
+ axis_grain = x_axis.get("timeGrain") if isinstance(x_axis, Mapping) else None
+ time_grain = (
+ axis_grain
+ or (query.get("extras") or {}).get("time_grain_sqla")
+ or form_data.get("time_grain_sqla")
+ or "P1D"
+ )
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": time_grain,
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ query.setdefault("metrics", []).append(sort_metric)
Review Comment:
Fixed in 6a6f8d51884404ba979ded002fe961c3e64a68ce. The extra sort metric is appended to SELECT metrics after the displayed-metric post-processing labels are derived. It remains available to pivot and sort, but rolling/cumulative operators only transform displayed metrics. test_timeseries_sort_only_metric_keeps_raw_values executes both pipelines: orders [100, 1] remain raw and sort ascending to [1, 100], while sales are transformed. Both cases failed before and pass after.
##########
superset/mcp_service/chart/response_preflight.py:
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Schema-preserving final wire-size gates for MCP chart responses."""
+
+from typing import Any, TypeVar
+
+from pydantic import BaseModel, RootModel
+
+from superset.mcp_service.chart.query_result import response_json_failure
+from superset.mcp_service.chart.schemas import (
+ ChartData,
+ ChartError,
+ GenerateChartResponse,
+)
+from superset.mcp_service.dataset.schemas import DatasetError, QueryDatasetResponse
+from superset.mcp_service.semantic_layer.schemas import (
+ GetTableResponse,
+ SemanticLayerError,
+)
+
+_ResponseModel = TypeVar("_ResponseModel", bound=BaseModel)
+_MAX_EXCEPTION_PARTS = 3
+_MAX_EXCEPTION_TEXT_BYTES = 256
+
+
+class UpdateChartPreviewResponse(RootModel[dict[str, Any]]):
+ """Typed wire projection for the legacy dict-shaped preview response."""
+
+
+def bounded_exception_message(exception: BaseException) -> str: # noqa: C901
+ """Extract bounded primitive exception arguments without formatting hooks."""
+ try:
+ args = object.__getattribute__(exception, "args")
+ except Exception: # pragma: no cover - BaseException provides args
+ args = ()
+ parts: list[str] = []
+ used = 0
+ if type(args) is tuple:
+ for index in range(min(tuple.__len__(args), _MAX_EXCEPTION_PARTS)):
+ value = tuple.__getitem__(args, index)
+ if type(value) is str:
+ text = value
+ elif type(value) is bool:
+ text = "True" if value else "False"
+ elif type(value) is int and int.bit_length(value) <= 4_096:
+ text = int.__str__(value)
+ elif type(value) is float:
+ text = float.__repr__(value)
+ elif value is None:
+ text = "None"
+ else:
+ continue
+ remaining = _MAX_EXCEPTION_TEXT_BYTES - used - (2 if parts else 0)
+ if remaining <= 0:
+ break
+ encoded = str.encode(text[:remaining], "utf-8", errors="replace")[
+ :remaining
+ ]
+ bounded = bytes.decode(encoded, "utf-8", errors="ignore")
+ if bounded:
+ parts.append(bounded)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ if parts:
+ return "; ".join(parts)
+ return "request failed"
+
+
+def finalize_chart_data_response(
+ response: ChartData | ChartError,
+) -> ChartData | ChartError:
+ """Preflight every chart-data result while preserving its public union."""
+ if response_json_failure(response) is None:
+ return response
+ return ChartError(
+ error="Chart data response could not be returned safely.",
+ error_type="InvalidQueryResult",
+ details="The complete response exceeded its serialization safety limits.",
+ suggestions=["Request fewer rows or use a narrower export."],
+ error_code="CHART_DATA_RESPONSE_TOO_LARGE",
+ )
+
+
+def finalize_query_dataset_response(
+ response: QueryDatasetResponse | DatasetError,
+) -> QueryDatasetResponse | DatasetError:
+ """Preflight every dataset-query result while preserving its public union."""
+ if response_json_failure(response) is None:
+ return response
+ return DatasetError.create(
+ error="Dataset response could not be returned safely.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def finalize_get_table_response(
+ response: GetTableResponse | SemanticLayerError,
+) -> GetTableResponse | SemanticLayerError:
+ """Preflight every semantic-table result while preserving its public union."""
+ if response_json_failure(response) is None:
+ return response
+ return SemanticLayerError.create(
+ error="Semantic table response could not be returned safely.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def finalize_chart_response(
+ response: _ResponseModel,
+) -> _ResponseModel | ChartError:
+ """Preserve a chart response schema or return a bounded structured error."""
+ return response_json_failure(response) or response
+
+
+def finalize_generate_chart_response(
+ response: GenerateChartResponse,
+) -> GenerateChartResponse:
+ """Preflight generate/update responses and preserve their response schema."""
+ failure = response_json_failure(response)
+ if failure is None:
+ return response
+ return GenerateChartResponse.model_validate(
+ {
+ "chart": None,
Review Comment:
Fixed in 6a6f8d51884404ba979ded002fe961c3e64a68ce. When a successful response describes a committed chart, preflight retains success=true and the saved chart.id, drops oversized optional content, and returns a warning to retrieve that ID rather than retry the write. Unsaved responses still return the structured failure. Tests cover oversized previews and form data, plus actual generate/update paths with the create/update commands already executed; the committed-write regressions failed before and pass after.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1694,6 +1772,789 @@ def map_bubble_config(config: BubbleChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "GaugeChartConfig": {
+ key: ((key,),)
+ for key in (
+ "sort_by_metric",
+ "row_limit",
+ "min_val",
+ "max_val",
+ "color_scheme",
+ "font_size",
+ "number_format",
+ "currency_format",
+ "value_formatter",
+ "start_angle",
+ "end_angle",
+ "show_pointer",
+ "animation",
+ "show_axis_tick",
+ "show_split_line",
+ "split_number",
+ "show_progress",
+ "overlap",
+ "round_cap",
+ "intervals",
+ "interval_color_indices",
+ "time_range",
+ "granularity_sqla",
+ )
+ },
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "GanttChartConfig": {
+ "tooltip_columns": (("tooltip_columns",),),
+ "tooltip_metrics": (("tooltip_metrics",),),
+ "order_by_cols": (("order_by",),),
+ "row_limit": (("row_limit",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+_SAVED_PREDICATE_FORM_DATA_KEYS = frozenset(
+ {"adhoc_filters", "extra_filters", "filters", "having", "where"}
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+
+ A dataset rebind prunes every dataset-bound role from the saved state and
+ then merges as a same-dataset update, unless the owning plugin declares a
+ strict rebind contract (``strict_dataset_rebind``), in which case its
+ ``merge_update_form_data`` hook receives ``dataset_rebind=True``. Plugins
+ that declare ``owns_update_merge`` merge same-viz updates themselves;
+ every other update takes the shared overlay and then the plugin's
+ ``finalize_update_form_data`` hook.
+ """
+ from superset.mcp_service.chart.registry import plugin_for_viz_type
+
+ plugin = plugin_for_viz_type(new_form_data.get("viz_type"))
+ if dataset_rebind and not (plugin is not None and plugin.strict_dataset_rebind):
+ existing_form_data = scrub_dataset_bound_form_data(
+ existing_form_data,
+ target_viz_type=new_form_data.get("viz_type"),
+ )
+ dataset_rebind = False
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ if same_viz and plugin is not None and (dataset_rebind or plugin.owns_update_merge):
+ plugin_merged = plugin.merge_update_form_data(
+ existing_form_data,
+ new_form_data,
+ config,
+ dataset_rebind=dataset_rebind,
+ )
+ if plugin_merged is not None:
+ return plugin_merged
+ if dataset_rebind:
+ # A strict rebind never inherits saved state the plugin did not merge.
+ return dict(new_form_data)
+
+ merged = overlay_update_form_data(existing_form_data, new_form_data, config)
+ if plugin is not None:
+ merged = plugin.finalize_update_form_data(
+ existing_form_data, new_form_data, merged, config
+ )
+ return merged
+
+
+def overlay_update_form_data(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Overlay a same-dataset update on the saved state with shared semantics."""
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
Review Comment:
Fixed in 6a6f8d51884404ba979ded002fe961c3e64a68ce. Table order_by_cols is mapped to the modeled sort_by update control: omission retains saved ordering, an explicit empty list clears it, and supplied sorting replaces it. test_table_update_retains_omitted_ordering covers Table and AG Grid Table through merge_form_data_for_update and the common query builder with row_limit=1. Both omission cases failed before and pass after; clear/replacement cases also pass. Full MCP/common-query suite: 8,226 passed, 4 skipped. Pre-commit passed on all 80 branch-changed files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r4206809018
##########
superset/mcp_service/chart/chart_helpers.py:
##########
@@ -450,12 +582,200 @@ def _resolve_deck_gl_metrics(
if value:
metrics.append(value)
elif isinstance(prf, str) and _is_metric_ref(prf):
- # Legacy deck_scatter: point_radius_fixed as a bare non-numeric metric key
- logger.debug("Legacy point_radius_fixed string metric encountered: %s", prf)
metrics.append(prf)
return metrics
+def _deck_query_adapter( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any], viz_type: str
+) -> dict[str, Any]:
+ """Apply the native frontend builder for a single Deck.gl layer."""
+ base_columns = list(query.get("columns") or [])
+ base_metrics = list(query.get("metrics") or [])
+ filters = list(query.get("filters") or [])
+ tooltips = _deck_tooltip_columns(form_data.get("tooltip_contents"))
+
+ def add_null(column: str, *, value: Any = ...) -> None:
+ clause: dict[str, Any] = {"col": column, "op": "IS NOT NULL"}
+ if value is not ...:
+ clause["val"] = value
+ filters.append(clause)
+
+ if viz_type == "deck_geojson":
+ geometry = form_data.get("geojson")
+ if not isinstance(geometry, str) or not geometry:
+ raise ValueError("GeoJSON column is required for GeoJSON charts")
+ columns = _add_deck_columns(base_columns, [geometry] if geometry else [])
+ cross_filter = form_data.get("cross_filter_column")
+ if cross_filter:
+ columns = _add_deck_columns(columns, [cross_filter])
+ columns = _add_deck_columns(columns, tooltips)
+ if form_data.get("filter_nulls", True) and isinstance(geometry, str):
+ add_null(geometry)
+ query.update(
+ columns=columns,
+ metrics=[],
+ groupby=[],
+ filters=filters,
+ is_timeseries=False,
+ )
+ return query
+
+ if viz_type == "deck_polygon":
+ line_column = form_data.get("line_column")
+ if not isinstance(line_column, str) or not line_column:
+ raise ValueError("Polygon column is required for Polygon charts")
+ columns = _add_deck_columns(base_columns, [line_column] if line_column else [])
+ cross_filter = form_data.get("cross_filter_column")
+ if cross_filter:
+ columns = _add_deck_columns(columns, [cross_filter])
+ columns = _add_deck_columns(columns, tooltips)
+ metrics: list[Any] = []
+ if metric := form_data.get("metric"):
+ metrics.append(metric)
+ radius = form_data.get("point_radius_fixed")
+ if (
+ isinstance(radius, dict)
+ and radius.get("type") == "metric"
+ and radius.get("value") is not None
+ ):
+ metrics.append(radius["value"])
+ if form_data.get("filter_nulls", True) and isinstance(line_column, str):
+ add_null(line_column)
+ if metric:
+ add_null(_deck_metric_label(metric))
+ query.update(
+ columns=columns,
+ metrics=metrics,
+ filters=filters,
+ is_timeseries=False,
+ )
+ return query
+
+ if viz_type == "deck_path":
+ line_column = form_data.get("line_column")
+ if not isinstance(line_column, str) or not line_column:
+ raise ValueError("Line column is required for Path charts")
+ columns = list(base_columns)
+ metrics = [metric for metric in base_metrics if _is_deck_metric_value(metric)]
+ groupby = list(query.get("groupby") or [])
+ metric = form_data.get("metric")
+ if metrics or metric:
+ if metric and metric not in metrics:
+ metrics.append(metric)
+ if line_column and line_column not in groupby:
+ groupby.append(line_column)
+ elif line_column:
+ columns = _add_deck_columns(columns, [line_column])
+ if dimension := form_data.get("dimension"):
+ columns = _add_deck_columns(columns, [dimension])
+
+ line_width = form_data.get("line_width")
+ raw_width = (
+ line_width
+ if isinstance(line_width, str)
+ else line_width.get("value")
+ if isinstance(line_width, dict)
+ else None
+ )
+ width_metric = (
+ raw_width
+ if _is_deck_metric_value(line_width)
+ and raw_width is not None
+ and not isinstance(raw_width, (int, float))
+ else None
+ )
+ for extra_metric in (width_metric, form_data.get("breakpoint_metric")):
+ if extra_metric is None:
+ continue
+ labels = {_deck_metric_label(item) for item in metrics}
+ if _deck_metric_label(extra_metric) not in labels:
+ metrics.append(extra_metric)
+ if line_column and line_column not in groupby:
+ groupby.append(line_column)
+ columns = _add_deck_columns(columns, tooltips)
+ groupby = _add_deck_columns(groupby, tooltips)
+ if not any(
+ filter_.get("col") == line_column and filter_.get("op") == "IS NOT NULL"
+ for filter_ in filters
+ if isinstance(filter_, dict)
+ ):
+ add_null(line_column)
+ query.update(
+ columns=columns,
+ metrics=metrics,
+ groupby=groupby,
Review Comment:
Fixed in 31bff3652ae76ff7db15a230ad8eeeb9563d2fdd.
The Path fallback now places line_column in canonical columns alongside dimensions/tooltips and removes the deprecated groupby key. test_deck_path_metric_keeps_canonical_path_grouping exercises the real QueryObjectFactory and SQL generation, asserting path selection and GROUP BY path with count/width/breakpoint metric controls. All three cases failed before the fix and pass after; the existing spatial query-contract expectation was updated.
Validation: MCP/common unit suites: 9,198 passed, 4 skipped; touched modules: 343 passed plus 351 Sunburst tests passed. Pre-commit passed on all 83 branch-changed files, including mypy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3944693734
##########
superset/mcp_service/chart/sunburst.py:
##########
@@ -0,0 +1,329 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Shared native Sunburst result-role resolution and validation."""
+
+import math
+from collections.abc import Mapping
+from dataclasses import dataclass
+from decimal import Decimal
+from typing import Any
+
+from superset.mcp_service.chart.schemas import ChartError
+from superset.mcp_service.common.error_schemas import DatasetContext
+
+
+@dataclass(frozen=True)
+class SunburstResultRoles:
+ """Resolved query-output fields required to render a Sunburst."""
+
+ hierarchy: tuple[str, ...]
+ primary_metric: str
+ secondary_metric: str | None = None
+
+
+def _column_result_label(column: Any) -> str | None:
+ """Resolve a bounded native column reference to its query output label."""
+ if isinstance(column, str) and column:
+ return column
+ if not isinstance(column, Mapping) or not 0 < len(column) <= 20:
+ return None
+ for key in ("label", "column_name", "columnName", "sqlExpression"):
+ value = column.get(key)
+ if isinstance(value, str) and value:
+ return value
+ if value not in (None, ""):
+ return None
+ return None
+
+
+def _metric_result_label(metric: Any) -> str | None:
+ """Resolve saved, SIMPLE, and SQL metric aliases like the frontend."""
+ if isinstance(metric, str) and metric:
+ return metric
+ if not isinstance(metric, Mapping) or not 0 < len(metric) <= 20:
+ return None
+
+ label = metric.get("label")
+ if isinstance(label, str) and label:
+ return label
+ if label not in (None, ""):
+ return None
+
+ expression_type = metric.get("expressionType")
+ if expression_type == "SIMPLE":
+ aggregate = metric.get("aggregate")
+ column = metric.get("column")
+ if (
+ not isinstance(aggregate, str)
+ or not aggregate
+ or len(aggregate) > 100
+ or not isinstance(column, Mapping)
+ or not 0 < len(column) <= 50
+ ):
+ return None
+ column_name = column.get("column_name") or column.get("columnName")
+ if not isinstance(column_name, str) or not column_name:
+ return None
+ return f"{aggregate}({column_name})"
+ if expression_type == "SQL":
+ sql_expression = metric.get("sqlExpression")
+ if (
+ isinstance(sql_expression, str)
+ and sql_expression
+ and len(sql_expression) <= 2000
+ ):
+ return sql_expression
+ return None
+
+
+def _metric_query_identity(metric: Any) -> tuple[str, ...] | None:
+ """Return a hook-free identity for one exact native metric value."""
+ if type(metric) is str and metric:
+ return ("saved", metric)
+ if type(metric) is not dict:
+ return None
+ expression_type = dict.get(metric, "expressionType")
+ if expression_type == "SQL":
+ expression = dict.get(metric, "sqlExpression")
+ return ("sql", expression) if type(expression) is str and expression else None
+ if expression_type == "SIMPLE":
+ aggregate = dict.get(metric, "aggregate")
+ column = dict.get(metric, "column")
+ if type(aggregate) is not str:
+ return None
+ column_name: Any
+ if type(column) is str:
+ column_name = column
+ elif type(column) is dict:
+ column_name = dict.get(column, "column_name")
+ if column_name is None:
+ column_name = dict.get(column, "columnName")
+ else:
+ return None
+ if type(column_name) is str and column_name:
+ return ("simple", aggregate, column_name)
+ return None
+
+
+def resolve_sunburst_result_roles(
+ form_data: Mapping[str, Any],
+) -> tuple[SunburstResultRoles | None, ChartError | None]:
+ """Resolve and validate all native Sunburst query-result roles."""
+ columns = form_data.get("columns")
+ if not isinstance(columns, list) or not columns:
+ return None, ChartError(
+ error="Sunburst form data requires one or more hierarchy columns.",
+ error_type="InvalidSunburstFormData",
+ )
+ hierarchy: list[str] = []
+ for index, column in enumerate(columns):
+ label = _column_result_label(column)
+ if label is None:
+ return None, ChartError(
+ error=f"Sunburst hierarchy column {index + 1} is malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ hierarchy.append(label)
+
+ primary = _metric_result_label(form_data.get("metric"))
+ if primary is None:
+ return None, ChartError(
+ error="Sunburst primary metric is missing or malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ primary_metric = form_data.get("metric")
+ secondary: str | None = None
+ if (secondary_metric := form_data.get("secondary_metric")) is not None:
+ secondary = _metric_result_label(secondary_metric)
+ if secondary is None:
+ return None, ChartError(
+ error="Sunburst secondary metric is malformed.",
+ error_type="InvalidSunburstFormData",
+ )
+ if (
+ secondary.casefold() == primary.casefold()
+ and _metric_query_identity(primary_metric)
+ == _metric_query_identity(secondary_metric)
+ and _metric_query_identity(primary_metric) is not None
+ ):
+ # The frontend interprets a repeated primary metric as categorical
+ # color mode and the query has one physical metric output.
+ secondary = None
+
+ labels = [*hierarchy, primary, *([secondary] if secondary else [])]
+ seen: dict[str, str] = {}
+ for label in labels:
+ folded = label.casefold()
+ if previous := seen.get(folded):
+ return None, ChartError(
+ error=(
+ f"Sunburst result label {label!r} is ambiguous with {previous!r}."
+ ),
+ error_type="InvalidSunburstFormData",
+ )
+ seen[folded] = label
+
+ return SunburstResultRoles(tuple(hierarchy), primary, secondary), None
+
+
+def _finite_number(value: Any) -> bool:
+ """Return whether a normalized database value is exactly numeric and finite."""
+ value_type = type(value)
+ if value_type is int:
+ return True
+ if value_type is float:
+ return math.isfinite(value)
+ if value_type is Decimal:
+ return Decimal.is_finite(value)
+ return False
+
+
+def _valid_hierarchy_value(value: Any) -> bool:
+ """Reject nested/container values that cannot form a stable node label."""
+ return not isinstance(value, (Mapping, list, tuple, set))
+
+
+def validate_sunburst_result_data(
+ data: Any, form_data: Mapping[str, Any]
+) -> tuple[SunburstResultRoles | None, ChartError | None]:
+ """Validate every Sunburst row and all resolved result aliases."""
+ roles, error = resolve_sunburst_result_roles(form_data)
+ if error is not None:
+ return None, error
+ assert roles is not None
+ if not isinstance(data, list):
+ return None, ChartError(
+ error="Sunburst query result data is not an array of rows.",
+ error_type="InvalidSunburstResult",
+ )
+
+ required_fields = [
+ *roles.hierarchy,
+ roles.primary_metric,
+ *([roles.secondary_metric] if roles.secondary_metric else []),
+ ]
+ for index, row in enumerate(data, start=1):
+ if not isinstance(row, Mapping):
+ return None, ChartError(
+ error=f"Sunburst result row {index} is not an object.",
+ error_type="InvalidSunburstResult",
+ )
+ missing = [field for field in required_fields if field not in row]
+ if missing:
+ return None, ChartError(
+ error=(
+ f"Sunburst result row {index} is missing required field(s): "
+ f"{', '.join(str(field) for field in missing)}."
+ ),
+ error_type="InvalidSunburstResult",
+ )
+ for field in roles.hierarchy:
+ if not _valid_hierarchy_value(row[field]):
+ return None, ChartError(
+ error=(
+ f"Sunburst result row {index} has a malformed hierarchy "
+ f"value for {field!r}."
+ ),
+ error_type="InvalidSunburstResult",
+ )
+ for field in (
+ roles.primary_metric,
+ *([roles.secondary_metric] if roles.secondary_metric else []),
+ ):
+ assert field is not None
+ if not _finite_number(row[field]):
Review Comment:
Addressed in bc8ac02f9a0dd3da8bdc3909079de1a99459efad. After generic producer normalization, the shared Sunburst validator converts NULL metric cells to zero, matching `Sunburst/transformProps.ts` โ `utils/tree.ts:getMetricValue`. This applies before compile/preview/data/export consumers; absent result columns, strings, booleans, and infinity remain invalid. NULL hierarchy cells are not changed.
Passing regressions: `test_compile_proves_numeric_metric_by_resolved_alias` covers real producer NULL/NaN output with saved/SIMPLE/SQL aliases; `test_saved_preview_is_sunburst_faithful_and_vega_is_unsupported` covers NULL saved previews; `test_null_sunburst_metrics_normalize_without_changing_null_hierarchy` covers either metric role; `test_decimal_sunburst_get_data_is_numeric_and_json_safe` exercises the real DataFrame/QueryContextProcessor/ChartDataCommand producer through registered FastMCP saved, unsaved, and cached-update JSON/CSV calls. Focused suite: **776 passed**. Final full-suite and exact-head CI verification are underway; no new review requested.
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -0,0 +1,1520 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+
+import math
+import time as system_time
+from bisect import bisect_right
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
+from decimal import Decimal
+from enum import Enum
+from typing import Any
+from uuid import UUID
+from zoneinfo import ZoneInfo
+
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
+
+FAILED_QUERY_STATUSES = frozenset(
+ {"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
+)
+
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
+ return None
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
+
+
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = list.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (list.__getitem__(item, index), item, index, depth + 1, False)
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ children: list[tuple[Any, dict[str, Any], str, int, bool]] = []
+ for key, child in dict.items(item):
+ if type(key) is not str:
+ return None, "a non-string object key"
+ if reason := _charge_text(key, budget, key=True, metadata=metadata):
+ return None, reason
+ children.append((child, item, key, depth + 1, False))
+ stack.append((item, None, None, depth, True))
+ stack.extend(reversed(children))
+ continue
+
+ source_item = item
+ if type(item) in enum_types:
+ normalized = _safe_enum_value(item, enum_types)
+ if normalized is None:
+ return None, "an unsupported enum"
+ item = normalized
+ elif any(base is Enum for base in _type_mro(type(item))):
+ return None, "an enum outside its expected metadata slot"
+
+ normalized, reason = _normalize_scalar(item)
+ if reason is not None:
+ return None, reason
+ max_string_bytes = (
+ MAX_QUERY_RESULT_METADATA_BYTES if metadata else MAX_RESULT_STRING_LENGTH
+ )
+ if type(normalized) is str:
+ scalar_size = _json_string_size(normalized, max_string_bytes)
+ if scalar_size is None:
+ return None, "invalid text data"
+ else:
+ scalar_size = _normalized_scalar_json_size(normalized)
+ if reason := _charge_json_bytes(
+ budget,
+ scalar_size,
+ metadata=metadata,
+ ):
+ return None, reason
+ if parent is None:
+ root = normalized
+ elif normalized is not source_item:
+ if type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized)
+ return root, None
+
+
+def _normalize_metadata_value(
+ payload: dict[str, Any], key: str, budget: _ResultBudget
+) -> str | None:
+ enum_slots: dict[str, frozenset[type[Any]]] = {
+ "status": frozenset({QueryStatus}),
+ "result_format": frozenset({ChartDataResultFormat}),
+ "coltypes": frozenset({GenericDataType}),
+ "applied_filters": frozenset({ExtraFiltersTimeColumnType}),
+ "rejected_filters": frozenset(
+ {ExtraFiltersReasonType, ExtraFiltersTimeColumnType}
+ ),
+ }
+ value = dict.__getitem__(payload, key)
+ normalized, reason = _normalize_value(
+ value,
+ budget,
+ enum_types=enum_slots.get(key, frozenset()),
+ metadata=True,
+ )
+ if reason is None and normalized is not value:
+ dict.__setitem__(payload, key, normalized)
+ return reason
+
+
+def _error_text(value: Any) -> str | None:
+ """Extract a bounded error from an already validated primitive tree."""
+ stack: list[Any] = [value]
+ parts: list[str] = []
+ used = 0
+ while stack and len(parts) < 3 and used < _MAX_ERROR_TEXT_BYTES:
+ item = stack.pop()
+ if type(item) is dict:
+ stack.extend(
+ reversed(
+ [dict.__getitem__(item, key) for key in _ERROR_KEYS if key in item]
+ )
+ )
+ continue
+ if type(item) is list:
+ stack.extend(
+ list.__getitem__(item, index)
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if item is None or item is False:
+ continue
+ if type(item) is str:
+ remaining = _MAX_ERROR_TEXT_BYTES - used - (2 if parts else 0)
+ encoded = str.encode(item, "utf-8")[:remaining]
+ text = bytes.decode(encoded, "utf-8", errors="ignore")
+ if text:
+ parts.append(text)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ return "; ".join(parts) or None
+
+
+def _failure_for_payload(payload: dict[str, Any], label: str) -> ChartError | None:
+ for key in ("error", "errors", "error_message"):
+ if key in payload and (message := _error_text(dict.__getitem__(payload, key))):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+
+ raw_status = dict.get(payload, "status")
+ status = raw_status if type(raw_status) is str else ""
+ normalized_status = status.strip().casefold().replace("-", "_").replace(" ", "_")
+ if normalized_status in FAILED_QUERY_STATUSES:
+ message = (
+ _error_text(dict.get(payload, "message"))
+ or _error_text(dict.get(payload, "error_message"))
+ or normalized_status
+ )
+ return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if dict.get(payload, "success") is False:
+ message = _error_text(dict.get(payload, "message")) or "request failed"
+ return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if raw_status is None and "data" not in payload and "queries" not in payload:
+ if message := _error_text(dict.get(payload, "message")):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+ return None
+
+
+def _metadata_shape_error( # noqa: C901
+ payload: dict[str, Any], label: str, budget: _ResultBudget
+) -> ChartError | None:
+ if "success" in payload and type(dict.__getitem__(payload, "success")) is not bool:
+ return _invalid_metadata(label)
+ if "status" in payload and type(dict.__getitem__(payload, "status")) is not str:
+ return _invalid_metadata(label)
+ if (
+ "result_format" in payload
+ and dict.__getitem__(payload, "result_format") not in _RESULT_FORMAT_VALUES
+ ):
+ return _invalid_metadata(label)
+ for count_key in ("rowcount", "sql_rowcount", "total_rows"):
+ if count_key in payload:
+ count = dict.__getitem__(payload, count_key)
+ if count is not None and not (
+ type(count) is int and 0 <= count <= MAX_QUERY_RESULT_ROWCOUNT
+ ):
+ return _invalid_metadata(label)
+ if "is_cached" in payload:
+ cached = dict.__getitem__(payload, "is_cached")
+ if cached is None:
+ if reason := _charge_json_bytes(
+ budget, 1, metadata=True
+ ): # ``false`` vs ``null``
+ return _invalid_result(reason)
+ dict.__setitem__(payload, "is_cached", False)
+ elif type(cached) is not bool:
+ return _invalid_metadata(label)
+ if "cache_timeout" in payload:
+ timeout = dict.__getitem__(payload, "cache_timeout")
+ if timeout is not None and not (
Review Comment:
Addressed in bc8ac02f9a0dd3da8bdc3909079de1a99459efad. The result-envelope validator imports and accepts Superset's exact integer `CACHE_DISABLED_TIMEOUT` sentinel, retaining the non-negative bounded range for ordinary timeouts. It preserves -1 without rewriting it and still rejects -2, boolean/float/string lookalikes, integer subclasses, and out-of-range values.
Passing regressions: `test_producer_cache_timeout_preserves_disabled_and_bounded_values`, `test_cache_timeout_rejects_noncanonical_sentinel_lookalikes`, plus producer-to-registered-FastMCP `test_decimal_sunburst_get_data_is_numeric_and_json_safe` (saved/unsaved/cached-update, JSON/CSV) and compile/saved-preview matrices at cache timeouts -1, 0, and 300. Focused suite: **776 passed**; staged and full branch-diff pre-commit/MyPy/Ruff/Pylint checks pass. Final full-suite and exact-head CI verification are underway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5500458547
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3906355906
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1044,6 +1045,109 @@ def map_pie_config(config: PieChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+ form_data["granularity_sqla"] = config.temporal_column
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields whose mapper defaults must not overwrite existing values when
+# a same-type update omitted the corresponding typed field. Required query roles
+# (hierarchy and metric) are deliberately absent: a full replacement always
+# updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "secondary_metric": "secondary_metric",
+ "filters": "adhoc_filters",
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+}
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Merge a same-viz update without resetting omitted Sunburst UI state.
+
+ Native keys that MCP does not model are retained, matching the established
+ preview-update behavior. For a same-type Sunburst update, mapper defaults
+ are replaced by saved values when the caller omitted that field; explicit
+ values (including ``False``, ``0``, ``None`` and ``[]``) win.
+ """
+ merged = {**existing_form_data, **new_form_data}
+ if not isinstance(config, SunburstChartConfig) or existing_form_data.get(
+ "viz_type"
+ ) != new_form_data.get("viz_type"):
+ return merged
+
+ fields_set = config.model_fields_set
+ for field_name, form_key in _SUNBURST_UPDATE_FIELD_KEYS.items():
+ if field_name not in fields_set:
+ if form_key in existing_form_data:
+ merged[form_key] = existing_form_data[form_key]
+ else:
+ merged.pop(form_key, None)
+ continue
+
+ value = getattr(config, field_name)
+ if value is None or (field_name == "filters" and value == []):
+ merged.pop(form_key, None)
Review Comment:
**Suggestion:** Clearing `time_range` or `temporal_column` removes only top-level controls; preserved `adhoc_filters` still apply the old `TEMPORAL_RANGE` filter during updates. [stale reference]
**Assessment:** ๐ `Major` ยท ๐ `Occurrence: Sometimes`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=e5f578d3c0f14a8e9ac729e4d34006be&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=e5f578d3c0f14a8e9ac729e4d34006be&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent ๐ค </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/mcp_service/chart/chart_utils.py
**Line:** 1139:1141
**Comment:**
*Stale Reference: Clearing `time_range` or `temporal_column` removes only top-level controls; preserved `adhoc_filters` still apply the old `TEMPORAL_RANGE` filter during updates.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F43771&comment_hash=56fd97886dcf6e7f12ddde42864b7d965a5819bd4846ae1fa76e25167a80345b&reaction=like'>๐</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F43771&comment_hash=56fd97886dcf6e7f12ddde42864b7d965a5819bd4846ae1fa76e25167a80345b&reaction=dislike'>๐</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5498232742
### <span aria-hidden="true">โ
</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">๐จ</span> Latest commit | 04f45d4cdf8c60b8263f4860903b03f9173d38ff |
|<span aria-hidden="true">๐</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6a97134c4c7efc000818cb00 |
|<span aria-hidden="true">๐</span> Deploy Preview | [https://deploy-preview-43771--superset-docs-preview.netlify.app](https://deploy-preview-43771--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">๐ฑ</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">๐ค</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DcoxDoAgDADAv3SWpFUC4g98hhSMg0IDOBjj32W74V6Qki9psAAM4MuW-OiurIgcmVFdLKreyd-ltj5ClDM_a-jHbM7SpFmzjTsj4kwze0T4fg) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-43771--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5736905190
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5739282291
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5729388656
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3972014690
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -15,82 +15,1521 @@
# specific language governing permissions and limitations
# under the License.
-"""Helpers for interpreting ChartDataCommand result envelopes."""
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
import math
+import time as system_time
+from bisect import bisect_right
from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
+from decimal import Decimal
+from enum import Enum
from typing import Any
+from uuid import UUID
+from zoneinfo import ZoneInfo
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
FAILED_QUERY_STATUSES = frozenset(
{"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
)
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
-def _query_error_text(value: Any) -> str | None:
- """Convert a bounded query error payload into a useful message."""
- if value is None or value is False:
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
return None
- if isinstance(value, Mapping):
- for key in ("error", "error_message", "message", "detail"):
- if text := _query_error_text(value.get(key)):
- return text
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
return None
- if isinstance(value, (list, tuple)):
- parts = [text for item in value if (text := _query_error_text(item))]
- return "; ".join(parts[:3]) or None
- text = str(value)
- return text[:2000] if text else None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
-def _failure_for_query_payload(
- payload: Mapping[str, Any], label: str
-) -> ChartError | None:
- """Extract one failure from a top-level or per-query payload."""
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if offset is None:
+ # Static pytz zones store their fixed offset on the generated class.
+ class_namespace = type.__getattribute__(type(tzinfo), "__dict__")
+ offset = class_namespace.get("_utcoffset")
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if tzinfo is pytz.UTC:
+ return timezone.utc
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
+ _canonical_timezone(tzinfo) is not None
+ or _pytz_named_offset_without_hooks(tzinfo) is not None
+ or type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE
+ or _dateutil_named_state_without_hooks(tzinfo) is not None
+ )
+ if not supported_timezone:
+ return None, "a timestamp with an unsupported timezone"
+ canonical_tz = _timestamp_offset_without_hooks(value)
+ if canonical_tz is None:
+ return None, "an invalid timestamp"
+ raw_value = value.asm8.view("i8")
+ value = pd.Timestamp(raw_value, unit=value.unit, tz="UTC").tz_convert(
+ canonical_tz
+ )
+ return pd.Timestamp.isoformat(value), None
+ except (KeyError, OverflowError, TypeError, ValueError):
+ return None, "an invalid timestamp"
+
+
+def _normalize_scalar(value: Any) -> tuple[Any, str | None]: # noqa: C901
+ """Convert one exact trusted producer scalar to a JSON-safe scalar."""
+ value_type = type(value)
+ if value is None or value_type is bool or value_type is str:
+ return value, None
+ if value_type is int:
+ return value, _integer_failure(value)
+ if value_type is float:
+ if math.isnan(value):
+ return None, None
+ return (value, None) if math.isfinite(value) else (None, "a non-finite number")
+ if value_type is Decimal:
+ return value, _decimal_failure(value)
+ if value_type is datetime:
+ return _canonical_datetime(value)
+ if value_type is time:
+ return _canonical_time(value)
+ if value_type is date:
+ return date.isoformat(value), None
+ if value_type is timedelta:
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid duration"
+ if value_type is UUID:
+ return UUID.__str__(value), None
+
+ if value_type is _PANDAS_NAT_TYPE or value_type is _PANDAS_NA_TYPE:
+ return None, None
+ if value_type is pd.Timestamp:
+ return _canonical_timestamp(value)
+ if value_type is pd.Timedelta:
+ if pd.isna(value):
+ return None, None
+ try:
+ return pd.Timedelta.isoformat(value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas duration"
+ if value_type is _PANDAS_PERIOD_TYPE or value_type is _PANDAS_INTERVAL_TYPE:
+ # These concrete immutable pandas extension scalars are trusted. Exact
+ # type checks deliberately exclude subclasses with conversion hooks.
+ try:
+ normalized_text = str(value)
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid pandas scalar"
+ if _bounded_utf8_length(normalized_text, MAX_RESULT_STRING_LENGTH) is None:
+ return None, "an oversized pandas scalar"
+ return normalized_text, None
+ if value_type in _NUMPY_INTEGER_TYPES:
+ normalized = int(value)
+ return normalized, _integer_failure(normalized)
+ if value_type in _NUMPY_FLOAT_TYPES:
+ normalized_float = float(value)
+ if math.isnan(normalized_float):
+ return None, None
+ return (
+ (normalized_float, None)
+ if math.isfinite(normalized_float)
+ else (None, "a non-finite NumPy number")
+ )
+ if value_type is np.bool_:
+ return bool(value), None
+ if value_type is np.str_:
+ return str(value), None
+ if value_type is np.datetime64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return _canonical_timestamp(pd.Timestamp(value))
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy timestamp"
+ if value_type is np.timedelta64:
+ if np.isnat(value):
+ return None, None
+ try:
+ return pd.Timedelta(value).isoformat(), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid NumPy duration"
+ return None, "an unsupported or subclassed value"
+
+
+def _normalize_value( # noqa: C901
+ value: Any,
+ budget: _ResultBudget,
+ *,
+ enum_types: frozenset[type[Any]] = frozenset(),
+ metadata: bool = False,
+) -> tuple[Any, str | None]:
+ """Iteratively normalize one bounded exact-container value tree."""
+ stack: list[
+ tuple[Any, list[Any] | dict[str, Any] | None, int | str | None, int, bool]
+ ] = [(value, None, None, 0, False)]
+ active_containers: set[int] = set()
+ root = value
+
+ while stack:
+ item, parent, slot, depth, leaving = stack.pop()
+ if leaving:
+ active_containers.remove(id(item))
+ continue
+ if depth > MAX_RESULT_VALUE_DEPTH:
+ return None, "excessively nested data"
+ if reason := _charge_value(budget, metadata=metadata):
+ return None, reason
+
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = list.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized array"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=False),
+ metadata=metadata,
+ ):
+ return None, reason
+ stack.append((item, None, None, depth, True))
+ stack.extend(
+ (list.__getitem__(item, index), item, index, depth + 1, False)
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ return None, "cyclic containers"
+ active_containers.add(identity)
+ width = dict.__len__(item)
+ if width > MAX_RESULT_VALUE_ITEMS:
+ return None, "an oversized object"
+ if reason := _charge_json_bytes(
+ budget,
+ _container_json_syntax_size(width, mapping=True),
+ metadata=metadata,
+ ):
+ return None, reason
+ children: list[tuple[Any, dict[str, Any], str, int, bool]] = []
+ for key, child in dict.items(item):
+ if type(key) is not str:
+ return None, "a non-string object key"
+ if reason := _charge_text(key, budget, key=True, metadata=metadata):
+ return None, reason
+ children.append((child, item, key, depth + 1, False))
+ stack.append((item, None, None, depth, True))
+ stack.extend(reversed(children))
+ continue
+
+ source_item = item
+ if type(item) in enum_types:
+ normalized = _safe_enum_value(item, enum_types)
+ if normalized is None:
+ return None, "an unsupported enum"
+ item = normalized
+ elif any(base is Enum for base in _type_mro(type(item))):
+ return None, "an enum outside its expected metadata slot"
+
+ normalized, reason = _normalize_scalar(item)
+ if reason is not None:
+ return None, reason
+ max_string_bytes = (
+ MAX_QUERY_RESULT_METADATA_BYTES if metadata else MAX_RESULT_STRING_LENGTH
+ )
+ if type(normalized) is str:
+ scalar_size = _json_string_size(normalized, max_string_bytes)
+ if scalar_size is None:
+ return None, "invalid text data"
+ else:
+ scalar_size = _normalized_scalar_json_size(normalized)
+ if reason := _charge_json_bytes(
+ budget,
+ scalar_size,
+ metadata=metadata,
+ ):
+ return None, reason
+ if parent is None:
+ root = normalized
+ elif normalized is not source_item:
+ if type(parent) is list:
+ assert type(slot) is int
+ list.__setitem__(parent, slot, normalized)
+ else:
+ assert type(parent) is dict
+ assert type(slot) is str
+ dict.__setitem__(parent, slot, normalized)
+ return root, None
+
+
+def _normalize_metadata_value(
+ payload: dict[str, Any], key: str, budget: _ResultBudget
+) -> str | None:
+ enum_slots: dict[str, frozenset[type[Any]]] = {
+ "status": frozenset({QueryStatus}),
+ "result_format": frozenset({ChartDataResultFormat}),
+ "coltypes": frozenset({GenericDataType}),
+ "applied_filters": frozenset({ExtraFiltersTimeColumnType}),
+ "rejected_filters": frozenset(
+ {ExtraFiltersReasonType, ExtraFiltersTimeColumnType}
+ ),
+ }
+ value = dict.__getitem__(payload, key)
+ normalized, reason = _normalize_value(
+ value,
+ budget,
+ enum_types=enum_slots.get(key, frozenset()),
+ metadata=True,
+ )
+ if reason is None and normalized is not value:
+ dict.__setitem__(payload, key, normalized)
+ return reason
+
+
+def _error_text(value: Any) -> str | None:
+ """Extract a bounded error from an already validated primitive tree."""
+ stack: list[Any] = [value]
+ parts: list[str] = []
+ used = 0
+ while stack and len(parts) < 3 and used < _MAX_ERROR_TEXT_BYTES:
+ item = stack.pop()
+ if type(item) is dict:
+ stack.extend(
+ reversed(
+ [dict.__getitem__(item, key) for key in _ERROR_KEYS if key in item]
+ )
+ )
+ continue
+ if type(item) is list:
+ stack.extend(
+ list.__getitem__(item, index)
+ for index in range(list.__len__(item) - 1, -1, -1)
+ )
+ continue
+ if item is None or item is False:
+ continue
+ if type(item) is str:
+ remaining = _MAX_ERROR_TEXT_BYTES - used - (2 if parts else 0)
+ encoded = str.encode(item, "utf-8")[:remaining]
+ text = bytes.decode(encoded, "utf-8", errors="ignore")
+ if text:
+ parts.append(text)
+ used += bytes.__len__(encoded) + (2 if len(parts) > 1 else 0)
+ return "; ".join(parts) or None
+
+
+def _failure_for_payload(payload: dict[str, Any], label: str) -> ChartError | None:
for key in ("error", "errors", "error_message"):
- if message := _query_error_text(payload.get(key)):
+ if key in payload and (message := _error_text(dict.__getitem__(payload, key))):
return ChartError(
error=f"{label} failed: {message}", error_type="QueryError"
)
- raw_status = payload.get("status")
- status = str(getattr(raw_status, "value", raw_status) or "")
+ raw_status = dict.get(payload, "status")
+ status = raw_status if type(raw_status) is str else ""
normalized_status = status.strip().casefold().replace("-", "_").replace(" ", "_")
if normalized_status in FAILED_QUERY_STATUSES:
message = (
- _query_error_text(payload.get("message"))
- or _query_error_text(payload.get("error_message"))
+ _error_text(dict.get(payload, "message"))
+ or _error_text(dict.get(payload, "error_message"))
or normalized_status
)
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
- if payload.get("success") is False:
- message = _query_error_text(payload.get("message")) or "request failed"
+ if dict.get(payload, "success") is False:
+ message = _error_text(dict.get(payload, "message")) or "request failed"
return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ if raw_status is None and "data" not in payload and "queries" not in payload:
+ if message := _error_text(dict.get(payload, "message")):
+ return ChartError(
+ error=f"{label} failed: {message}", error_type="QueryError"
+ )
+ return None
+
+
+def _metadata_shape_error( # noqa: C901
+ payload: dict[str, Any], label: str, budget: _ResultBudget
+) -> ChartError | None:
+ if "success" in payload and type(dict.__getitem__(payload, "success")) is not bool:
+ return _invalid_metadata(label)
+ if "status" in payload and type(dict.__getitem__(payload, "status")) is not str:
+ return _invalid_metadata(label)
if (
- raw_status is None
- and "data" not in payload
- and "queries" not in payload
- and (message := _query_error_text(payload.get("message")))
+ "result_format" in payload
+ and dict.__getitem__(payload, "result_format") not in _RESULT_FORMAT_VALUES
):
- return ChartError(error=f"{label} failed: {message}", error_type="QueryError")
+ return _invalid_metadata(label)
+ for count_key in ("rowcount", "sql_rowcount", "total_rows"):
+ if count_key in payload:
+ count = dict.__getitem__(payload, count_key)
+ if count is not None and not (
+ type(count) is int and 0 <= count <= MAX_QUERY_RESULT_ROWCOUNT
+ ):
+ return _invalid_metadata(label)
+ if "is_cached" in payload:
+ cached = dict.__getitem__(payload, "is_cached")
+ if cached is None:
+ if reason := _charge_json_bytes(
+ budget, 1, metadata=True
+ ): # ``false`` vs ``null``
+ return _invalid_result(reason)
+ dict.__setitem__(payload, "is_cached", False)
+ elif type(cached) is not bool:
+ return _invalid_metadata(label)
+ if "cache_timeout" in payload:
+ timeout = dict.__getitem__(payload, "cache_timeout")
+ if timeout is not None and not (
+ type(timeout) is int
+ and (
+ timeout == CACHE_DISABLED_TIMEOUT
+ or 0 <= timeout <= MAX_QUERY_RESULT_CACHE_TIMEOUT
+ )
+ ):
+ return _invalid_metadata(label)
+ if "cache_key" in payload:
+ cache_key = dict.__getitem__(payload, "cache_key")
+ if cache_key is not None and (type(cache_key) is not str or not cache_key):
+ return _invalid_metadata(label)
+ for timestamp_key in ("cached_dttm", "cache_dttm", "queried_dttm"):
+ if timestamp_key not in payload:
+ continue
+ timestamp = dict.__getitem__(payload, timestamp_key)
+ if timestamp is None:
+ continue
+ if (
+ type(timestamp) is not str
+ or not timestamp
+ or len(timestamp) > MAX_QUERY_RESULT_TIMESTAMP_LENGTH
+ ):
+ return _invalid_metadata(label)
+ normalized = f"{timestamp[:-1]}+00:00" if timestamp.endswith("Z") else timestamp
+ try:
+ parsed = datetime.fromisoformat(normalized)
+ except ValueError:
+ return _invalid_metadata(label)
+ if parsed.tzinfo is None or parsed.utcoffset() != timedelta(0):
Review Comment:
Addressed: naive cache timestamps are interpreted as UTC and canonicalized, while non-UTC aware timestamps remain rejected. The cache metadata regression now covers a naive `cache_dttm`; MyPy and Ruff pass.
##########
superset/mcp_service/chart/tool/get_chart_data.py:
##########
@@ -1115,56 +1224,22 @@ async def _query_from_form_data( # noqa: C901
error_type="NoData",
)
- columns = []
- for col_name in raw_columns:
- sample_values = [
- row.get(col_name) for row in data[:3] if row.get(col_name) is not None
- ]
- data_type = "string"
- if sample_values and all(
- isinstance(v, (int, float)) for v in sample_values
- ):
- data_type = "numeric"
- columns.append(
- DataColumn(
- name=col_name,
- display_name=col_name.replace("_", " ").title(),
- data_type=data_type,
- sample_values=sample_values[:3],
- null_count=sum(1 for row in data if row.get(col_name) is None),
- unique_count=len({str(row.get(col_name)) for row in data}),
- )
- )
+ columns = _build_data_columns(
+ data, raw_columns, query_result.get("coltypes", [])
+ )
cache_status = get_cache_status_from_result(
query_result, force_refresh=request.force_refresh
)
chart_name = form_data.get("slice_name", "Unsaved chart")
- if request.format in {"csv", "excel"}:
- from superset.models.slice import Slice
-
- # A transient chart supplies export metadata without saving anything.
- chart = Slice(id=0, slice_name=chart_name, viz_type=viz_type)
- export = (
- _export_data_as_csv
- if request.format == "csv"
- else _export_data_as_excel
- )
- return export(
- chart,
- data[: request.limit] if request.limit else data,
- raw_columns,
- cache_status,
- PerformanceMetadata(query_duration_ms=0, cache_status="fresh_query"),
- )
summary = (
f"Unsaved chart ({viz_type}). "
f"Contains {len(data)} rows across {len(raw_columns)} columns."
)
await ctx.report_progress(4, 4, "Building response")
- return ChartData(
+ response = ChartData(
Review Comment:
Addressed: the unsaved form-data path now branches to the existing CSV/Excel exporters before constructing JSON ChartData. The unsaved Decimal matrix now includes CSV and verifies the export payload; MyPy and Ruff pass.
##########
superset/mcp_service/app.py:
##########
@@ -395,8 +395,8 @@ def get_default_instructions(
- chart_type="table": Data table for detailed views
- chart_type="table", viz_type="ag-grid-table": Interactive AG Grid table
- chart_type="pie": Pie chart for proportional data (set donut=True for donut)
-- chart_type="gauge": Gauge/dial for one numeric metric, optionally grouped
- into up to 10 dials (native viz_type is "gauge_chart")
+- chart_type="sunburst": Hierarchical part-to-whole chart (hierarchy + metric
Review Comment:
Addressed: Gauge remains listed alongside Sunburst in the MCP capability catalog; its existing registry/display-name behavior is preserved. Gauge/Sunburst shared tests remain in the exact-head unit suite.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1249,6 +1336,659 @@ def map_gauge_config(config: GaugeChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def preserve_previous_adhoc_filters(
+ new_form_data: Dict[str, Any], previous_form_data: Mapping[str, Any]
+) -> None:
+ """Compatibility entry point backed by the shared filter merge."""
+ filters = _merge_preserved_adhoc_filters(
+ previous_form_data,
+ new_form_data,
+ drop_existing_temporal=False,
+ )
+ if filters is not None:
+ new_form_data["adhoc_filters"] = filters
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update( # noqa: C901
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+ """
+ if dataset_rebind:
+ existing_form_data = scrub_dataset_bound_form_data(existing_form_data)
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
Review Comment:
Addressed: `GaugeChartConfig` is included in modeled update controls, so omitted presentation fields preserve saved values instead of mapper defaults replacing them. `test_partial_gauge_update_preserves_omitted_controls` covers `show_progress=False` and `split_number=5`.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1249,6 +1336,659 @@ def map_gauge_config(config: GaugeChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.temporal_column is not None:
+ form_data["granularity_sqla"] = config.temporal_column
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+
+ _copy_sunburst_native_envelope(form_data, config)
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields with explicit omission/clear semantics. Mapper defaults must
+# not overwrite same-viz state when the typed field was omitted, while explicit
+# clears must also beat the shared preservation registry on cross-viz updates.
+# Required query roles (hierarchy and metric) are deliberately absent: a full
+# replacement always updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+ "extra_form_data": "extra_form_data",
+ "url_params": "url_params",
+ "standardized_form_data": "standardizedFormData",
+}
+
+
+# Presentation controls emitted sparsely by chart mappers need three-way update
+# semantics: omitted preserves saved native state, an explicit value replaces
+# it, and explicit ``None``/``False`` clears a truthy saved value when the mapper
+# has no canonical false/null representation. Query roles are intentionally
+# absent: a replacement config always owns those through the plugin contract.
+# Paths below also cover nested axis/legend models so an omitted nested property
+# is not mistaken for an explicit clear of the whole control.
+_MODELED_UPDATE_CONTROL_PATHS: dict[str, dict[str, tuple[tuple[str, ...], ...]]] = {
+ "PieChartConfig": {
+ "color_scheme": (("color_scheme",),),
+ "show_labels": (("show_labels",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "label_type": (("label_type",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ "sort_by_metric": (("sort_by_metric",),),
+ "row_limit": (("row_limit",),),
+ "donut": (("donut",),),
+ "show_total": (("show_total",),),
+ "labels_outside": (("labels_outside",),),
+ "outerRadius": (("outer_radius",),),
+ "innerRadius": (("inner_radius",),),
+ "currency_format": (("currency_format",),),
+ },
+ "TableChartConfig": {
+ "row_limit": (("row_limit",),),
+ "color_scheme": (("color_scheme",),),
+ "column_config": (("column_config",),),
+ },
+ "XYChartConfig": {
+ "row_limit": (("row_limit",),),
+ "series_limit": (("series_limit",),),
+ "stack": (("stacked",),),
+ "orientation": (("orientation",),),
+ "x_axis_title": (("x_axis", "title"),),
+ "x_axis_format": (("x_axis", "format"),),
+ "y_axis_title": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "y_axis_scale": (("y_axis", "scale"),),
+ "show_legend": (("legend", "show"),),
+ "legendOrientation": (("legend", "position"), ("legend_orientation",)),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "show_value": (("show_value",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ },
+ "HistogramChartConfig": {
+ "bins": (("bins",),),
+ "normalize": (("normalize",),),
+ "cumulative": (("cumulative",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BoxPlotChartConfig": {
+ "whiskerOptions": (
+ ("whisker_type",),
+ ("percentile_low",),
+ ("percentile_high",),
+ ),
+ "row_limit": (("row_limit",),),
+ "number_format": (("number_format",),),
+ "date_format": (("date_format",),),
+ },
+ "WaterfallChartConfig": {
+ "show_total": (("show_total",),),
+ "show_legend": (("show_legend",),),
+ "increase_label": (("increase_label",),),
+ "decrease_label": (("decrease_label",),),
+ "total_label": (("total_label",),),
+ "x_axis_time_format": (("x_axis_time_format",),),
+ "y_axis_format": (("y_axis_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "BigNumberChartConfig": {
+ "subheader": (("subheader",),),
+ "y_axis_format": (("y_axis_format",),),
+ "time_format": (("time_format",),),
+ "currency_format": (("currency_format",),),
+ "color_scheme": (("color_scheme",),),
+ "start_y_axis_at_zero": (("start_y_axis_at_zero",),),
+ "compare_lag": (("compare_lag",),),
+ "aggregation": (("aggregation",),),
+ },
+ "HandlebarsChartConfig": {
+ "row_limit": (("row_limit",),),
+ "order_desc": (("order_desc",),),
+ "styleTemplate": (("style_template",),),
+ },
+ "PivotTableChartConfig": {
+ "aggregateFunction": (("aggregate_function",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "transposePivot": (("transpose",),),
+ "combineMetric": (("combine_metric",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "row_limit": (("row_limit",),),
+ },
+ "InteractivePivotChartConfig": {
+ "order_desc": (("sort_descending",),),
+ "row_limit": (("row_limit",),),
+ "rowGroupCounts": (("show_row_group_counts",),),
+ "rowTotals": (("show_row_totals",),),
+ "colTotals": (("show_column_totals",),),
+ "colSubTotals": (("show_column_subtotals",),),
+ "valueFormat": (("value_format",),),
+ "date_format": (("date_format",),),
+ "currency_format": (("currency_format",),),
+ "colOrder": (("column_sort",),),
+ "allow_render_html": (("allow_render_html",),),
+ "expand_pivot_groups": (("expand_pivot_groups",),),
+ "time_compare": (("comparison_period",),),
+ "comparison_type": (("comparison_type",),),
+ },
+ "MixedTimeseriesChartConfig": {
+ "seriesType": (("primary_kind",),),
+ "area": (("primary_kind",),),
+ "seriesTypeB": (("secondary_kind",),),
+ "areaB": (("secondary_kind",),),
+ "show_legend": (("show_legend",),),
+ "legendOrientation": (("legend_orientation",),),
+ "show_value": (("show_value",),),
+ "color_scheme": (("color_scheme",),),
+ "currency_format": (("currency_format",),),
+ "currency_format_secondary": (("currency_format_secondary",),),
+ "xAxisTitle": (("x_axis", "title"),),
+ "x_axis_time_format": (("x_axis", "format"),),
+ "yAxisTitle": (("y_axis", "title"),),
+ "y_axis_format": (("y_axis", "format"),),
+ "logAxis": (("y_axis", "scale"),),
+ "yAxisTitleSecondary": (("y_axis_secondary", "title"),),
+ "y_axis_format_secondary": (("y_axis_secondary", "format"),),
+ "logAxisSecondary": (("y_axis_secondary", "scale"),),
+ "row_limit": (("row_limit",),),
+ },
+}
+
+
+def _model_path_was_set(config: Any, path: tuple[str, ...]) -> bool:
+ """Return whether every component of a Pydantic model path was supplied."""
+ current = config
+ for field_name in path:
+ if field_name not in getattr(current, "model_fields_set", set()):
+ return False
+ current = getattr(current, field_name, None)
+ if current is None:
+ # An explicit null parent clears all of its mapped descendants.
+ return True
+ return True
+
+
+def _apply_modeled_update_semantics(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> set[str]:
+ """Preserve truly omitted modeled controls and return explicit clears."""
+ explicit_clears: set[str] = set()
+ controls = _MODELED_UPDATE_CONTROL_PATHS.get(type(config).__name__, {})
+ for form_key, paths in controls.items():
+ if any(_model_path_was_set(config, path) for path in paths):
+ if form_key not in new_form_data:
+ explicit_clears.add(form_key)
+ continue
+ if form_key in existing_form_data:
+ new_form_data[form_key] = existing_form_data[form_key]
+ else:
+ new_form_data.pop(form_key, None)
+ return explicit_clears
+
+
+_TEMPORAL_FORM_DATA_KEYS = frozenset(
+ {
+ "granularity",
+ "granularity_sqla",
+ "since",
+ "time_grain",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+)
+
+
+def _is_temporal_filter(filter_: Any) -> bool:
+ """Return whether a native, adhoc, or legacy filter carries a time range."""
+ return isinstance(filter_, dict) and (
+ filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("op") == FilterOperator.TEMPORAL_RANGE.value
+ or filter_.get("col") in {"__time_col", "__time_grain", "__time_range"}
+ )
+
+
+def _without_temporal_filters(value: Any) -> Any:
+ """Copy a filter list without temporal predicates, preserving other shapes."""
+ if not isinstance(value, list):
+ return value
+ return [filter_ for filter_ in value if not _is_temporal_filter(filter_)]
+
+
+def _scrub_temporal_form_data(form_data: Mapping[str, Any]) -> Dict[str, Any]:
+ """Remove every source capable of reconstructing explicitly cleared time state."""
+ scrubbed = dict(form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ scrubbed.pop(key, None)
+ scrubbed.pop(MCP_DASHBOARD_TIME_FILTER_SUBJECT, None)
+
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in scrubbed:
+ scrubbed[key] = _without_temporal_filters(scrubbed[key])
+
+ extra_form_data = scrubbed.get("extra_form_data")
+ if isinstance(extra_form_data, dict):
+ cleaned_extra = dict(extra_form_data)
+ for key in _TEMPORAL_FORM_DATA_KEYS:
+ cleaned_extra.pop(key, None)
+ for key in ("adhoc_filters", "extra_filters", "filters"):
+ if key in cleaned_extra:
+ cleaned_extra[key] = _without_temporal_filters(cleaned_extra[key])
+ scrubbed["extra_form_data"] = cleaned_extra
+ elif extra_form_data is None:
+ scrubbed.pop("extra_form_data", None)
+ return scrubbed
+
+
+# One bounded registry owns state that may survive a form-data replacement.
+# Query roles and plugin-specific controls are deliberately absent. This keeps
+# cross-viz transitions preview/save-safe without chart-by-chart allowlists that
+# can drift as new plugins are registered.
+FORM_DATA_UPDATE_PRESERVE_KEYS: dict[str, frozenset[str]] = {
+ "envelope": frozenset(
+ {
+ "dashboardId",
+ "dashboards",
+ "datasource",
+ "extra_form_data",
+ "slice_id",
+ "slice_name",
+ "standardizedFormData",
+ "url_params",
+ }
+ ),
+ "presentation": frozenset(
+ {
+ "color_scheme",
+ "currency_format",
+ "date_format",
+ "legendOrientation",
+ "linear_color_scheme",
+ "number_format",
+ "show_legend",
+ }
+ ),
+ "filters": frozenset({"adhoc_filters", "extra_filters", "filters"}),
+ "time": frozenset(
+ {
+ "granularity_sqla",
+ "since",
+ "time_grain_sqla",
+ "time_range",
+ "until",
+ }
+ ),
+}
+_FORM_DATA_UPDATE_PRESERVE_KEYS = frozenset().union(
+ *FORM_DATA_UPDATE_PRESERVE_KEYS.values()
+)
+
+
+def _merge_preserved_adhoc_filters(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+ *,
+ drop_existing_temporal: bool,
+) -> list[Any] | None:
+ """Merge omitted structured filters while removing stale time bindings."""
+ previous = existing_form_data.get("adhoc_filters")
+ generated = new_form_data.get("adhoc_filters")
+ if not isinstance(previous, list):
+ return list(generated) if isinstance(generated, list) else None
+
+ previous_binding = existing_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ new_binding = new_form_data.get(MCP_DASHBOARD_TIME_FILTER_SUBJECT)
+ merged: list[Any] = []
+ for filter_ in previous:
+ is_temporal = (
+ isinstance(filter_, dict)
+ and filter_.get("operator") == FilterOperator.TEMPORAL_RANGE.value
+ )
+ stale_generated_binding = (
+ is_temporal
+ and previous_binding
+ and previous_binding != new_binding
+ and filter_.get("subject") == previous_binding
+ and filter_.get("comparator") == NO_TIME_RANGE
+ )
+ if (drop_existing_temporal and is_temporal) or stale_generated_binding:
+ continue
+ merged.append(filter_)
+
+ for filter_ in generated if isinstance(generated, list) else []:
+ if isinstance(filter_, dict):
+ same_filter = any(
+ isinstance(previous_filter, dict)
+ and previous_filter.get("clause") == filter_.get("clause")
+ and previous_filter.get("expressionType")
+ == filter_.get("expressionType")
+ and previous_filter.get("subject") == filter_.get("subject")
+ and previous_filter.get("operator") == filter_.get("operator")
+ for previous_filter in merged
+ )
+ if same_filter:
+ continue
+ elif filter_ in merged:
+ continue
+ merged.append(filter_)
+ return merged
+
+
+def preserve_previous_adhoc_filters(
+ new_form_data: Dict[str, Any], previous_form_data: Mapping[str, Any]
+) -> None:
+ """Compatibility entry point backed by the shared filter merge."""
+ filters = _merge_preserved_adhoc_filters(
+ previous_form_data,
+ new_form_data,
+ drop_existing_temporal=False,
+ )
+ if filters is not None:
+ new_form_data["adhoc_filters"] = filters
+
+
+def _merge_allowlisted_form_data(
+ existing_form_data: Mapping[str, Any],
+ new_form_data: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Start from mapped target state and add only registry-approved omissions."""
+ merged = dict(new_form_data)
+ for key in _FORM_DATA_UPDATE_PRESERVE_KEYS:
+ if key not in merged and key in existing_form_data:
+ merged[key] = existing_form_data[key]
+ return merged
+
+
+def merge_form_data_for_update( # noqa: C901
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+ *,
+ dataset_rebind: bool = False,
+) -> Dict[str, Any]:
+ """Merge mapped updates without leaking query roles across visualizations.
+
+ Same-viz updates retain native controls outside the simplified MCP schema by
+ starting from saved form data. Cross-viz updates remain bounded by the
+ shared preservation registry. Explicit clears are applied last.
+ """
+ if dataset_rebind:
+ existing_form_data = scrub_dataset_bound_form_data(existing_form_data)
+
+ same_viz = existing_form_data.get("viz_type") == new_form_data.get("viz_type")
+ explicit_control_clears = (
+ _apply_modeled_update_semantics(existing_form_data, new_form_data, config)
+ if same_viz
+ else set()
+ )
+ if same_viz:
+ from superset.mcp_service.chart.registry import (
+ query_role_keys_for_viz_type,
+ )
+
+ # Strip every target-owned query role first, then overlay the mapper's
+ # complete replacement. This removes mutually exclusive aliases (for
+ # example Pie ``metrics`` vs ``metric`` and raw vs aggregate table
+ # roles) without dropping unmodeled native presentation controls.
+ query_role_keys = query_role_keys_for_viz_type(
+ str(new_form_data.get("viz_type"))
+ )
+ merged = {
+ key: value
+ for key, value in existing_form_data.items()
+ if key not in query_role_keys
+ }
+ merged.update(new_form_data)
+ if new_form_data.get("viz_type") == "mixed_timeseries" and not dataset_rebind:
+ from superset.common.form_data_query_context import (
+ MIXED_TIMESERIES_SECONDARY_QUERY_KEYS,
+ )
+
+ # Query B inherits unsuffixed controls only when the suffixed key is
+ # absent. Preserve explicit native clears for controls the typed
+ # mapper did not replace, so []/None never turns into accidental
+ # inheritance from query A. Valid comparison state is also retained;
+ # malformed/stale dataset roles remain fail-closed and are dropped.
+ for key in MIXED_TIMESERIES_SECONDARY_QUERY_KEYS:
+ if key in new_form_data or key not in existing_form_data:
+ continue
+ value = existing_form_data[key]
+ is_explicit_clear = value is None or value in ([], {}, "")
+ is_valid_comparison = key == "comparison_type_b" and value in {
+ "values",
+ "difference",
+ "percentage",
+ "ratio",
+ }
+ is_valid_list_state = key in {
+ "adhoc_filters_b",
+ "annotation_layers_b",
+ "time_compare_b",
+ } and isinstance(value, list)
+ if is_explicit_clear or is_valid_comparison or is_valid_list_state:
+ merged[key] = value
+ else:
+ merged = _merge_allowlisted_form_data(existing_form_data, new_form_data)
+
+ for key in explicit_control_clears:
+ merged.pop(key, None)
+
+ fields_set: set[str] = getattr(config, "model_fields_set", set())
+ if getattr(config, "filters", None) == []:
+ merged.pop("adhoc_filters", None)
+ elif getattr(config, "filters", None) is None:
+ filters = _merge_preserved_adhoc_filters(
+ existing_form_data,
+ new_form_data,
+ drop_existing_temporal=bool(
+ {"temporal_column", "time_grain", "time_range"} & fields_set
+ ),
+ )
+ if filters is not None:
+ merged["adhoc_filters"] = filters
+
+ if not isinstance(config, SunburstChartConfig):
+ return merged
+
+ temporal_fields = {"time_grain", "temporal_column"}
+ for field_name, form_key in _SUNBURST_UPDATE_FIELD_KEYS.items():
+ if field_name in temporal_fields:
+ continue
+ if field_name not in fields_set:
+ if same_viz:
+ if form_key in existing_form_data:
+ merged[form_key] = existing_form_data[form_key]
+ else:
+ merged.pop(form_key, None)
+ continue
+
+ value = getattr(config, field_name)
+ if value is None:
+ merged.pop(form_key, None)
+ if field_name == "time_range":
+ # The generic query-context mapper reconstructs time_range from
+ # these legacy keys. A clear must remove all three sources.
+ merged.pop("since", None)
+ merged.pop("until", None)
+
+ # A null temporal control is an atomic clear. Apply it after every merge so
+ # cached/native aliases and extra-form-data overrides cannot recreate time
+ # state in QueryContextFactory.
+ explicit_temporal_clear = any(
+ field_name in fields_set and getattr(config, field_name) is None
+ for field_name in ("temporal_column", "time_grain", "time_range")
+ )
+ if explicit_temporal_clear:
+ merged = _scrub_temporal_form_data(merged)
Review Comment:
Addressed: after an atomic temporal clear, an explicitly supplied non-null `time_range` is reapplied. `test_explicit_range_survives_clearing_saved_grain` covers setting `Last month` while clearing the saved grain.
##########
superset/mcp_service/chart/chart_helpers.py:
##########
@@ -446,12 +564,200 @@ def _resolve_deck_gl_metrics(
if value:
metrics.append(value)
elif isinstance(prf, str) and _is_metric_ref(prf):
- # Legacy deck_scatter: point_radius_fixed as a bare non-numeric metric key
- logger.debug("Legacy point_radius_fixed string metric encountered: %s", prf)
metrics.append(prf)
return metrics
+def _deck_query_adapter( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any], viz_type: str
+) -> dict[str, Any]:
+ """Apply the native frontend builder for a single Deck.gl layer."""
+ base_columns = list(query.get("columns") or [])
+ base_metrics = list(query.get("metrics") or [])
+ filters = list(query.get("filters") or [])
+ tooltips = _deck_tooltip_columns(form_data.get("tooltip_contents"))
+
+ def add_null(column: str, *, value: Any = ...) -> None:
+ clause: dict[str, Any] = {"col": column, "op": "IS NOT NULL"}
+ if value is not ...:
+ clause["val"] = value
+ filters.append(clause)
+
+ if viz_type == "deck_geojson":
+ geometry = form_data.get("geojson")
+ if not isinstance(geometry, str) or not geometry:
+ raise ValueError("GeoJSON column is required for GeoJSON charts")
+ columns = _add_deck_columns(base_columns, [geometry] if geometry else [])
+ cross_filter = form_data.get("cross_filter_column")
+ if cross_filter:
+ columns = _add_deck_columns(columns, [cross_filter])
+ columns = _add_deck_columns(columns, tooltips)
+ if form_data.get("filter_nulls", True) and isinstance(geometry, str):
+ add_null(geometry)
+ query.update(
+ columns=columns,
+ metrics=[],
+ groupby=[],
+ filters=filters,
+ is_timeseries=False,
+ )
+ return query
+
+ if viz_type == "deck_polygon":
+ line_column = form_data.get("line_column")
+ if not isinstance(line_column, str) or not line_column:
+ raise ValueError("Polygon column is required for Polygon charts")
+ columns = _add_deck_columns(base_columns, [line_column] if line_column else [])
+ cross_filter = form_data.get("cross_filter_column")
+ if cross_filter:
+ columns = _add_deck_columns(columns, [cross_filter])
+ columns = _add_deck_columns(columns, tooltips)
+ metrics: list[Any] = []
+ if metric := form_data.get("metric"):
+ metrics.append(metric)
+ radius = form_data.get("point_radius_fixed")
+ if (
+ isinstance(radius, dict)
+ and radius.get("type") == "metric"
+ and radius.get("value") is not None
+ ):
+ metrics.append(radius["value"])
+ if form_data.get("filter_nulls", True) and isinstance(line_column, str):
+ add_null(line_column)
+ if metric:
+ add_null(_deck_metric_label(metric))
+ query.update(
+ columns=columns,
+ metrics=metrics,
+ filters=filters,
+ is_timeseries=False,
+ )
+ return query
+
+ if viz_type == "deck_path":
+ line_column = form_data.get("line_column")
+ if not isinstance(line_column, str) or not line_column:
+ raise ValueError("Line column is required for Path charts")
+ columns = list(base_columns)
+ metrics = list(base_metrics)
Review Comment:
Addressed: Deck Path filters `base_metrics` through the fixed-vs-metric discriminator and bases aggregate mode on the filtered metrics. The regression now asserts fixed `size="100"` yields no metrics and retains `path_col` as a raw column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3927150024
##########
superset/mcp_service/chart/schemas.py:
##########
@@ -1045,6 +1045,323 @@ def reject_sql_expression_on_dimensions(self) -> "PieChartConfig":
return self
+class SunburstChartConfig(BaseChartConfig):
+ """Config for the ECharts Sunburst plugin (viz_type ``sunburst_v2``).
+
+ ``hierarchy`` follows the frontend ``columns`` control: the first entry is
+ the innermost ring and each later entry adds a child level. The primary
+ metric sizes arcs; an optional secondary metric colors arcs by the
+ secondary/primary ratio.
+ """
+
+ model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+ chart_type: Literal["sunburst"] = "sunburst"
+ viz_type: Literal["sunburst_v2"] = Field(
+ "sunburst_v2",
+ description="Exact Superset frontend visualization tag",
+ )
+ hierarchy: List[ColumnRef] = Field(
+ ...,
+ min_length=1,
+ description=(
+ "Hierarchy dimensions in ring order, from the innermost/root level "
+ "to the outermost/leaf level"
+ ),
+ validation_alias=AliasChoices("hierarchy", "columns", "groupby"),
+ )
+ metric: ColumnRef = Field(
+ ...,
+ description=(
+ "Primary metric used to size arcs. Use aggregate for a SIMPLE "
+ "adhoc metric, saved_metric=True for a dataset metric, or "
+ "sql_expression plus label for a SQL metric."
+ ),
+ )
+ secondary_metric: ColumnRef | None = Field(
+ None,
+ description=(
+ "Optional metric used to color arcs by secondary/primary ratio. "
+ "When omitted, colors are categorical."
+ ),
+ validation_alias=AliasChoices("secondary_metric", "secondaryMetric"),
+ )
+ filters: List[FilterConfig] | None = Field(
+ None,
+ description=(
+ "Structured WHERE filters (column/op/value). An omitted list is "
+ "preserved on updates; an explicit [] clears saved filters."
+ ),
+ )
+ time_range: str | None = Field(
+ None,
+ min_length=1,
+ max_length=1000,
+ description=(
+ "Superset time range, for example 'Last year', "
+ "'2025-01-01 : 2025-12-31', or 'No filter'"
+ ),
+ )
+ time_grain: TimeGrain | None = Field(
+ None,
+ description="Optional bucket for temporal hierarchy columns",
+ validation_alias=AliasChoices("time_grain", "time_grain_sqla"),
+ )
+ sort_by_metric: bool = Field(
+ False,
+ description=(
+ "Order hierarchy rows by the primary metric descending before "
+ "applying row_limit, matching the frontend buildQuery transform"
+ ),
+ )
+ row_limit: int = Field(10000, description="Maximum hierarchy rows", ge=1, le=50000)
+ color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Categorical scheme used when secondary_metric is omitted",
+ )
+ linear_color_scheme: str | None = Field(
+ None,
+ max_length=100,
+ description="Sequential scheme used when secondary_metric is present",
+ )
+ show_labels: bool = False
+ show_labels_threshold: float = Field(
+ 5,
+ ge=0,
+ le=100,
+ description="Minimum arc size in percentage points for showing a label",
+ )
+ show_total: bool = False
+ show_null_values: bool = Field(
+ True,
+ description="Keep null-valued hierarchy nodes in the rendered tree",
+ )
+ label_type: Literal["key", "value", "key_value"] = "key"
+ number_format: str = Field("SMART_NUMBER", min_length=1, max_length=50)
+ date_format: str = Field("smart_date", min_length=1, max_length=50)
+ currency_format: CurrencyFormat | None = None
+
+ @staticmethod
+ def _looks_like_native_form_data(data: Any) -> bool:
+ """Identify saved Explore payloads without weakening typed typo checks."""
+ if not isinstance(data, dict) or data.get("viz_type") != "sunburst_v2":
+ return False
+ metric = data.get("metric")
+ return (
+ any(
+ key in data
+ for key in (
+ "adhoc_filters",
+ "annotation_layers",
+ "datasource",
+ "extra_form_data",
+ "since",
+ "slice_id",
+ "standardizedFormData",
+ "until",
+ )
+ )
+ or isinstance(metric, str)
+ or (isinstance(metric, dict) and "expressionType" in metric)
+ )
+
+ @staticmethod
+ def _coerce_native_metric(value: Any) -> Any:
+ """Accept saved metric names and native SIMPLE/SQL metric objects."""
+ if isinstance(value, str):
+ return {"name": value, "saved_metric": True}
+ if not isinstance(value, dict) or "expressionType" not in value:
+ return value
+
+ expression_type = value.get("expressionType")
+ label = value.get("label") if value.get("hasCustomLabel", True) else None
+ if expression_type == "SQL":
+ return {
+ "sql_expression": value.get("sqlExpression"),
+ "label": label,
+ }
Review Comment:
Addressed in the current head. `_coerce_native_metric` preserves a nonempty native label and falls back to `sqlExpression` when `hasCustomLabel=false`; `test_native_noncustom_sql_metric_uses_effective_frontend_label` verifies validation and mapped form_data retain the effective label and provenance. The full MCP suite passes at 5306ee42.
##########
superset/mcp_service/chart/tool/update_chart_preview.py:
##########
@@ -240,6 +245,11 @@ def update_chart_preview( # noqa: C901
if previous_form_data:
merge_table_column_config(previous_form_data, new_form_data)
merge_interactive_pivot_ui_config(previous_form_data, new_form_data)
+ new_form_data = merge_form_data_for_update(
+ previous_form_data, new_form_data, config
+ )
Review Comment:
Addressed in the current head. Cross-viz updates start from the mapped target and preserve only the bounded registry, so source query roles cannot survive; same-viz updates strip the target role vocabulary before overlay. `test_cross_viz_preview_and_compile_start_from_mapped_sunburst_roles` and `test_cross_viz_preview_update_and_immediate_save_report_sunburst_state` exercise divergent cached source roles through preview/compile/save.
##########
superset/mcp_service/chart/chart_utils.py:
##########
@@ -1044,6 +1045,109 @@ def map_pie_config(config: PieChartConfig) -> Dict[str, Any]:
return form_data
+def map_sunburst_config(config: SunburstChartConfig) -> Dict[str, Any]:
+ """Map typed Sunburst config to the ECharts ``sunburst_v2`` form_data.
+
+ The frontend control panel stores hierarchy levels under ``columns`` and
+ metrics under singular ``metric`` / ``secondary_metric`` keys. Its
+ buildQuery adds primary-metric descending ordering when ``sort_by_metric``
+ is enabled; server-side query builders mirror that transform separately.
+ """
+ form_data: Dict[str, Any] = {
+ "viz_type": "sunburst_v2",
+ "columns": [dimension.name for dimension in config.hierarchy],
+ "metric": create_metric_object(config.metric),
+ "sort_by_metric": config.sort_by_metric,
+ "row_limit": config.row_limit,
+ "show_labels": config.show_labels,
+ "show_labels_threshold": config.show_labels_threshold,
+ "show_total": config.show_total,
+ "show_null_values": config.show_null_values,
+ "label_type": config.label_type,
+ "number_format": config.number_format,
+ "date_format": config.date_format,
+ }
+ if config.secondary_metric is not None:
+ form_data["secondary_metric"] = create_metric_object(config.secondary_metric)
+ if config.color_scheme is not None:
+ form_data["color_scheme"] = config.color_scheme
+ if config.linear_color_scheme is not None:
+ form_data["linear_color_scheme"] = config.linear_color_scheme
+ if config.time_range is not None:
+ form_data["time_range"] = config.time_range
+ if config.time_grain is not None:
+ form_data["time_grain_sqla"] = config.time_grain
+ form_data["granularity_sqla"] = config.temporal_column
+
+ add_currency_format(form_data, config.currency_format)
+ _add_adhoc_filters(form_data, config.filters)
+ return form_data
+
+
+# Sunburst fields whose mapper defaults must not overwrite existing values when
+# a same-type update omitted the corresponding typed field. Required query roles
+# (hierarchy and metric) are deliberately absent: a full replacement always
+# updates them.
+_SUNBURST_UPDATE_FIELD_KEYS: dict[str, str] = {
+ "secondary_metric": "secondary_metric",
+ "filters": "adhoc_filters",
+ "time_range": "time_range",
+ "time_grain": "time_grain_sqla",
+ "temporal_column": "granularity_sqla",
+ "sort_by_metric": "sort_by_metric",
+ "row_limit": "row_limit",
+ "color_scheme": "color_scheme",
+ "linear_color_scheme": "linear_color_scheme",
+ "show_labels": "show_labels",
+ "show_labels_threshold": "show_labels_threshold",
+ "show_total": "show_total",
+ "show_null_values": "show_null_values",
+ "label_type": "label_type",
+ "number_format": "number_format",
+ "date_format": "date_format",
+ "currency_format": "currency_format",
+}
+
+
+def merge_form_data_for_update(
+ existing_form_data: Dict[str, Any],
+ new_form_data: Dict[str, Any],
+ config: Any,
+) -> Dict[str, Any]:
+ """Merge a same-viz update without resetting omitted Sunburst UI state.
+
+ Native keys that MCP does not model are retained, matching the established
+ preview-update behavior. For a same-type Sunburst update, mapper defaults
+ are replaced by saved values when the caller omitted that field; explicit
+ values (including ``False``, ``0``, ``None`` and ``[]``) win.
+ """
+ merged = {**existing_form_data, **new_form_data}
+ if not isinstance(config, SunburstChartConfig) or existing_form_data.get(
+ "viz_type"
+ ) != new_form_data.get("viz_type"):
+ return merged
+
+ fields_set = config.model_fields_set
+ for field_name, form_key in _SUNBURST_UPDATE_FIELD_KEYS.items():
+ if field_name not in fields_set:
+ if form_key in existing_form_data:
+ merged[form_key] = existing_form_data[form_key]
+ else:
+ merged.pop(form_key, None)
+ continue
+
+ value = getattr(config, field_name)
+ if value is None or (field_name == "filters" and value == []):
+ merged.pop(form_key, None)
Review Comment:
Addressed in the current head. Explicit time-range/column/grain changes invoke `_scrub_temporal_form_data`, removing temporal state and TEMPORAL_RANGE filters from top-level and `extra_form_data` reconstruction sources before merge. `test_explicit_temporal_clear_removes_preserved_temporal_filters`, `test_each_temporal_clear_scrubs_every_reconstruction_source`, and the native explicit-null test cover this behavior.
##########
superset/mcp_service/utils/response_utils.py:
##########
@@ -167,53 +171,208 @@ def build(self) -> Dict[str, str]:
STATS_ROW_CAP: int = 5000
+STATS_SAMPLE_VALUE_COUNT: int = 3
+STATS_TOTAL_WORK_CAP: int = 100_000
+
+_GENERIC_DATA_TYPE_NAMES: dict[int, str] = {
+ GenericDataType.NUMERIC: "numeric",
+ GenericDataType.STRING: "string",
+ GenericDataType.TEMPORAL: "temporal",
+ GenericDataType.BOOLEAN: "boolean",
+}
+_MAX_PROFILE_INTEGER_BITS = 4_096
+_MAX_PROFILE_STRING_LENGTH = 65_536
+
+
+@dataclass
+class _ColumnStatsBudget:
+ """One nested-node budget shared by all result columns."""
+
+ nodes: int = 0
+
+
+def data_column_stats_row_limit(row_count: int, column_count: int) -> int:
+ """Return a row sample whose aggregate top-level cell work is bounded."""
+ if row_count <= 0 or column_count <= 0:
+ return 0
+ return min(row_count, STATS_ROW_CAP, STATS_TOTAL_WORK_CAP // column_count)
+
+
+def _profile_value_identity( # noqa: C901
+ value: Any, budget: _ColumnStatsBudget
+) -> tuple[Any, ...] | None:
+ """Build a hook-free identity under the shared iterative node budget."""
+ tokens: list[Any] = []
+ stack: list[tuple[str, Any]] = [("value", value)]
+ active_containers: set[int] = set()
+ while stack:
+ action, item = stack.pop()
+ budget.nodes += 1
+ if budget.nodes > STATS_TOTAL_WORK_CAP:
+ return None
+ if action == "token":
+ tokens.append(item)
+ continue
+ if action == "leave":
+ active_containers.remove(id(item))
+ continue
+ if type(item) is list:
+ identity = id(item)
+ if identity in active_containers:
+ tokens.append(("cyclic_list",))
+ continue
+ active_containers.add(identity)
+ width = list.__len__(item)
+ tokens.append(("list", width))
+ stack.append(("leave", item))
+ stack.append(("token", "list_end"))
+ stack.extend(
+ ("value", list.__getitem__(item, index))
+ for index in range(width - 1, -1, -1)
+ )
+ continue
+ if type(item) is dict:
+ identity = id(item)
+ if identity in active_containers:
+ tokens.append(("cyclic_dict",))
+ continue
+ active_containers.add(identity)
+ entries = list(dict.items(item))
+ tokens.append(("dict", list.__len__(entries)))
+ stack.append(("leave", item))
+ stack.append(("token", "dict_end"))
+ for key, child in reversed(entries):
+ key_token = (
+ ("key", key)
+ if type(key) is str
+ and str.__len__(key) <= _MAX_PROFILE_STRING_LENGTH
+ else ("opaque_key", id(type(key)), id(key))
+ )
+ stack.append(("value", child))
+ stack.append(("token", key_token))
+ continue
+
+ value_type = type(item)
Review Comment:
Addressed and regression-tested at 5306ee42. Booleans use the distinct `("boolean", value)` identity while integers retain numeric rational identities; `test_boolean_and_integer_cardinality_remain_distinct` asserts `[True, 1, False, 0]` has `unique_count == 4`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3950138317
##########
superset/mcp_service/chart/preview_utils.py:
##########
@@ -160,11 +205,17 @@ def _generate_ascii_preview_from_data(
content = _generate_safe_ascii_line_chart(data)
elif viz_type == "pie":
content = _generate_safe_ascii_pie_chart(data)
+ elif viz_type == "sunburst_v2":
Review Comment:
Sunburst now enters the generic 20-line clamp after rendering two headers, up to 20 rows, and its `... more rows` footer, so a normal default-height preview silently loses rows and also drops the truncation notice. Could the Sunburst renderer reserve space for the footer (and mark width truncation) before returning the preview?
##########
superset/common/form_data_query_context.py:
##########
@@ -288,6 +461,1156 @@ def _pie_contribution_post_processing(metrics: list[Any]) -> list[dict[str, Any]
]
+def _as_list(value: Any) -> list[Any]:
+ """Return the frontend ``ensureIsArray`` representation of a value."""
+ if value is None:
+ return []
+ return list(value) if isinstance(value, (list, tuple)) else [value]
+
+
+def _label(value: Any, *, metric: bool = False) -> str:
+ """Resolve a frontend-compatible query-field label."""
+ try:
+ return get_metric_name(value) if metric else get_column_name(value)
+ except (AttributeError, KeyError, TypeError, ValueError):
+ if isinstance(value, Mapping):
+ return str(
+ value.get("label")
+ or value.get("column_name")
+ or value.get("sqlExpression")
+ or value
+ )
+ return str(value)
+
+
+def _deduplicate_fields(values: list[Any], *, metric: bool = False) -> list[Any]:
+ """Deduplicate query fields by their frontend-visible label."""
+ result: list[Any] = []
+ labels: set[str] = set()
+ for value in values:
+ if value is None or value == "":
+ continue
+ label = _label(value, metric=metric)
+ if label in labels:
+ continue
+ labels.add(label)
+ result.append(value)
+ return result
+
+
+def retain_mixed_timeseries_secondary_form_data(
+ form_data: Mapping[str, Any],
+) -> dict[str, Any]:
+ """Mirror ``retainFormDataSuffix(formData, '_b')`` exactly.
+
+ Suffixed values are installed first, including falsey values, and shared
+ unsuffixed controls fill only keys that query B did not explicitly set.
+ """
+ secondary: dict[str, Any] = {}
+ for key, value in form_data.items():
+ if key.endswith("_b"):
+ secondary[key[:-2]] = value
+ for key, value in form_data.items():
+ if not key.endswith("_b") and key not in secondary:
+ secondary[key] = value
+ secondary_filter_keys = {
+ "adhoc_filters": "adhoc_filters_b",
+ "extra_filters": "extra_filters_b",
+ "filters": "filters_b",
+ "having": "having_b",
+ "where": "where_b",
+ }
+ if any(suffixed in form_data for suffixed in secondary_filter_keys.values()):
+ # The frontend exposes adhoc_filters_b, while saved/server payloads can
+ # carry equivalent legacy aliases. Treat the family atomically: an
+ # explicit clear in any B alias must not be repopulated by query A's
+ # differently named filter representation.
+ for primary, suffixed in secondary_filter_keys.items():
+ if suffixed not in form_data:
+ secondary.pop(primary, None)
+ return secondary
+
+
+def _base_query_object( # noqa: C901
+ form_data: dict[str, Any],
+ *,
+ row_limit: int | None,
+ order_desc: bool | None,
+ filters_prepared: bool,
+) -> dict[str, Any]:
+ """Build the shared frontend-equivalent portion of a QueryObject."""
+ columns, metrics, orderby = query_fields_from_form_data(form_data)
+ query: dict[str, Any] = {
+ "columns": columns,
+ "metrics": metrics,
+ }
+ if orderby:
+ query["orderby"] = orderby
+
+ if filters_prepared:
+ query["filters"] = list(form_data.get("filters") or [])
+ for clause in ("where", "having"):
+ if form_data.get(clause):
+ query[clause] = form_data[clause]
+ if form_data.get("extras"):
+ query["extras"] = dict(form_data["extras"])
+ else:
+ filters = adhoc_filters_to_query_filters(
+ form_data.get("adhoc_filters", []), where_only=True
+ )
+ filters.extend(
+ filter_
+ for filter_ in form_data.get("filters") or []
+ if isinstance(filter_, dict) and filter_.get("col") is not None
+ )
+ query["filters"] = filters
+ if extras := freeform_where_having(form_data):
+ query["extras"] = extras
+
+ extras = dict(query.get("extras") or {})
+ if form_data.get("time_grain_sqla") is not None:
+ extras["time_grain_sqla"] = form_data["time_grain_sqla"]
+ if extras:
+ query["extras"] = extras
+
+ effective_limit = row_limit if row_limit is not None else form_data.get("row_limit")
+ if effective_limit is not None:
+ query["row_limit"] = effective_limit
+ if form_data.get("row_offset") is not None:
+ query["row_offset"] = form_data["row_offset"]
+ if order_desc is not None:
+ query["order_desc"] = order_desc
+ elif "order_desc" in form_data and form_data["order_desc"] is not None:
+ query["order_desc"] = form_data["order_desc"]
+
+ time_range = form_data.get("time_range")
+ if not time_range and (form_data.get("since") or form_data.get("until")):
+ time_range = f"{form_data.get('since') or ''} : {form_data.get('until') or ''}"
+ if time_range:
+ query["time_range"] = time_range
+ for key in ("since", "until", "annotation_layers", "url_params", "custom_params"):
+ if form_data.get(key) is not None:
+ query[key] = form_data[key]
+
+ granularity = form_data.get("granularity") or form_data.get("granularity_sqla")
+ if granularity:
+ query["granularity"] = granularity
+ series_limit = form_data.get("series_limit", form_data.get("limit"))
+ if series_limit is not None:
+ query["series_limit"] = series_limit
+ series_limit_metric = form_data.get("series_limit_metric")
+ if series_limit_metric is None:
+ series_limit_metric = form_data.get("timeseries_limit_metric")
+ if series_limit_metric is not None:
+ query["series_limit_metric"] = series_limit_metric
+ if form_data.get("group_others_when_limit_reached") is not None:
+ query["group_others_when_limit_reached"] = form_data[
+ "group_others_when_limit_reached"
+ ]
+ return query
+
+
+def _temporalized_columns(form_data: dict[str, Any], columns: list[Any]) -> list[Any]:
+ """Apply the pivot BASE_AXIS temporal-column contract."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ result: list[Any] = []
+ for column in columns:
+ if (
+ isinstance(column, str)
+ and time_grain
+ and (
+ temporal_lookup.get(column)
+ or form_data.get("granularity_sqla") == column
+ )
+ ):
+ result.append(
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ )
+ else:
+ result.append(column)
+ return result
+
+
+def _box_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Convert only physical columns confirmed temporal by Box Plot metadata."""
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup")
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+ return [
+ {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ if isinstance(column, str) and temporal_lookup.get(column) is True
+ else column
+ for column in columns
+ ]
+
+
+def _table_temporalized_columns(
+ form_data: dict[str, Any], columns: list[Any]
+) -> list[Any]:
+ """Promote the first temporal table group-by to the frontend BASE_AXIS.
+
+ Table's builder treats only physical columns named in
+ ``temporal_columns_lookup`` as temporal and moves the first match to the
+ front. Later temporal dimensions remain ordinary group-bys.
+ """
+ time_grain = form_data.get("time_grain_sqla")
+ temporal_lookup = form_data.get("temporal_columns_lookup") or {}
+ if not time_grain or not isinstance(temporal_lookup, Mapping):
+ return columns
+
+ temporal_column: dict[str, Any] | None = None
+ remaining: list[Any] = []
+ for column in columns:
+ if (
+ temporal_column is None
+ and isinstance(column, str)
+ and temporal_lookup.get(column)
+ ):
+ temporal_column = {
+ "timeGrain": time_grain,
+ "columnType": "BASE_AXIS",
+ "sqlExpression": column,
+ "label": column,
+ "expressionType": "SQL",
+ }
+ else:
+ remaining.append(column)
+ return [temporal_column, *remaining] if temporal_column else columns
+
+
+def _histogram_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ column = form_data.get("column")
+ query["columns"] = [*groupby, *([column] if column is not None else [])]
+ query["post_processing"] = [
+ {
+ "operation": "histogram",
+ "options": {
+ "column": _label(column),
+ "groupby": [_label(value) for value in groupby],
+ "bins": int(form_data.get("bins", 5)),
+ "cumulative": form_data.get("cumulative", False),
+ "normalize": form_data.get("normalize", False),
+ },
+ }
+ ]
+ if any(
+ isinstance(filter_, dict) and filter_.get("clause") == "HAVING"
+ for filter_ in form_data.get("adhoc_filters") or []
+ ):
+ query["metrics"] = [
+ {
+ "expressionType": "SQL",
+ "sqlExpression": "COUNT(*)",
+ "label": "COUNT(*)",
+ }
+ ]
+ else:
+ query["metrics"] = []
+
+
+def _box_plot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ distributed = _as_list(form_data.get("columns"))
+ if not distributed and form_data.get("granularity_sqla"):
+ distributed = [form_data["granularity_sqla"]]
+ groupby = _as_list(form_data.get("groupby"))
+ query["columns"] = [*_box_temporalized_columns(form_data, distributed), *groupby]
+ query["series_columns"] = groupby
+ whisker = form_data.get("whiskerOptions")
+ if not whisker:
+ query["post_processing"] = []
+ return
+ whisker_type = "tukey"
+ percentiles: list[int] | None = None
+ if whisker == "Min/max (no outliers)":
+ whisker_type = "min/max"
+ elif isinstance(whisker, str) and whisker.endswith(" percentiles"):
+ low, high = whisker.removesuffix(" percentiles").split("/", 1)
+ whisker_type = "percentile"
+ percentiles = [int(low), int(high)]
+ query["post_processing"] = [
+ {
+ "operation": "boxplot",
+ "options": {
+ "whisker_type": whisker_type,
+ "percentiles": percentiles,
+ "groupby": [_label(value) for value in groupby],
+ "metrics": [_label(value, metric=True) for value in query["metrics"]],
+ },
+ }
+ ]
+
+
+_PIVOT_ADDITIVE_AGGREGATES = frozenset({"SUM", "COUNT", "MIN", "MAX"})
+
+
+def _all_metrics_additive(metrics: list[Any]) -> bool:
+ """Mirror Pivot's conservative additive-metric fast-path."""
+ return bool(metrics) and all(
+ isinstance(metric, Mapping)
+ and metric.get("expressionType") == "SIMPLE"
+ and metric.get("aggregate") in _PIVOT_ADDITIVE_AGGREGATES
+ for metric in metrics
+ )
+
+
+def _pivot_grouping_sets(
+ form_data: dict[str, Any], rows: list[Any], columns: list[Any]
+) -> list[list[str]]:
+ """Enumerate the rollup levels requested by Pivot's frontend builder."""
+ row_prefixes = [[], *(rows[: index + 1] for index in range(len(rows)))]
+ column_prefixes = [
+ [],
+ *(columns[: index + 1] for index in range(len(columns))),
+ ]
+ show_values_as = form_data.get("showValuesAs")
+ needs_rows_collapsed = show_values_as in {"percent_col", "percent_total"}
+ needs_columns_collapsed = show_values_as in {"percent_row", "percent_total"}
+
+ def row_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(rows):
+ return True
+ if not prefix:
+ return bool(form_data.get("colTotals")) or needs_rows_collapsed
+ return bool(form_data.get("rowSubTotals"))
+
+ def column_prefix_needed(prefix: list[Any]) -> bool:
+ if len(prefix) == len(columns):
+ return True
+ if not prefix:
+ return bool(form_data.get("rowTotals")) or needs_columns_collapsed
+ return bool(form_data.get("colSubTotals"))
+
+ levels = [
+ (row_prefix, column_prefix)
+ for row_prefix in row_prefixes
+ if row_prefix_needed(row_prefix)
+ for column_prefix in column_prefixes
+ if column_prefix_needed(column_prefix)
+ ]
+ if form_data.get("combineMetric"):
+ metrics_layout = form_data.get("metricsLayout")
+
+ def forced_denominator(level: tuple[list[Any], list[Any]]) -> bool:
+ row_prefix, column_prefix = level
+ return (needs_rows_collapsed and not row_prefix) or (
+ needs_columns_collapsed and not column_prefix
+ )
+
+ if metrics_layout == "ROWS":
+ levels = [
+ level
+ for level in levels
+ if len(level[0]) == len(rows) or forced_denominator(level)
+ ]
+ else:
+ levels = [
+ level
+ for level in levels
+ if len(level[1]) == len(columns) or forced_denominator(level)
+ ]
+
+ return [
+ [_label(value) for value in _deduplicate_fields([*row_prefix, *column_prefix])]
+ for row_prefix, column_prefix in levels
+ ]
+
+
+def _pivot_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ rows = _as_list(form_data.get("groupbyRows"))
+ columns = _as_list(form_data.get("groupbyColumns"))
+ if form_data.get("transposePivot"):
+ rows, columns = columns, rows
+ query["columns"] = _temporalized_columns(
+ form_data, _deduplicate_fields([*rows, *columns])
+ )
+ metric = query.get("series_limit_metric") or next(
+ iter(query.get("metrics") or []), None
+ )
+ query["orderby"] = (
+ [[metric, not bool(query.get("order_desc", True))]]
+ if metric is not None
+ else []
+ )
+ if not _all_metrics_additive(query.get("metrics") or []):
+ query["grouping_sets"] = _pivot_grouping_sets(form_data, rows, columns)
+
+
+def _waterfall_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ x_axis = form_data.get("x_axis") or form_data.get("granularity_sqla")
+ columns = [*_as_list(x_axis), *_as_list(form_data.get("groupby"))]
+ query["columns"] = _deduplicate_fields(columns)
+ query["orderby"] = [[column, True] for column in query["columns"]]
+
+
+def _gantt_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("series"))
+ orderby = query_fields_from_form_data(form_data)[2]
+ columns = [
+ form_data.get("start_time"),
+ form_data.get("end_time"),
+ form_data.get("y_axis"),
+ *groupby,
+ *_as_list(form_data.get("tooltip_columns")),
+ *(entry[0] for entry in orderby if entry),
+ ]
+ query["columns"] = _deduplicate_fields(columns)
+ query["metrics"] = _as_list(form_data.get("tooltip_metrics"))
+ query["orderby"] = orderby
+ query["series_columns"] = groupby
+
+
+def _normalize_query_orderby(query: dict[str, Any]) -> None:
+ """Mirror ``normalizeOrderBy`` while retaining limit-direction controls."""
+ orderby = query.get("orderby")
+ if (
+ isinstance(orderby, list)
+ and orderby
+ and isinstance(orderby[0], (list, tuple))
+ and len(orderby[0]) == 2
+ and orderby[0][0]
+ and isinstance(orderby[0][1], bool)
+ ):
+ return
+ metric = (
+ query.get("series_limit_metric")
+ or query.get("legacy_order_by")
+ or next(iter(query.get("metrics") or []), None)
+ )
+ if metric is None:
+ query.pop("orderby", None)
+ return
+ query["orderby"] = [[metric, not bool(query.get("order_desc", True))]]
+
+
+_TIME_COMPARISON_TYPES = frozenset({"values", "difference", "percentage", "ratio"})
+
+
+def _metric_offset_map(
+ form_data: dict[str, Any],
+ metric_labels: list[str],
+ offsets: list[Any] | None = None,
+) -> dict[str, str]:
+ """Return the frontend time-comparison metric label map."""
+ if form_data.get("comparison_type") not in _TIME_COMPARISON_TYPES:
+ return {}
+ return {
+ f"{metric}__{offset}": metric
+ for metric in metric_labels
+ for offset in (
+ offsets if offsets is not None else _as_list(form_data.get("time_compare"))
+ )
+ }
+
+
+def _table_time_offsets(form_data: dict[str, Any]) -> list[Any]:
+ """Resolve Table custom/inherited shifts like its frontend query adapter."""
+ raw_offsets = _as_list(form_data.get("time_compare"))
+ offsets = [offset for offset in raw_offsets if offset not in {"custom", "inherit"}]
+ if "custom" in raw_offsets and form_data.get("start_date_offset") is not None:
+ offsets.append(form_data["start_date_offset"])
+ extra_form_data = form_data.get("extra_form_data")
+ if isinstance(extra_form_data, Mapping) and extra_form_data.get("time_compare"):
+ inherited = extra_form_data["time_compare"]
+ if inherited not in offsets:
+ offsets = [inherited]
+ return offsets
+
+
+def _x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Return a supported x-axis column, excluding legacy granularity.
+
+ ``column_name`` mappings are retained for old server/native payloads. Big
+ Number uses the stricter frontend predicate below.
+ """
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if isinstance(x_axis, Mapping):
+ if isinstance(column_name := x_axis.get("column_name"), str) and column_name:
+ return column_name
+ # Frontend SQL adhoc columns remain objects in the QueryObject.
+ return x_axis if x_axis else None
+ return None
+
+
+def _frontend_x_axis_column(form_data: Mapping[str, Any]) -> Any | None:
+ """Mirror ``isQueryFormColumn`` for physical and SQL adhoc columns."""
+ x_axis = form_data.get("x_axis")
+ if isinstance(x_axis, str):
+ return x_axis if x_axis else None
+ if (
+ isinstance(x_axis, Mapping)
+ and "sqlExpression" in x_axis
+ and "label" in x_axis
+ and x_axis.get("expressionType") in {None, "SQL"}
+ ):
+ return x_axis
+ return None
+
+
+def normalize_time_column(
+ form_data: Mapping[str, Any], query: dict[str, Any]
+) -> dict[str, Any]:
+ """Apply the final shared frontend ``normalizeTimeColumn`` mutator."""
+ x_axis = _frontend_x_axis_column(form_data)
+ columns = query.get("columns")
+ if x_axis is None or not isinstance(columns, list):
+ return query
+
+ axis_index: int | None = None
+ for index, column in enumerate(columns):
+ if isinstance(x_axis, str) and isinstance(column, str) and column == x_axis:
+ axis_index = index
+ break
+ if (
+ isinstance(x_axis, Mapping)
+ and isinstance(column, Mapping)
+ and column.get("sqlExpression") == x_axis.get("sqlExpression")
+ ):
+ axis_index = index
+ break
+ if axis_index is None:
+ return query
+
+ normalized = dict(query)
+ normalized_columns = list(columns)
+ grain = (query.get("extras") or {}).get("time_grain_sqla")
+ if isinstance(columns[axis_index], Mapping):
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ **({"timeGrain": grain} if grain is not None else {}),
+ **columns[axis_index],
+ }
+ else:
+ normalized_axis = {
+ "columnType": "BASE_AXIS",
+ "sqlExpression": x_axis,
+ "label": x_axis,
+ "expressionType": "SQL",
+ "isColumnReference": True,
+ **({"timeGrain": grain} if grain is not None else {}),
+ }
+ normalized_columns[axis_index] = normalized_axis
+ normalized["columns"] = normalized_columns
+ normalized.pop("is_timeseries", None)
+ return normalized
+
+
+def _finalize_query_objects(
+ form_data: Mapping[str, Any], queries: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """Run shared query-context mutators after every visualization adapter."""
+ return [normalize_time_column(form_data, query) for query in queries]
+
+
+def _x_axis_label(
+ form_data: Mapping[str, Any], *, frontend_strict: bool = False
+) -> str | None:
+ """Mirror getXAxisColumn/getXAxisLabel for explicit and legacy axes."""
+ explicit = (
+ _frontend_x_axis_column(form_data)
+ if frontend_strict
+ else _x_axis_column(form_data)
+ )
+ if explicit:
+ return _label(explicit)
+ if form_data.get("granularity_sqla"):
+ return DTTM_ALIAS
+ return None
+
+
+def _rename_operator(
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+) -> dict[str, Any] | None:
+ """Mirror the ECharts ``renameOperator`` for Timeseries and Mixed charts."""
+ metrics = list(query.get("metrics") or [])
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ series_columns = query.get("series_columns")
+ columns = _as_list(
+ series_columns if series_columns is not None else query.get("columns")
+ )
+ time_offsets = _as_list(form_data.get("time_compare"))
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ is_time_comparison = bool(offset_map)
+ truncate_metric = form_data.get("truncate_metric")
+
+ should_rename = (
+ bool(metrics)
+ and bool(x_axis_label)
+ and (
+ is_time_comparison
+ or (
+ (bool(columns) or len(time_offsets) > 1)
+ and "truncate_metric" in form_data
+ and bool(truncate_metric)
+ )
+ )
+ )
+ if not should_rename:
+ return None
+
+ renamed: dict[str, str | None] = {}
+ comparison_type = form_data.get("comparison_type")
+ if is_time_comparison:
+ for metric_with_offset, metric_only in offset_map.items():
+ offset_label = next(
+ (
+ str(offset)
+ for offset in time_offsets
+ if metric_with_offset.endswith(f"__{offset}")
+ ),
+ None,
+ )
+ source = (
+ metric_with_offset
+ if comparison_type == "values"
+ else f"{comparison_type}__{metric_only}__{metric_with_offset}"
+ )
+ renamed[source] = (
+ f"{metric_only}, {offset_label}" if len(metrics) > 1 else offset_label
+ )
+
+ if (
+ comparison_type not in {"difference", "percentage", "ratio"}
+ and len(metrics) == 1
+ and not renamed
+ ):
+ renamed[metric_labels[0]] = None
+ if not renamed:
+ return None
+ return {
+ "operation": "rename",
+ "options": {"columns": renamed, "level": 0, "inplace": True},
+ }
+
+
+def _timeseries_post_processing( # noqa: C901
+ form_data: dict[str, Any],
+ query: dict[str, Any],
+ *,
+ x_axis_label: str | None,
+ groupby: list[Any],
+ mixed: bool,
+) -> tuple[list[dict[str, Any]], list[Any]]:
+ """Build the Timeseries/Mixed operator pipeline in frontend order."""
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ offset_map = _metric_offset_map(form_data, metric_labels)
+ time_offsets = _as_list(form_data.get("time_compare")) if offset_map else []
+ post_processing: list[dict[str, Any]] = []
+
+ if x_axis_label and metric_labels:
+ aggregate_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [_label(value) for value in groupby],
+ "aggregates": {
+ label: {"operator": "mean"} for label in aggregate_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+
+ rolling_labels = (
+ [*offset_map.values(), *offset_map] if offset_map else metric_labels
+ )
+ columns_map = {label: label for label in rolling_labels}
+ rolling_type = form_data.get("rolling_type")
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {
+ "operation": "cum",
+ "options": {"operator": "sum", "columns": columns_map},
+ }
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+
+ comparison_type = form_data.get("comparison_type")
+ if offset_map and comparison_type != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": comparison_type,
+ "drop_original_columns": True,
+ },
+ }
+ )
+
+ if not mixed and form_data.get("contributionMode"):
+ post_processing.append(
+ {
+ "operation": "contribution",
+ "options": {
+ "orientation": form_data["contributionMode"],
+ "time_shifts": time_offsets,
+ },
+ }
+ )
+
+ if rename := _rename_operator(form_data, query, x_axis_label=x_axis_label):
+ post_processing.append(rename)
+
+ if not mixed:
+ sortable = {
+ x_axis_label or "",
+ *metric_labels,
+ }
+ if (
+ "x_axis_sort" in form_data
+ and "x_axis_sort_asc" in form_data
+ and form_data.get("x_axis_sort") in sortable
+ and not groupby
+ ):
+ options: dict[str, Any] = {"ascending": form_data.get("x_axis_sort_asc")}
+ if form_data.get("x_axis_sort") == x_axis_label:
+ options["is_sort_index"] = True
+ else:
+ options["by"] = form_data.get("x_axis_sort")
+ post_processing.append({"operation": "sort", "options": options})
+
+ post_processing.append({"operation": "flatten"})
+ if not mixed and form_data.get("forecastEnabled") and x_axis_label:
+ post_processing.append(
+ {
+ "operation": "prophet",
+ "options": {
+ "time_grain": form_data.get("time_grain_sqla"),
+ "periods": int(form_data.get("forecastPeriods") or 0),
+ "confidence_interval": float(
+ form_data.get("forecastInterval") or 0
+ ),
+ "yearly_seasonality": form_data.get("forecastSeasonalityYearly"),
+ "weekly_seasonality": form_data.get("forecastSeasonalityWeekly"),
+ "daily_seasonality": form_data.get("forecastSeasonalityDaily"),
+ "index": x_axis_label,
+ },
+ }
+ )
+ return post_processing, time_offsets
+
+
+def _timeseries_query(form_data: dict[str, Any], query: dict[str, Any]) -> None:
+ groupby = _as_list(form_data.get("groupby"))
+ x_axis = _x_axis_column(form_data)
+ x_axis_label = _x_axis_label(form_data)
+ query["columns"] = _deduplicate_fields([*_as_list(x_axis), *groupby])
+ query["series_columns"] = groupby
+ if not x_axis:
+ query["is_timeseries"] = True
+
+ # Timeseries includes its sort-only metric in the SELECT when no series is
+ # present. This lets the post-processing sort operator use a metric not
+ # otherwise displayed.
+ sort_metric = form_data.get("timeseries_limit_metric")
+ if isinstance(sort_metric, list):
+ sort_metric = next(iter(sort_metric), None)
+ if (
+ not groupby
+ and sort_metric is not None
+ and _label(sort_metric, metric=True) == form_data.get("x_axis_sort")
+ and _label(sort_metric, metric=True)
+ not in {_label(metric, metric=True) for metric in query.get("metrics") or []}
+ ):
+ query.setdefault("metrics", []).append(sort_metric)
+ _normalize_query_orderby(query)
+ post_processing, time_offsets = _timeseries_post_processing(
+ form_data,
+ query,
+ x_axis_label=x_axis_label,
+ groupby=groupby,
+ mixed=form_data.get("viz_type") == "mixed_timeseries",
+ )
+ query["post_processing"] = post_processing
+ query["time_offsets"] = time_offsets
+ if form_data.get("viz_type") != "mixed_timeseries":
+ query["time_compare_full_range"] = bool(
+ time_offsets and form_data.get("time_compare_full_range")
+ )
+
+
+def _big_number_queries(
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ """Mirror Big Number with Trendline's one/two-query contract."""
+ explicit_x_axis = _frontend_x_axis_column(form_data)
+ time_column = _as_list(explicit_x_axis)
+ x_axis_label = _x_axis_label(form_data, frontend_strict=True)
+ query["columns"] = time_column
+ if not time_column:
+ query["is_timeseries"] = True
+ metric_labels = [_label(value, metric=True) for value in query.get("metrics") or []]
+ post_processing: list[dict[str, Any]] = []
+ if x_axis_label and metric_labels:
+ post_processing.append(
+ {
+ "operation": "pivot",
+ "options": {
+ "index": [x_axis_label],
+ "columns": [],
+ "aggregates": {
+ label: {"operator": "mean"} for label in metric_labels
+ },
+ "drop_missing_columns": not form_data.get(
+ "show_empty_columns", False
+ ),
+ },
+ }
+ )
+ if form_data.get("resample_method") and form_data.get("resample_rule"):
+ zero_fill = form_data["resample_method"] == "zerofill"
+ post_processing.append(
+ {
+ "operation": "resample",
+ "options": {
+ "method": "asfreq" if zero_fill else form_data["resample_method"],
+ "rule": form_data["resample_rule"],
+ "fill_value": 0 if zero_fill else None,
+ },
+ }
+ )
+ rolling_type = form_data.get("rolling_type")
+ columns_map = {label: label for label in metric_labels}
+ if rolling_type == "cumsum":
+ post_processing.append(
+ {"operation": "cum", "options": {"operator": "sum", "columns": columns_map}}
+ )
+ elif rolling_type in {"sum", "mean", "std"}:
+ post_processing.append(
+ {
+ "operation": "rolling",
+ "options": {
+ "rolling_type": rolling_type,
+ "window": int(form_data.get("rolling_periods") or 1),
+ "min_periods": int(form_data.get("min_periods") or 0),
+ "columns": columns_map,
+ },
+ }
+ )
+ post_processing.append({"operation": "flatten"})
+ query["post_processing"] = post_processing
+ queries = [query]
+ if form_data.get("aggregation") == "raw":
+ overall = dict(query)
+ overall.update(
+ {
+ "columns": [],
+ "is_timeseries": False,
+ "post_processing": [],
+ }
+ )
+ queries.append(overall)
+ return queries
+
+
+def _table_queries( # noqa: C901
+ form_data: dict[str, Any], query: dict[str, Any]
+) -> list[dict[str, Any]]:
+ if is_raw_query_mode(form_data):
+ # The extractor already applies the raw-mode contract, including native
+ # ``order_by_cols`` parsing. Do not synthesize metric ordering.
+ query["columns"] = list(
+ form_data.get("all_columns") or form_data.get("columns") or []
+ )
+ query["metrics"] = []
+ if raw_orderby := orderby_from_form_data(form_data, [], "table"):
+ query["orderby"] = raw_orderby
+ else:
+ query.pop("orderby", None)
+ return [query]
+
+ metrics = list(query.get("metrics") or [])
+ query["columns"] = _table_temporalized_columns(
+ form_data, list(query.get("columns") or [])
+ )
+ percent_metrics = _as_list(form_data.get("percent_metrics"))
+ for metric in percent_metrics:
+ if _label(metric, metric=True) not in {
+ _label(existing, metric=True) for existing in metrics
+ }:
+ metrics.append(metric)
+ query["metrics"] = metrics
+ query["orderby"] = orderby_from_form_data(form_data, metrics, "table")
+ post_processing: list[dict[str, Any]] = []
+ resolved_offsets = _table_time_offsets(form_data)
+ comparison_enabled = (
+ form_data.get("comparison_type") in _TIME_COMPARISON_TYPES
+ and bool(metrics)
+ and bool(_as_list(form_data.get("time_compare")))
+ )
+ contribution: dict[str, Any] | None = None
+ if percent_metrics:
+ base_labels = [_label(metric, metric=True) for metric in percent_metrics]
+ labels = [
+ label
+ for metric_label in base_labels
+ for label in (
+ [
+ metric_label,
+ *(f"{metric_label}__{offset}" for offset in resolved_offsets),
+ ]
+ if comparison_enabled
+ else [metric_label]
+ )
+ ]
+ labels = list(dict.fromkeys(labels))
+ contribution = {
+ "operation": "contribution",
+ "options": {
+ "columns": labels,
+ "rename_columns": [f"%{label}" for label in labels],
+ },
+ }
+ post_processing.append(contribution)
+
+ metric_labels = [_label(metric, metric=True) for metric in metrics]
+ offset_map = _metric_offset_map(form_data, metric_labels, resolved_offsets)
+ time_offsets = resolved_offsets if offset_map else []
+ if offset_map and form_data.get("comparison_type") != "values":
+ post_processing.append(
+ {
+ "operation": "compare",
+ "options": {
+ "source_columns": list(offset_map.values()),
+ "compare_columns": list(offset_map),
+ "compare_type": form_data.get("comparison_type"),
+ "drop_original_columns": True,
+ },
+ }
+ )
+ if post_processing:
+ query["post_processing"] = post_processing
+ else:
+ query.pop("post_processing", None)
+ query["time_offsets"] = time_offsets
+
+ is_download = form_data.get("result_format") in {"csv", "xlsx"} or (
+ form_data.get("result_format") == "json"
+ and form_data.get("result_type") == "results"
+ )
+ if is_download:
+ if form_data.get("row_limit") is not None:
+ query["row_limit"] = int(form_data["row_limit"])
+ query["row_offset"] = 0
+ elif form_data.get("server_pagination"):
Review Comment:
The Excel-export rebuild does not set a download result format, so a saved Table with `server_pagination` now produces only the first page plus an `is_rowcount` query; the exporter writes each query as a sheet and silently exports 10 rows instead of the configured full row limit. Could the export path set download semantics or reject server-pagination rebuilds so it cannot truncate the workbook?
##########
superset/mcp_service/chart/query_result.py:
##########
@@ -0,0 +1,1525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Canonicalize and validate ``ChartDataCommand`` result envelopes."""
+
+import math
+import time as system_time
+from bisect import bisect_right
+from dataclasses import dataclass
+from datetime import date, datetime, time, timedelta, timezone
+from decimal import Decimal
+from enum import Enum
+from typing import Any
+from uuid import UUID
+from zoneinfo import ZoneInfo
+
+import numpy as np
+import pandas as pd
+import pytz
+from dateutil import tz as dateutil_tz, zoneinfo as dateutil_zoneinfo
+from pydantic import BaseModel
+from pydantic_core import to_json
+
+from superset.common.chart_data import ChartDataResultFormat
+from superset.common.db_query_status import QueryStatus
+from superset.constants import CACHE_DISABLED_TIMEOUT
+from superset.mcp_service.chart.schemas import ChartError
+from superset.utils.core import (
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+)
+
+FAILED_QUERY_STATUSES = frozenset(
+ {"error", "failed", "stopped", "timed_out", "cancelled", "canceled"}
+)
+
+# These are aggregate envelope limits, not per-query allowances. In particular,
+# splitting a result across the maximum number of queries must not multiply the
+# permitted rows, nodes, or encoded bytes.
+MAX_QUERY_RESULTS = 32
+MAX_QUERY_RESULT_ROWS_PER_QUERY = 50_000
+MAX_QUERY_RESULT_ROWS = 100_000
+MAX_QUERY_RESULT_COLUMNS = 4_096
+MAX_QUERY_RESULT_VALUES = 2_500_000
+MAX_QUERY_RESULT_VALUE_BYTES = 16 * 1024 * 1024
+MAX_QUERY_RESULT_METADATA_BYTES = 1024 * 1024
+MAX_QUERY_RESULT_METADATA_ITEMS = 32_768
+MAX_RESULT_VALUE_ITEMS = 4_096
+MAX_RESULT_VALUE_DEPTH = 32
+MAX_RESULT_STRING_LENGTH = 65_536
+MAX_RESULT_KEY_LENGTH = 4_096
+MAX_RESULT_INTEGER_BITS = 4_096
+MAX_RESULT_INTEGER_DIGITS = 1_234
+MAX_RESULT_DECIMAL_DIGITS = 1_024
+MAX_RESULT_DECIMAL_MAGNITUDE = 4_096
+MAX_RESULT_DECIMAL_STORAGE = 2_048
+MAX_QUERY_RESULT_ROWCOUNT = 2**63 - 1
+MAX_QUERY_RESULT_CACHE_TIMEOUT = 2**31 - 1
+MAX_QUERY_RESULT_TIMESTAMP_LENGTH = 64
+
+_ERROR_KEYS = ("error", "errors", "error_message", "message", "detail")
+_MAX_ERROR_TEXT_BYTES = 2_000
+_TRUSTED_TIMEZONE_TYPES = (timezone, ZoneInfo)
+_SAFE_RESULT_ENUM_TYPES = frozenset(
+ {
+ ChartDataResultFormat,
+ QueryStatus,
+ ExtraFiltersReasonType,
+ ExtraFiltersTimeColumnType,
+ GenericDataType,
+ }
+)
+_RESULT_FORMAT_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in ChartDataResultFormat
+)
+_COLTYPE_VALUES = frozenset(
+ object.__getattribute__(member, "_value_") for member in GenericDataType
+)
+_NUMPY_INTEGER_TYPES = frozenset(
+ type(value)
+ for value in (
+ np.int8(0),
+ np.int16(0),
+ np.int32(0),
+ np.int64(0),
+ np.uint8(0),
+ np.uint16(0),
+ np.uint32(0),
+ np.uint64(0),
+ )
+)
+_NUMPY_FLOAT_TYPES = frozenset(
+ type(value)
+ for value in (np.float16(0), np.float32(0), np.float64(0), np.longdouble(0))
+)
+_PANDAS_NAT_TYPE = type(pd.NaT)
+_PANDAS_NA_TYPE = type(pd.NA)
+_PANDAS_PERIOD_TYPE = type(pd.Period("2000-01", freq="M"))
+_PANDAS_INTERVAL_TYPE = type(pd.Interval(0, 1))
+_DATEUTIL_FIXED_TIMEZONE_TYPES = frozenset(
+ {type(dateutil_tz.tzoffset(None, 0)), type(dateutil_tz.tzutc())}
+)
+_DATEUTIL_NAMED_TIMEZONE_TYPES = frozenset(
+ {dateutil_tz.tzfile, dateutil_zoneinfo.tzfile}
+)
+_DATEUTIL_TTINFO_TYPE = type(
+ object.__getattribute__(dateutil_tz.gettz("UTC"), "__dict__")["_ttinfo_std"]
+)
+_DATEUTIL_LOCAL_TIMEZONE_TYPE = type(dateutil_tz.tzlocal())
+_PYTZ_FIXED_TIMEZONE_TYPES = frozenset({type(pytz.FixedOffset(1))})
+_MAX_DATEUTIL_TRANSITIONS = 4_096
+_MAX_DATEUTIL_TTINFOS = 256
+_MAX_DATEUTIL_TRANSITION_MAGNITUDE = 10**12
+
+
+@dataclass
+class _ResultBudget:
+ """Aggregate counters shared by every query and metadata value."""
+
+ rows: int = 0
+ values: int = 0
+ json_bytes: int = 0
+ metadata_items: int = 0
+ metadata_bytes: int = 0
+
+
+@dataclass(frozen=True)
+class _DateutilTimezoneState:
+ """Hook-free subset of a validated exact dateutil tzfile transition table."""
+
+ transitions: tuple[int, ...]
+ transition_offsets: tuple[int, ...]
+ standard_offset: int
+ before_offset: int | None
+
+
+def _invalid_result(message: str) -> ChartError:
+ return ChartError(
+ error=f"Chart query returned {message}.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _invalid_metadata(label: str) -> ChartError:
+ return ChartError(
+ error=f"{label} returned hostile or malformed metadata.",
+ error_type="InvalidQueryResult",
+ )
+
+
+def _safe_enum_value(value: Any, expected: frozenset[type[Any]]) -> Any | None:
+ """Read trusted enum storage without invoking public conversion hooks."""
+ if type(value) not in expected or type(value) not in _SAFE_RESULT_ENUM_TYPES:
+ return None
+ return object.__getattribute__(value, "_value_")
+
+
+def _bounded_utf8_length(value: str, maximum: int) -> int | None:
+ """Return the exact UTF-8 size while bounding pre-encoding work."""
+ if str.__len__(value) > maximum:
+ return None
+ try:
+ encoded = str.encode(value, "utf-8", errors="strict")
+ except UnicodeEncodeError:
+ return None
+ size = bytes.__len__(encoded)
+ return size if size <= maximum else None
+
+
+def _json_string_size(value: str, maximum: int) -> int | None:
+ """Return compact UTF-8 JSON string size without serializing the value."""
+ raw_size = _bounded_utf8_length(value, maximum)
+ if raw_size is None:
+ return None
+ escaped_size = raw_size + 2
+ for character in value:
+ codepoint = ord(character)
+ if character in {'"', "\\", "\b", "\t", "\n", "\f", "\r"}:
+ escaped_size += 1
+ elif codepoint < 0x20:
+ escaped_size += 5
+ return escaped_size
+
+
+def _integer_json_size(value: int) -> int:
+ """Return exact decimal JSON size without rendering the bounded integer."""
+ magnitude = -value if value < 0 else value
+ if magnitude == 0:
+ digits = 1
+ else:
+ bits = int.bit_length(magnitude)
+ digits = ((bits - 1) * 30103) // 100000 + 1
+ if magnitude >= 10**digits:
+ digits += 1
+ return digits + (value < 0)
+
+
+def _container_json_syntax_size(item_count: int, *, mapping: bool) -> int:
+ """Return braces/brackets plus compact separators and mapping colons."""
+ if item_count == 0:
+ return 2
+ return 2 + item_count - 1 + (item_count if mapping else 0)
+
+
+def _normalized_scalar_json_size(value: Any) -> int:
+ """Return exact compact JSON size for a normalized scalar."""
+ value_type = type(value)
+ if value is None:
+ return 4
+ if value_type is bool:
+ return 4 if value else 5
+ if value_type is str:
+ size = _json_string_size(value, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ if value_type is int:
+ return _integer_json_size(value)
+ if value_type is float:
+ return len(float.__repr__(value))
+ if value_type is Decimal:
+ # Pydantic serializes Decimal values as JSON strings so their exact
+ # finite value survives the wire projection without binary rounding.
+ text = Decimal.__str__(value)
+ size = _json_string_size(text, MAX_RESULT_STRING_LENGTH)
+ assert size is not None
+ return size
+ raise AssertionError("result scalar was not normalized")
+
+
+def _pydantic_scalar_json_size(value: Any) -> int:
+ """Return the scalar size emitted by Pydantic's JSON serializer."""
+ if type(value) is float:
+ # pydantic-core uses the shortest exponent (``1e-7``), while Python's
+ # repr retains a leading zero (``1e-07``).
+ return len(to_json(value))
+ return _normalized_scalar_json_size(value)
+
+
+def _charge_json_bytes(
+ budget: _ResultBudget, size: int, *, metadata: bool = False
+) -> str | None:
+ budget.json_bytes += size
+ if budget.json_bytes > MAX_QUERY_RESULT_VALUE_BYTES:
+ return "too many aggregate JSON bytes"
+ if metadata:
+ budget.metadata_bytes += size
+ if budget.metadata_bytes > MAX_QUERY_RESULT_METADATA_BYTES:
+ return "too many aggregate metadata JSON bytes"
+ return None
+
+
+def _charge_value(budget: _ResultBudget, *, metadata: bool = False) -> str | None:
+ budget.values += 1
+ if budget.values > MAX_QUERY_RESULT_VALUES:
+ return "too many aggregate values"
+ if metadata:
+ budget.metadata_items += 1
+ if budget.metadata_items > MAX_QUERY_RESULT_METADATA_ITEMS:
+ return "too many aggregate metadata values"
+ return None
+
+
+def _charge_text(
+ value: str,
+ budget: _ResultBudget,
+ *,
+ key: bool = False,
+ metadata: bool = False,
+) -> str | None:
+ maximum = (
+ MAX_RESULT_KEY_LENGTH
+ if key
+ else MAX_QUERY_RESULT_METADATA_BYTES
+ if metadata
+ else MAX_RESULT_STRING_LENGTH
+ )
+ size = _json_string_size(value, maximum)
+ if size is None:
+ return "an invalid or oversized object key" if key else "invalid text data"
+ return _charge_json_bytes(budget, size, metadata=metadata)
+
+
+def _integer_failure(value: int) -> str | None:
+ bits = int.bit_length(value)
+ if bits > MAX_RESULT_INTEGER_BITS:
+ return "an oversized integer"
+ digits = 1 if bits == 0 else ((bits - 1) * 30103) // 100000 + 1
+ if digits > MAX_RESULT_INTEGER_DIGITS:
+ return "an oversized integer"
+ return None
+
+
+def _decimal_failure(value: Decimal) -> str | None:
+ if Decimal.__sizeof__(value) > MAX_RESULT_DECIMAL_STORAGE:
+ return "an oversized Decimal"
+ if not Decimal.is_finite(value):
+ return "a non-finite Decimal"
+ parts = Decimal.as_tuple(value)
+ if tuple.__len__(parts.digits) > MAX_RESULT_DECIMAL_DIGITS:
+ return "an oversized Decimal"
+ exponent = parts.exponent
+ if type(exponent) is not int or abs(exponent) > MAX_RESULT_DECIMAL_MAGNITUDE:
+ return "an oversized Decimal"
+ return None
+
+
+def _type_mro(value_type: type[Any]) -> tuple[type[Any], ...]:
+ """Read a concrete type's MRO without consulting metaclass overrides."""
+ try:
+ mro = type.__getattribute__(value_type, "__mro__")
+ except (AttributeError, TypeError): # pragma: no cover - defensive metaclass
+ return ()
+ return mro if type(mro) is tuple else ()
+
+
+def _timezone_name_without_hooks(tzinfo: Any) -> str | None: # noqa: C901
+ """Read common pytz/dateutil zone state without dispatching timezone hooks."""
+ value_mro = _type_mro(type(tzinfo))
+ if any(base is pytz.tzinfo.BaseTzInfo for base in value_mro):
+ for base in value_mro:
+ try:
+ namespace = type.__getattribute__(base, "__dict__")
+ except (AttributeError, TypeError): # pragma: no cover
+ continue
+ zone = namespace.get("zone")
+ if type(zone) is str and _bounded_utf8_length(zone, 256) is not None:
+ try:
+ canonical = pytz.timezone(zone)
+ except (KeyError, ValueError):
+ return None
+ # Generated pytz types are trusted; arbitrary subclasses that
+ # inherit their internal fields are not.
+ return zone if type(canonical) is type(tzinfo) else None
+
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ filename = dict.get(namespace, "_filename")
+ if type(filename) is not str or _bounded_utf8_length(filename, 4_096) is None:
+ return None
+ marker = "/zoneinfo/"
+ if (offset := str.find(filename, marker)) >= 0:
+ name = str.__getitem__(filename, slice(offset + len(marker), None))
+ elif not str.startswith(filename, "/") and str.find(filename, "\\") < 0:
+ name = filename
+ else:
+ return None
+ parts = str.split(name, "/")
+ if not parts or any(part in {"", ".", ".."} for part in parts):
+ return None
+ return name if _bounded_utf8_length(name, 256) is not None else None
+
+
+def _object_namespace(value: Any) -> dict[str, Any] | None:
+ """Read exact instance storage without descriptor dispatch."""
+ try:
+ namespace = object.__getattribute__(value, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ return namespace if type(namespace) is dict else None
+
+
+def _dateutil_ttinfo_offset_without_hooks(value: Any) -> int | None:
+ """Validate one exact dateutil transition record and return its offset."""
+ if type(value) is not _DATEUTIL_TTINFO_TYPE:
+ return None
+ try:
+ offset = object.__getattribute__(value, "offset")
+ delta = object.__getattribute__(value, "delta")
+ isdst = object.__getattribute__(value, "isdst")
+ abbreviation = object.__getattribute__(value, "abbr")
+ is_standard = object.__getattribute__(value, "isstd")
+ is_gmt = object.__getattribute__(value, "isgmt")
+ dst_offset = object.__getattribute__(value, "dstoffset")
+ except (AttributeError, TypeError):
+ return None
+ if type(offset) is not int or not -86_400 < offset < 86_400:
+ return None
+ if type(delta) is not timedelta or delta != timedelta(seconds=offset):
+ return None
+ if type(isdst) is not int or isdst not in {0, 1}:
+ return None
+ if abbreviation is not None and (
+ type(abbreviation) is not str or _bounded_utf8_length(abbreviation, 256) is None
+ ):
+ return None
+ if type(is_standard) is not bool or type(is_gmt) is not bool:
+ return None
+ if type(dst_offset) is not timedelta:
+ return None
+ if not -timedelta(days=1) < dst_offset < timedelta(days=1):
+ return None
+ return offset
+
+
+def _dateutil_named_state_without_hooks( # noqa: C901
+ tzinfo: Any,
+) -> _DateutilTimezoneState | None:
+ """Validate bounded exact dateutil tzfile state without timezone hooks."""
+ if type(tzinfo) not in _DATEUTIL_NAMED_TIMEZONE_TYPES:
+ return None
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ transitions = dict.get(namespace, "_trans_list")
+ utc_transitions = dict.get(namespace, "_trans_list_utc")
+ transition_info = dict.get(namespace, "_trans_idx")
+ info_list = dict.get(namespace, "_ttinfo_list")
+ standard_info = dict.get(namespace, "_ttinfo_std")
+ before_info = dict.get(namespace, "_ttinfo_before")
+ first_info = dict.get(namespace, "_ttinfo_first")
+ if (
+ type(transitions) is not tuple
+ or type(utc_transitions) is not tuple
+ or type(transition_info) is not tuple
+ or type(info_list) is not list
+ or tuple.__len__(transitions) > _MAX_DATEUTIL_TRANSITIONS
+ or tuple.__len__(utc_transitions) != tuple.__len__(transitions)
+ or tuple.__len__(transition_info) != tuple.__len__(transitions)
+ or list.__len__(info_list) == 0
+ or list.__len__(info_list) > _MAX_DATEUTIL_TTINFOS
+ ):
+ return None
+
+ previous_transition: int | None = None
+ previous_utc_transition: int | None = None
+ for index in range(tuple.__len__(transitions)):
+ transition = tuple.__getitem__(transitions, index)
+ utc_transition = tuple.__getitem__(utc_transitions, index)
+ if (
+ type(transition) is not int
+ or type(utc_transition) is not int
+ or abs(transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or abs(utc_transition) > _MAX_DATEUTIL_TRANSITION_MAGNITUDE
+ or (previous_transition is not None and transition <= previous_transition)
+ or (
+ previous_utc_transition is not None
+ and utc_transition <= previous_utc_transition
+ )
+ ):
+ return None
+ previous_transition = transition
+ previous_utc_transition = utc_transition
+
+ known_info_ids: set[int] = set()
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if _dateutil_ttinfo_offset_without_hooks(info) is None:
+ return None
+ known_info_ids.add(id(info))
+ for info in (standard_info, before_info, first_info):
+ if info is not None and id(info) not in known_info_ids:
+ return None
+ for index in range(tuple.__len__(transition_info)):
+ if id(tuple.__getitem__(transition_info, index)) not in known_info_ids:
+ return None
+ if not transitions:
+ if (
+ standard_info is not list.__getitem__(info_list, 0)
+ or first_info is not standard_info
+ or before_info is not None
+ ):
+ return None
+ else:
+ expected_standard = None
+ expected_dst = None
+ for index in range(tuple.__len__(transition_info) - 1, -1, -1):
+ info = tuple.__getitem__(transition_info, index)
+ is_dst = object.__getattribute__(info, "isdst")
+ if expected_standard is None and not is_dst:
+ expected_standard = info
+ elif expected_dst is None and is_dst:
+ expected_dst = info
+ if expected_standard is not None and expected_dst is not None:
+ break
+ if expected_standard is None:
+ expected_standard = expected_dst
+ expected_before = None
+ for index in range(list.__len__(info_list)):
+ info = list.__getitem__(info_list, index)
+ if not object.__getattribute__(info, "isdst"):
+ expected_before = info
+ break
+ if expected_before is None:
+ expected_before = list.__getitem__(info_list, 0)
+ if standard_info is not expected_standard or before_info is not expected_before:
+ return None
+ standard_offset = _dateutil_ttinfo_offset_without_hooks(standard_info)
+ if standard_offset is None:
+ return None
+ before_offset = (
+ _dateutil_ttinfo_offset_without_hooks(before_info)
+ if before_info is not None
+ else None
+ )
+ transition_offsets: list[int] = []
+ previous_offset: int | None = None
+ previous_base_offset: int | None = None
+ previous_is_dst: int | None = None
+ previous_dst_offset = 0
+ for index in range(tuple.__len__(transition_info)):
+ info = tuple.__getitem__(transition_info, index)
+ if id(info) not in known_info_ids:
+ return None
+ offset = _dateutil_ttinfo_offset_without_hooks(info)
+ if offset is None:
+ return None
+ is_dst = object.__getattribute__(info, "isdst")
+ dst_offset_seconds = 0
+ if previous_is_dst is not None and is_dst:
+ if not previous_is_dst:
+ assert previous_offset is not None
+ dst_offset_seconds = offset - previous_offset
+ if not dst_offset_seconds and previous_dst_offset:
+ dst_offset_seconds = previous_dst_offset
+ previous_dst_offset = dst_offset_seconds
+ base_offset = offset - dst_offset_seconds
+ adjustment = base_offset
+ if (
+ previous_base_offset is not None
+ and base_offset != previous_base_offset
+ and is_dst != previous_is_dst
+ ):
+ adjustment = previous_base_offset
+ if (
+ tuple.__getitem__(transitions, index)
+ != tuple.__getitem__(utc_transitions, index) + adjustment
+ ):
+ return None
+ transition_offsets.append(offset)
+ previous_offset = offset
+ previous_base_offset = base_offset
+ previous_is_dst = is_dst
+ if transitions and before_offset is None:
+ return None
+ return _DateutilTimezoneState(
+ transitions=transitions,
+ transition_offsets=tuple(transition_offsets),
+ standard_offset=standard_offset,
+ before_offset=before_offset,
+ )
+
+
+def _dateutil_named_offset_without_hooks( # noqa: C901
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Preserve dateutil's source-selected wall offset from validated state."""
+ state = _dateutil_named_state_without_hooks(tzinfo)
+ if state is None:
+ return None
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_timestamp = (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3_600
+ + value.minute * 60
+ + value.second
+ )
+ transitions = state.transitions
+ selected_offset: int | None
+ if not transitions:
+ selected_offset = state.standard_offset
+ else:
+ index = bisect_right(transitions, wall_timestamp) - 1
+
+ def offset_at(transition_index: int | None) -> int | None:
+ if transition_index is None or transition_index + 1 >= len(transitions):
+ return state.standard_offset
+ if transition_index < 0:
+ return state.before_offset
+ return state.transition_offsets[transition_index]
+
+ if index > 0:
+ selected_offset = offset_at(index)
+ previous_offset = offset_at(index - 1)
+ if selected_offset is None or previous_offset is None:
+ return None
+ is_ambiguous = wall_timestamp < transitions[index] + (
+ previous_offset - selected_offset
+ )
+ if not value.fold and is_ambiguous:
+ index -= 1
+ selected_offset = offset_at(index)
+ if selected_offset is None:
+ return None
+ try:
+ return timezone(timedelta(seconds=selected_offset))
+ except (OverflowError, ValueError):
+ return None
+
+
+def _pytz_named_offset_without_hooks(tzinfo: Any) -> timezone | None:
+ """Return a localized pytz zone's stored offset without calling hooks."""
+ if _timezone_name_without_hooks(tzinfo) is None:
+ return None
+ namespace = _object_namespace(tzinfo)
+ offset = dict.get(namespace, "_utcoffset") if namespace is not None else None
+ if type(offset) is not timedelta:
+ return None
+ try:
+ return timezone(offset)
+ except ValueError:
+ return None
+
+
+def _dateutil_local_offset_without_hooks(
+ value: datetime, tzinfo: Any
+) -> timezone | None:
+ """Select a dateutil local offset using builtin system-time data."""
+ if type(tzinfo) is not _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ return None
+ namespace = _object_namespace(tzinfo)
+ if namespace is None:
+ return None
+ standard_offset = dict.get(namespace, "_std_offset")
+ daylight_offset = dict.get(namespace, "_dst_offset")
+ has_daylight = dict.get(namespace, "_hasdst")
+ if (
+ type(standard_offset) is not timedelta
+ or type(daylight_offset) is not timedelta
+ or type(has_daylight) is not bool
+ ):
+ return None
+ selected_offset = standard_offset
+ if has_daylight:
+ epoch = datetime(1970, 1, 1)
+ naive = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ )
+ timestamp = (naive - epoch).total_seconds()
+ try:
+ is_daylight = bool(
+ system_time.localtime(timestamp + system_time.timezone).tm_isdst
+ )
+ daylight_saved = daylight_offset - standard_offset
+ previous_is_daylight = bool(
+ system_time.localtime(
+ timestamp
+ - timedelta.total_seconds(daylight_saved)
+ + system_time.timezone
+ ).tm_isdst
+ )
+ except (OverflowError, OSError, ValueError):
+ return None
+ if not is_daylight and is_daylight != previous_is_daylight:
+ is_daylight = not bool(value.fold)
+ selected_offset = daylight_offset if is_daylight else standard_offset
+ try:
+ return timezone(selected_offset)
+ except ValueError:
+ return None
+
+
+def _canonical_timezone(tzinfo: Any) -> timezone | ZoneInfo | None: # noqa: C901
+ if any(type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES):
+ return tzinfo
+ if type(tzinfo) in _DATEUTIL_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if type(namespace) is not dict:
+ return None
+ offset = dict.get(namespace, "_offset")
+ if type(offset) is not timedelta:
+ return timezone.utc if type(tzinfo) is type(dateutil_tz.tzutc()) else None
+ if abs(offset) >= timedelta(days=1):
+ return None
+ return timezone(offset)
+ if type(tzinfo) in _PYTZ_FIXED_TIMEZONE_TYPES:
+ try:
+ namespace = object.__getattribute__(tzinfo, "__dict__")
+ except (AttributeError, TypeError):
+ return None
+ if type(namespace) is not dict:
+ return None
+ minutes = dict.get(namespace, "_minutes")
+ if type(minutes) is not int or not -1_440 < minutes < 1_440:
+ return None
+ return timezone(timedelta(minutes=minutes))
+ return None
+
+
+def _timestamp_offset_without_hooks(value: pd.Timestamp) -> timezone | None:
+ """Recover a timestamp's stored wall-clock offset without timezone hooks."""
+ multipliers = {"s": 1_000_000_000, "ms": 1_000_000, "us": 1_000, "ns": 1}
+ multiplier = multipliers.get(value.unit)
+ if multiplier is None:
+ return None
+ try:
+ instant_ns = int(value.asm8.view("i8")) * multiplier
+ epoch_ordinal = date.toordinal(date(1970, 1, 1))
+ wall_ns = (
+ (
+ (datetime.toordinal(value) - epoch_ordinal) * 86_400
+ + value.hour * 3600
+ + value.minute * 60
+ + value.second
+ )
+ * 1_000_000_000
+ + value.microsecond * 1000
+ + value.nanosecond
+ )
+ offset_ns = wall_ns - instant_ns
+ if offset_ns % 1000:
+ return None
+ return timezone(timedelta(microseconds=offset_ns // 1000))
+ except (OverflowError, TypeError, ValueError):
+ return None
+
+
+def _canonical_datetime(value: datetime) -> tuple[str | None, str | None]:
+ """Serialize an exact datetime through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ canonical_tz = (
+ _pytz_named_offset_without_hooks(tzinfo)
+ or _dateutil_local_offset_without_hooks(value, tzinfo)
+ or _dateutil_named_offset_without_hooks(value, tzinfo)
+ or _canonical_timezone(tzinfo)
+ )
+ if canonical_tz is None:
+ return None, "a datetime with an unsupported timezone"
+ canonical_value = datetime(
+ value.year,
+ value.month,
+ value.day,
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return datetime.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid datetime"
+
+
+def _canonical_time(value: time) -> tuple[str | None, str | None]:
+ """Serialize an exact time through trusted timezone state only."""
+ tzinfo = value.tzinfo
+ canonical_value = value
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ if _dateutil_named_state_without_hooks(tzinfo) is not None:
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ fold=value.fold,
+ )
+ else:
+ canonical_tz = _canonical_timezone(tzinfo)
+ if canonical_tz is None and type(tzinfo) is _DATEUTIL_LOCAL_TIMEZONE_TYPE:
+ namespace = _object_namespace(tzinfo)
+ if namespace is not None and dict.get(namespace, "_hasdst") is False:
+ offset = dict.get(namespace, "_std_offset")
+ if type(offset) is timedelta:
+ try:
+ canonical_tz = timezone(offset)
+ except ValueError:
+ canonical_tz = None
+ if canonical_tz is None:
+ return None, "a time with an unsupported timezone"
+ canonical_value = time(
+ value.hour,
+ value.minute,
+ value.second,
+ value.microsecond,
+ tzinfo=canonical_tz,
+ fold=value.fold,
+ )
+ try:
+ return time.isoformat(canonical_value), None
+ except (OverflowError, TypeError, ValueError):
+ return None, "an invalid time"
+
+
+def _canonical_timestamp(value: pd.Timestamp) -> tuple[str | None, str | None]:
+ """Preserve a trusted timestamp's instant, offset, nanoseconds, and fold."""
+ try:
+ tzinfo = value.tzinfo
+ if tzinfo is not None and not any(
+ type(tzinfo) is trusted for trusted in _TRUSTED_TIMEZONE_TYPES
+ ):
+ supported_timezone = (
Review Comment:
Timezone-aware query results commonly arrive from pandas with `pytz.UTC`, but this allowlist only accepts `datetime.timezone` and `ZoneInfo`; the result-envelope check then rejects the timestamp before every MCP data/preview tool can return it. Could this normalize or accept the standard pytz UTC/named-zone instances rather than rejecting ordinary timestamptz results?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43771:
URL: https://github.com/apache/superset/pull/43771#discussion_r3989225934
##########
docs/docs/using-superset/using-ai-with-superset.mdx:
##########
@@ -341,3 +341,20 @@ Ask your admin for the MCP server URL and any authentication tokens you need.
- Verify the connection in your AI client (e.g., the hammer icon in Claude Desktop)
- Ask the AI "What Superset tools are available?" to confirm the connection
- Restart your AI client if you recently changed the configuration
+
+
+### Sunburst chart data
+
+MCP chart tools support ordered Sunburst hierarchies with a primary metric and
+an optional secondary color metric. Use `chart_type: "sunburst"` in typed requests;
Review Comment:
This wording suggests color encodes the secondary metric directly, but the implementation colors by the secondary-to-primary ratio, so a profit/revenue chart could be read as profit rather than margin. Could this say that the ratio drives the sequential color scale and that omitting it uses categorical colors?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5864922768
Merged master without rewriting history and resolved the preview-validation conflict while preserving Sunburst normalization and the decimal-ID lookup fix. Validation: 343 Sunburst tests passed; pre-commit passed against all files changed from master. Live streamable-HTTP MCP checks against real SQLite data passed generation, update, compile, URL/ASCII/table previews, JSON data, SQL, CSV/Excel exports, and schema/example validation. Vega-Lite returns the documented UnsupportedFormat response instead of misrepresenting a hierarchy. Exact-head CI and independent review remain required before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(mcp): add typed Sunburst chart support [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43771:
URL: https://github.com/apache/superset/pull/43771#issuecomment-5580604803
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org