You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "hughhhh (via GitHub)" <gi...@apache.org> on 2026/09/03 21:23:09 UTC
[PR] feat(datasets): add the partition filter mapping editor UI [superset]
hughhhh opened a new pull request, #43832:
URL: https://github.com/apache/superset/pull/43832
### SUMMARY
Fifth of the partition filter mapping stack, and the PR that makes the feature reachable. Until now a mapping could only be configured with a `PUT /api/v1/dataset/<pk>`, and the preview endpoint added in #pfm-3 had no caller — its own commit message says the visual controls "are not in this PR". This implements them: wireframes **1a, 1b, 1c, 1g, 1h**.
Stacked on `hughhhh/pfm-4-docs`. Review the [compare against pfm-4](https://github.com/apache/superset/compare/hughhhh/pfm-4-docs...hughhhh/pfm-5-editor-ui) rather than the diff against master.
**What's new in the editor**
- **Partition column** select in the Columns tab's Default Column Settings, plus a read-only computed **Maps to partition**. It's deliberately not a peer dropdown — it reflects `partition_mapped_column ?? main_dttm_col`, and a second select would let it drift from the default datetime column silently.
- The partition column's row is muted, carries a `PARTITION` tag, and defaults `Is filterable` / `Is dimension` off — still manually togglable.
- A **Partition filter mapping** section in the row expand with three states: the mapped column holds the transform, every other column offers to take the mapping over, and the partition column itself shows nothing.
- A live **Preview** panel showing sample input → emitted predicate, and a destructive **Remove mapping**.
**Two places the spec and the merged backend disagreed**
1. The mockups have no monotonicity control, but the query path only mirrors ranges when the transform is declared order-preserving — so 1d/1e's own headline example (a time range producing two `dt_epoch` bounds) was unreachable through the UI. This adds the **Transform preserves ordering** checkbox that the docs in pfm-4 already describe.
2. The PRD asks for `unix_timestamp(:value)` as the temporal default, but that's Hive syntax and would not parse on Postgres, Trino or BigQuery. The default moves to the engine spec (`partition_value_transform_default`, set on Hive/Impala/Spark); engines without one offer no pre-fill rather than a wrong one.
**Backend changes**
- The preview endpoint takes `sample_values` + an `operator` and builds its predicate with `build_mirrored_predicates` — the same function the query path uses — so what the panel shows is what a chart emits, `IN` included. It also accepts a candidate `partition_column`, because the editor previews a mapping the owner hasn't saved yet and a preview that requires saving first isn't a preview.
- Failed probes carry the engine's own message through an opt-in `errors` sink (the hot query path passes nothing and stays silent). sqlglot parses unknown functions happily, so a misspelled one is an *engine* error and was previously reported as an unexplained blank.
- Parse failures now name a position, mapped back through the `SELECT` prefix and the `:value` → `NULL` substitution so it points at what the owner actually typed.
**Two silent read-path bugs this uncovered**
Neither was reachable from unit tests; both needed the running app.
- `columns.partition_value_transform` and its monotonic flag were in the model, the export fields and the PUT schema but **not** in `show_columns` — so the editor reopened a saved mapping as if it had none, *and the next save wrote that emptiness back*. Related-model fields must be listed in `show_columns`, not only `show_select_columns` (`columns.advanced_data_type` is in both for the same reason).
- `partition_value_transform_default` needed the same treatment to reach the pre-fill.
Both are now pinned by tests in `partition_mapping_serialization_test.py`, whose whole premise is that a field missing from any one layer is dropped without a sound.
**One small shared change:** `Field` gains an opt-in `passItemToControl`. The row-expand section keys off the whole column record, not just the transform it edits, and handing an unknown `item` prop to every `TextControl` and `Select` wasn't worth the convenience.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
_Screenshots captured during validation; attaching separately._
- `1a` Partition column + Maps to partition, and `1b` the muted `PARTITION` row
- `1c` Row expand with transform, ordering checkbox and a valid preview
- `1c` Parse error state — only one of preview/error is ever visible
- `1g` Partition column set with nothing mapped
- `1h` Non-temporal `country → region_key`
### TESTING INSTRUCTIONS
Automated: `1800` backend unit tests and `225` frontend Datasource tests pass; `tsc`, `ruff`, `ruff format` and `oxlint` are clean (oxlint warnings in `src/components/Datasource` go 32 → 30, since the duplicated column-name renderer was folded into one helper).
Manually, against a real engine:
1. Enable the flag: `FEATURE_FLAGS = {"PARTITION_FILTER_MAPPING": True}`.
2. Create a physical dataset on a table with both a business column and a partition column, where the partition column really is the transform of the other. On Postgres:
```sql
CREATE TABLE partition_demo (event_time TIMESTAMP, dt_epoch BIGINT, country TEXT, region_key TEXT, revenue DOUBLE PRECISION);
INSERT INTO partition_demo
SELECT ts, EXTRACT(epoch FROM ts)::bigint, t.c, lower(t.c), (random()*100)::numeric(10,2)
FROM generate_series(timestamp '2026-07-01', timestamp '2026-08-15', interval '6 hour') ts,
(VALUES ('US'),('CA'),('MX')) AS t(c);
```
3. Edit the dataset → **Columns** → set **Partition column** to `dt_epoch`. Confirm *Maps to partition* shows `event_time` tagged *Default datetime column*, and that `dt_epoch`'s row is muted with a `PARTITION` tag and its toggles off.
4. Follow **Map a different column instead →**. Set the value transform to `cast(extract(epoch from cast(:value as timestamp)) as bigint)` and check **Transform preserves ordering**. The preview should read `event_time >= '2026-01-15 00:00:00'` → `dt_epoch >= 1768435200`. Unchecking it drops the preview back to `=`.
5. Break the transform (`lower(:value))`) → *Can't parse transform* with a position, and the preview panel disappears.
6. Save, reopen — everything round-trips.
7. Build a chart with a time range and open **View query**:
```sql
WHERE event_time >= TO_TIMESTAMP('2026-07-01 …') AND event_time < TO_TIMESTAMP('2026-08-01 …')
AND dt_epoch >= 1782864000 AND dt_epoch < 1785542400
```
8. Non-temporal: point **Partition column** at `region_key`, move the mapping to `country`, set `lower(:value)`. Preview shows `country IN ('US', 'CA')` → `region_key IN ('us', 'ca')`, and a chart filtered on `country` emits `AND region_key IN ('us', 'ca')`.
9. Negative: turn the flag off. The controls disappear, the preview endpoint 404s, and no mirrored predicate is added.
**Merge order.** pfm-4's docs describe this UI, so if pfm-4 merges first the published docs tell operators to click controls that don't exist yet. Merging this before or together with pfm-4 avoids that window; no docs edit is needed either way.
**Known limitation, not introduced here:** SQLAlchemy's `text()` misparses Postgres `::` casts, so a transform written as `:value::timestamp` leaves the placeholder unbound. This comes from `build_probe_sql` in pfm-2 and is harmless on Hive/Impala, which have no `::` syntax — the ANSI `cast(:value as timestamp)` form works. Worth a follow-up for Postgres/Redshift users.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [x] Required feature flags: `PARTITION_FILTER_MAPPING`
- [x] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API
- [ ] Removes existing feature or API
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(datasets): add the partition filter mapping editor UI [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43832:
URL: https://github.com/apache/superset/pull/43832#issuecomment-5532318660
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(datasets): add the partition filter mapping editor UI [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #43832:
URL: https://github.com/apache/superset/pull/43832#issuecomment-5532338468
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | c055c683640ee47a00df3e800890e1b6058ccdd0 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6a99e53d350d1d0008294e1b |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-43832--superset-docs-preview.netlify.app](https://deploy-preview-43832--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#q1YqKMrPLShRslJS0lFKKkrMS84AsjNK0zOAQL8gLVfXVDc1JbMkv0i3NBOoJCW1ICe_0jMFqMgs0dIy1dQ4xdjUIMUwxcDAwMLI0iTVMEmpFgA) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-43832--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(datasets): add the partition filter mapping editor UI [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #43832:
URL: https://github.com/apache/superset/pull/43832#issuecomment-5532361725
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/43832?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `33.96226%` with `35 lines` in your changes missing coverage. Please review.
:warning: Please [upload](https://docs.codecov.com/docs/codecov-uploader) report for BASE (`hughhhh/pfm-4-docs@20fd771`). [Learn more](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#section-missing-base-commit) about missing BASE report.
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/43832?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/connectors/sqla/partition\_mapping.py](https://app.codecov.io/gh/apache/superset/pull/43832?src=pr&el=tree&filepath=superset%2Fconnectors%2Fsqla%2Fpartition_mapping.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9zcWxhL3BhcnRpdGlvbl9tYXBwaW5nLnB5) | 17.07% | [34 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/43832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/connectors/sqla/models.py](https://app.codecov.io/gh/apache/superset/pull/43832?src=pr&el=tree&filepath=superset%2Fconnectors%2Fsqla%2Fmodels.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9zcWxhL21vZGVscy5weQ==) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/43832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## hughhhh/pfm-4-docs #43832 +/- ##
=====================================================
Coverage ? 57.16%
=====================================================
Files ? 2874
Lines ? 164127
Branches ? 37848
=====================================================
Hits ? 93819
Misses ? 69413
Partials ? 895
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/43832/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [presto](https://app.codecov.io/gh/apache/superset/pull/43832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.04% <33.96%> (?)` | |
| [python](https://app.codecov.io/gh/apache/superset/pull/43832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.04% <33.96%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/43832?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(datasets): add the partition filter mapping editor UI [superset]
Posted by "hughhhh (via GitHub)" <gi...@apache.org>.
hughhhh commented on PR #43832:
URL: https://github.com/apache/superset/pull/43832#issuecomment-5545438811
Reopening against `hughhhh/pfm-3-preview-and-editor` so this carries the docs from #43760 as well — GitHub will not let a PR that is part of a stack change its base, so the only way to combine them is a fresh PR from the same branch. Same commits, same branch (`hughhhh/pfm-5-editor-ui`), nothing rebased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(datasets): add the partition filter mapping editor UI [superset]
Posted by "hughhhh (via GitHub)" <gi...@apache.org>.
hughhhh closed pull request #43832: feat(datasets): add the partition filter mapping editor UI
URL: https://github.com/apache/superset/pull/43832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org