You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "msyavuz (via GitHub)" <gi...@apache.org> on 2026/10/05 17:28:15 UTC
[I] [SIP] Proposal for Canvas, the document that places widgets [superset]
msyavuz opened a new issue, #44989:
URL: https://github.com/apache/superset/issues/44989
## [SIP] Proposal for Canvas, the document that places widgets
Canvas is the next generation of Superset dashboards: a tree of placements of SIP-231 widget instances, edited live through small, validated operations by people and agents alike. It is the "companion Canvas SIP" that SIP-231 (#44875) defers to for the Canvas document, placements and their filter scope, promoting and inlining instances, the snapshot export, the listener bus (filter state, URL serialization, cascading filters) and Canvas permalinks.
- **Agents edit a canvas reliably and cheaply.** "Put the revenue trend next to the KPIs and make the region filter apply to the whole page." The agent reads a compact document with readable ids and sends a few named operations; the server validates every one.
- **A canvas can't be left broken.** Every write is checked against the tree, each widget's nesting and layout rules, and its props schema. A bad write is rejected with a JSON-pointer path and changes nothing.
- **Many editors at once.** Edits apply against a base revision; edits that don't overlap merge, overlapping ones are rejected with the placements in conflict. Open pages update live.
- **Any widget, from core or an extension.** Containers (tabs, groups, a filter bar) are widgets too; the canvas grammar stays small and fixed.
- **Filters that follow the layout.** A filter drives the widgets in its tab by default; a filter in the filter bar drives the whole canvas. Overrides are explicit and per placement.
- **Upgrades that never lose work.** Placements whose widget is missing or newer than the server stay as placeholders, exactly as stored.
Existing dashboards keep working alongside Canvas. Conversion and retirement are phased and gated on milestones.
**Scope.** This SIP covers everything *between* placements: the document, layout and nesting, filter scope and the listener bus, editing and concurrency, snapshots, permalinks, storage and access for canvases. Everything *inside* a placement (what props mean, their schema and validation, queries, data, renderers and widget versions) is SIP-231's, and this SIP only references it; the ops that write inline props are this SIP's.
> **Open questions for reviewers** (details at the end)
> 1. Changes this SIP needs from SIP-231: runtime choices for customizations, and one shared flag.
> 2. Unfinished instances: can a placement hold a widget that isn't fully configured yet?
> 3. Previewing a change before every viewer sees it.
> 4. Live layout edits: placing a widget can push others down for every viewer at once. Is that acceptable without an edit mode?
> 5. The op log lets viewers read changes made before they had access.
> 6. History and undo: the op log could power both, but canvases aren't versioned yet.
> 7. Filters across datasources: matching by column name, or an explicit mapping on the scope override.
> 8. Conversion from dashboards: which unconverted features block a dashboard from converting, and which are only reported.
### Motivation
1. **Dashboards are hard for agents to edit.** A dashboard is `position_json` plus `json_metadata`, saved whole. Every layout change rewrites the full tree, including `parents` arrays and wrapper `ROW`/`COLUMN` components, and the API accepts any JSON. In our evaluation (see Evaluation below), a small model left 4 of 27 hard-tier dashboards unrenderable without being told, and every model paid 2.7–3.8× more per edit than on the canvas schemas.
2. **The server doesn't know the dashboard grammar.** Nesting rules live in the frontend (`isValidChild.ts`); on save the server only repairs unreachable components. Ids are client-made and never checked.
3. **Last write wins.** A whole-document `PUT` silently overwrites concurrent edits, which matters more once agents edit alongside people.
4. **Filter scope is stored as layout paths plus caches.** `rootPath` and `excluded` with derived `chartsInScope`/`tabsInScope` caches; cross-filters and customizations each have their own config shape.
5. **SIP-231 needs a home for placements.** Widget instances are persisted or inline; both need a document that places them and scopes their filters, and inline instances need a way to change.
#### Evaluation: which schema works best for agents
We evaluated candidate schemas by having Claude models make real edits through MCP tools against a scored copy of a canvas: 26 tasks (16 single-intent edits on a 15-placement canvas, 10 harder ones on a 51-placement canvas), 3 samples each, with Haiku 4.5 and Sonnet 5.5 on seven canvas schemas and today's dashboards (which skip one hard task that has no dashboard equivalent), plus the factorial screening below: 1,722 runs. A run is **clean** when the requested end state holds and nothing else changed. Known-correct solutions score 100% clean on every schema, and no task passes on an unchanged canvas.
| Hard tier, clean % / cost per run | Today's dashboards | Canvas schemas |
| --- | --- | --- |
| Haiku 4.5 | 74% / $0.142 | 93% / $0.053 |
| Sonnet 5.5 | 93% / $0.295 | 97% / $0.078 |
A fractional factorial over six schema dimensions (16 schemas, 240 runs per level) found no reliability effect from referenced vs inline config, tree-derived vs explicit scope, sparse vs normalized props, or flat vs nested structure: every level landed within two points. Two dimensions changed cost by about a third with no reliability loss: **readable ids** instead of UUIDs, and **leaving resolved placements and scopes out** of the default read. Both are adopted below (§3, §9). Failures clustered by task (bulk resizing, vague references) rather than by schema.
### Proposed Change
#### 1. Concepts
Terms follow SIP-231.
- **Widget:** a kind of thing that can be placed, from core or an extension (SIP-231 §2). Containers are widgets.
- **Widget instance:** one configured occurrence of a widget, **persisted** (its own UUID, in SIP-231's `WidgetInstance` table) or **inline** (stored on its placement as SIP-231's `widget_id`, `schema_version` and `props`, spelled `widget`, `schemaVersion` and `props` in the document's camelCase).
- **Placement:** an instance's spot on a canvas (SIP-231 §5): its id, layout, children and filter scope. Scope overrides are stored under `interactions`, keyed by placement id, so every kind's overrides sit together; they belong to the placement and go when it goes.
- **Canvas:** a document of placements, interactions and settings, plus row metadata (title, slug, theme, CSS, certification, editors, viewers).
| Owned by this SIP | Owned by SIP-231 |
| --- | --- |
| The document: placements, tree, layout, interactions, settings | Widgets, instances and their props |
| Containers' layout and nesting rules, scope roles (fields added to `WidgetBehavior`) | Props schemas, validation, errors, progressive disclosure |
| Filter scope, the listener bus, cascading, permalinks | Runtime filter vocabulary and its hooks; applying filters to queries |
| Operations (including edits to inline props), concurrency, live updates | Edits to persisted instances |
| Canvas storage, slugs, access, embedding | `WidgetInstance` storage and access; instance and placement data routes |
| Document versions (Alembic) | Props versions (migrators) |
| Snapshot format and access rules | What an instance contributes to a snapshot |
| Canvas MCP tools | Widget MCP tools |
<details>
<summary><b>2. The canvas document</b></summary>
```json
{
"version": 1,
"root": {"layout": {"columns": 24, "gap": 16, "rowUnit": 40}, "children": ["kpis", "overview"]},
"nodes": {
"kpis": {"widget": "group", "schemaVersion": 1, "props": {}, "layout": {"colSpan": 24, "rowSpan": 4},
"children": ["revenue", "orders"]},
"revenue": {"instance": "6f1c2b7e-2d4a-4c1e-9a53-0f3b8d2e7a10", "layout": {"colSpan": 6}},
"orders": {"widget": "metric-tile", "schemaVersion": 1,
"props": {"dataBinding": {"datasetId": 7, "metrics": ["count"]}}, "layout": {"colSpan": 6}},
"overview": {"widget": "tabs", "schemaVersion": 1, "props": {}, "layout": {"colSpan": 24, "rowSpan": 12},
"children": ["emea"]},
"emea": {"widget": "tab", "schemaVersion": 1, "props": {"title": "EMEA"}, "layout": {},
"children": ["region", "trend"]},
"region": {"widget": "filter.select", "schemaVersion": 1,
"props": {"datasetId": 7, "column": "region"}, "layout": {"colSpan": 6, "rowSpan": 2}},
"trend": {"widget": "echarts", "schemaVersion": 1, "props": {"…": "…"}, "layout": {"colSpan": 18, "rowSpan": 8}}
},
"interactions": {
"filters": {"region": {"mode": "global", "exclude": ["revenue"]}},
"crossFilters": {},
"customizations": {}
},
"settings": {
"refresh": {"interval": 300, "stagger": 5000, "exempt": ["revenue"]},
"colors": {"scheme": "supersetColors", "labelColors": {"EMEA": "#1f77b4"}},
"display": {"showTimestamps": true},
"crossFilters": {"enabled": true}
}
}
```
| Part | Rule |
| --- | --- |
| `version` | Document format version, mirrored in `canvases.definition_version` |
| `root` | Not a placement; always exists, addressed as `"root"`. A grid: 24 columns, 16px gap, 40px rows by default |
| `nodes` | Placements, a flat map keyed by placement id. The tree is stored only in `children` lists, whose order is reading order |
| Persisted placement | `instance`: the instance's UUID. Props, schema version and access live with the instance |
| Inline placement | `widget`, `schemaVersion`, `props`: explicitly set values only, valid against the widget's schema at that version |
| `layout` | Where the placement sits in its parent, validated against the parent's rules |
| Grid layout | 1-based `col`, `row`, `colSpan`, `rowSpan`. `col` and `row` are set together or omitted to auto-place in reading order. Spans default to the widget's default size on `add`, else the full grid width and one row. Colliding explicit positions are pushed down in reading order |
| `interactions` | Scope overrides per kind, keyed by placement (§5) |
| `settings` | Refresh, shared colors, timestamps, cross-filters on or off |
An instance can be placed more than once, even on one canvas; each placement has its own layout, scope and filter values (SIP-231 §5, §9).
**Inline by default.** A widget added on a canvas is an inline instance, whatever the widget: it lives and is shared with the canvas, travels with it, and has no permissions of its own to manage. It becomes persisted when someone promotes it (§6), to place it elsewhere, use it outside the canvas, or give it its own access. Existing persisted instances are placed by reference.
</details>
<details>
<summary><b>3. Placement ids</b></summary>
Placement ids are readable slugs, unique within a canvas: `revenue`, `revenue-trend`, `revenue-trend-2` (`^[a-z0-9]+(-[a-z0-9]+)*$`, at most 64 characters; `root` and `settings` are reserved). The server derives one from an inline instance's `title` prop, else the widget's name; callers may choose their own on `add`, so later operations in the same request can reference it, a UI can draw the placement before the response, and a retried `add` fails instead of duplicating.
An inline instance's identity is the canvas UUID plus its placement id (SIP-231 §5). Persisted instances keep UUIDs, which are global.
Readable ids are an agent-facing decision backed by evidence: across 480 runs, readable ids cut agent cost by about a third against UUIDs with no loss in reliability (see Evaluation).
</details>
<details>
<summary><b>4. Containers and layout</b></summary>
The canvas ships no layout primitives. Containers are widgets, and widgets declare how they take part in a canvas in SIP-231's `WidgetBehavior` (agreed with SIP-231, beside its `emits_filters` and `drill`); widget ids in them are the registered ids, namespaced for extension widgets (`extensions.<publisher>.<name>.<id>`):
- `container`, `accepted_children` (SIP-231's "which child widgets they accept"), `allowed_parents`;
- `grid_columns` (children on a grid, collisions resolved by the server), a `child_layout_model` (any schema, arranged by the container's renderer), or neither (children take no layout, such as tab panes);
- `bounds_filter_scope` (default true);
- `filter` (sets runtime filters), `customization` (sets runtime choices, §5) and `filterable` (receives them).
`WidgetUi` carries `default_size` (applied on `add` when a grid layout leaves spans unset) and min and max spans. Core registers `tabs` (accepts only `tab`), `tab` (24-column grid, only in `tabs`), `group` (12-column grid) and `filter.bar` (24-column grid of `filter.select` that doesn't bound scope); `tabs`, `tab` and `group` bound scope. Of the other built-ins, `echarts` and `ag-grid-table` are filterable and cross-filter, `metric-tile` and `balloons` are filterable, `filter.select` is a filter and filterable (so filters cascade, §5) and `markdown` is a leaf.
The server resolves auto-placement and push-down and returns resolved positions, so clients never reimplement placement.
</details>
<details>
<summary><b>5. Interactivity: filter scope and the listener bus</b></summary>
**Scope.** Filter scope belongs to placements (SIP-231 §9). Filters, cross-filter sources and customizations (such as a dynamic group-by) share one model, declared by widget behavior (`filter`, SIP-231's `emits_filters`, `customization`) and received by `filterable` widgets:
- **`auto`** (default): drives every filterable placement under the nearest container that bounds scope, or the whole canvas, never itself, minus `exclude`. A filter in a tab drives the tab; a filter in the filter bar drives the canvas. An unresolved container keeps its filters contained.
- **`global`**: every filterable placement, minus `exclude`.
- **`custom`**: exactly `targets`.
Resolved scopes are computed on read, never stored, and returned as `filterScopes`, `crossFilterScopes` and `customizationScopes`. Cross-filter scopes resolve empty while cross-filters are off.
**The listener bus** connects placements on an open canvas:
- **Where it lives.** One bus per open canvas page, holding the current value each filter, cross-filter source and customization has published, keyed by placement id and kind. Values are view state and never enter the document.
- **How widgets take part.** Through SIP-231's hooks: `emitFilter()` publishes for the emitting placement, and `useRuntimeFilters()` returns the values published by placements whose resolved scope includes this one, in SIP-231's runtime-filter vocabulary (`{column, operator, value, datasource?}`). A placement sends what reaches it as `runtime_filters` when it requests data (SIP-231 §8).
- **Customizations.** A customization, such as the dynamic group-by of SIP-185 (#35158), publishes a choice rather than a filter, with `emitRuntimeProps({path: value})`. The receiving widget marks the props a viewer may change at runtime with `x-runtime`, listing the allowed values (for a group-by, the dimensions the instance offers); `useRuntimeProps()` returns the choices reaching its placement, and `useWidgetData()` sends them as `runtime_props`, validated against those values like a write. A viewer picks among what the stored instance offers and can't add dimensions, metrics or columns, so the stored definition still owns the query. These are amendments to SIP-231 (see "Changes this SIP asks of SIP-231").
- **Which placements a filter reaches.** Scope decides the candidates; datasource decides applicability. Every filter and cross-filter on the bus carries its `datasource`, and reaches only in-scope placements that query that datasource, which the server reports per placement with the resolved scopes. Applying a filter across datasources, by column name or an explicit mapping, is a later, additive field on scope overrides.
- **Cascading filters.** A filter widget that is also `filterable` receives the filters whose scope includes it and narrows its options with them, so a city filter in the scope of a country filter cascades with no separate dependency setting. Published values are never rewritten by options, so mutual scopes can't loop.
- **URLs and permalinks.** The bus state serializes to `{placement id: {kind: value}}`. `POST /api/v1/canvas/{uuid}/permalink` stores it, with the canvas revision, in the key-value store and returns a key; `/canvas/p/<key>/` opens the canvas with that state. Values for placements that no longer exist, or whose scope no longer reaches a target, are dropped on load.
</details>
<details>
<summary><b>6. Editing: operations</b></summary>
`POST /api/v1/canvas/` can seed a new canvas with a whole definition, validated in full. After that, every change goes through one route, atomically, against a base revision:
```
PATCH /api/v1/canvas/{id_or_uuid}/definition
{"base_revision": 7, "ops": [
{"op": "add", "id": "kpis", "widget": "group", "layout": {"colSpan": 24, "rowSpan": 4}},
{"op": "add", "id": "revenue", "parent": "kpis", "instance": "6f1c2b7e-…", "layout": {"colSpan": 6}},
{"op": "set_scope", "kind": "filter", "id": "region", "scope": {"mode": "global"}}
]}
→ 200 {"revision": 8, "ops": [{…, "id": "kpis"}, {…, "id": "revenue"}, {…}], "filterScopes": {…}, "placements": {…}}
```
| Op | Effect |
| --- | --- |
| `add` | Places a persisted (`instance`) or new inline (`widget`, `props`, optional `schemaVersion`) instance under `parent` (default `root`) at `index` (default last); inline props are migrated to and stored at the widget's current schema version |
| `set_props` | Replaces an inline instance's props, with an optional `schemaVersion` as on `add` |
| `patch_props` | Applies a JSON Patch (RFC 6902) to an inline instance's props, so an edit to one nested prop doesn't resend the rest |
| `place` | Changes position and size within the parent (drag, resize) |
| `move` | Reparents or reorders, optionally with a new `layout`; reparenting without one resets the layout to auto-place |
| `remove` | Removes a placement and its subtree, pruning them from scopes and refresh exemptions |
| `set_scope` | Sets or clears a placement's scope override for one kind |
| `set_settings` | Replaces one settings section; `null` resets it |
| `promote` | Saves an inline instance as a persisted one: creates a `WidgetInstance` from its widget, schema version and props, with the caller as editor, and the placement references it. Needs SIP-231 `can_write` on `Widget` |
| `inline` | Copies a persisted instance's widget, schema version and props onto its placement, detached from the original. Needs read access to the instance |
- **Editing inline instances.** Inline props change only through canvas ops: `add`, `set_props` and `patch_props`. Each write is validated in full on the instance it writes (schema, datasource access, referenced columns and metrics, and the queries it would build), and leaves other placements alone. `schemaVersion` on a write lets props saved at an older version be migrated rather than mislabelled. `promote` and `inline` touch the placement's `props`. Persisted instances change through SIP-231's instance API with its own revision check.
- **Atomic and live.** Ops apply in order to a copy; all land or none do. An accepted request is the stored canvas; there is no edit mode or Save.
- **Concurrency.** Each op touches `(placement, field group)` pairs: `tree`, `layout`, `props`, one group per scope kind, one per settings section. An `add` touches only the new placement. A write based on an older revision merges unless it overlaps a change logged since; then it gets 409 `{message, revision, conflicts, stale: false}`. `tree` overlaps every group. A base revision ahead of the canvas, or older than the retained log (or any base before the first write after creation), gets 409 with `stale: true`: reload. A merged write whose op no longer applies, such as an `add` into a parent someone removed, gets 422.
- **Live updates.** Each write publishes `entity.changed` with `{entity_type: "canvas", id}` only, and open pages refetch the definition through the authorized API. Without a realtime transport, pages poll `…/definition/changes?since=N` every 5 seconds and refetch when it reports a newer revision; that route also returns the logged ops, grouped by revision, and answers 409 `stale` when `since` is older than the retained log, upon which the page reloads.
- **Errors** carry JSON-pointer paths, in the SIP-231 Appendix C spirit: 400 `{message: [{path, message}]}` for a malformed request (paths into the body), and 422 `{message, errors: [{path, message}], operation?}` when an op can't apply (`/ops/<index>`) or the result is invalid (paths into the document).
</details>
<details>
<summary><b>7. Validation and evolution</b></summary>
- **Strict where a write touches, tolerant elsewhere.** Tree integrity is always checked. Widget behavior (nesting, spans, child layout, scope roles) is enforced on the placements a write touches, and inline props are validated where a write wrote them. A widget that tightens a rule or its schema never blocks edits elsewhere on the canvas.
- **Placeholders.** A placement whose instance is missing, whose widget isn't registered (say, an extension was removed), or whose props were saved by a newer widget version or fail to migrate stays exactly as stored, with its children. The rest of the canvas renders and stays editable; re-enabling the widget restores it.
- **Props versions** follow SIP-231 §12: inline props are migrated in memory on read by the widget's migrators and stored upgraded on the next write.
- **Document versions.** A change to the canvas document format bumps `version` and ships an Alembic migration with a downgrade, which also clears the op log so open pages reload; additive changes ship expand/contract across two releases. A document at any other version, such as one from a newer server, is refused with 422 rather than read with the wrong schema. Importing an older snapshot (§11) first converts it with the same transformations its migrations apply.
</details>
<details>
<summary><b>8. Storage, permissions and data</b></summary>
- **Tables:** `canvases` (integer id plus UUID, audit columns, `title`, `description`, `definition`, `definition_version`, `revision`, `slug`, `css`, `theme_id`, certification and external-management columns), `canvas_editors` / `canvas_viewers` (subject-based, like dashboards), and `canvas_ops` (the op log, last 500 revisions per canvas).
- **Slug:** optional, unique across canvases, letters, digits, `_` and `-`, at most 255 characters, never all digits and never `list`. The canvas page is `/canvas/<slug>/` when set, `/canvas/<id>/` otherwise.
- **Access:** admins, editors and viewers see a canvas, and `EXTRA_ACCESS_QUERY_FILTERS["canvases"]` can widen that; embedded guests see none. Writes need editorship (admins always have it); the creator is the default editor. Seeing a canvas grants no data access by itself: a viewer also needs access to each persisted instance placed on it and to its datasource (SIP-231 §8), and placements they can't read render as no-access placeholders. Granting placed instances through the canvas would add a row to the role and capability matrix in `SECURITY.md` and is left for a later proposal. Permissions are `can_read` and `can_write` on `Canvas`; widget APIs use SIP-231's `Widget` resource.
- **Data:** SIP-231 §8 defines how inline instances are read, through `POST /api/v1/canvas/{uuid}/placement/{placement_id}/data`. This SIP adds only the canvas side: the caller needs read access to the canvas, and the placement must resolve.
- **Feature flag:** one flag, `CANVAS` (off by default), shared with SIP-231 since neither ships without the other. It gates registration of the canvas and widget REST APIs and MCP tools, the SPA routes and the menu, which 404 and hide while it is off. Canvases are kept when it is turned off.
</details>
<details>
<summary><b>9. Agent access</b></summary>
MCP tools wrap the same commands as the REST API:
| Tool | Purpose |
| --- | --- |
| `get_canvas` | The document and its revision. Resolved positions and scopes only on request (`include_resolved`) |
| `apply_canvas_ops` | Apply operations atomically against a base revision; returns the new revision and the applied ops with their ids, or the errors |
| SIP-231 §13 tools | Discover widgets and read their schemas progressively; edit persisted instances |
The default read is lean because resolved positions and scopes cost context and didn't improve reliability in the evaluation. The REST `GET` and `PATCH` responses always include them, since the page needs them to render.
</details>
<details>
<summary><b>10. Rendering</b></summary>
The page renders the resolved grid. Widget renderers are SIP-231 renderers, registered once in its renderer registry by namespaced widget id with the schema versions they support (SIP-231 §10, §12); core registers the containers and `markdown`. Data and interactivity reach a renderer through SIP-231's hooks, which the canvas page backs with the placement's instance and the listener bus, so a renderer doesn't know whether its instance is inline or persisted or how often it is placed.
On a canvas, renderers also get canvas context: the placement id, shared colors, whether to show timestamps, and a refresh key that bumps on each automatic refresh. Containers get their rendered children (with ids, widget ids and layouts) or a ready-made grid. Each renderer runs inside an error boundary; placements that don't resolve, or whose widget has no renderer for its schema version, render a placeholder.
</details>
<details>
<summary><b>11. Snapshot export</b></summary>
`GET /api/v1/canvas/{uuid}/snapshot` returns one self-contained, immutable document (SIP-231 §5):
- the canvas metadata (title, description, CSS, theme by UUID) and its definition, with every persisted placement replaced by an inline copy of the instance as it is at export time;
- for each widget used, its id, `schema_version` and implementation version (SIP-231 §12).
Copying an instance inline puts it under whoever can open the snapshot, so the exporter needs read access to every persisted instance; placements they can't read are exported as unresolved placeholders without props. Snapshots hold definitions only, never query results: data is read when the snapshot is opened, under the reader's own access and row-level security. Snapshots aren't stored by the server. Importing one through `POST /api/v1/canvas/` creates a new canvas; widgets that are missing or newer than the server stay placeholders (§7).
</details>
<details>
<summary><b>12. Embedding, reports and export (M2)</b></summary>
- **Embedding.** A canvas gets an embedded configuration (allowed domains), as dashboards do. Guest tokens gain a `{"type": "canvas", "id": "<uuid>"}` resource that grants reading that canvas's definition and the data of its placements, inline or persisted, with the token's row-level security, and nothing else; the access check is on the placement, so an instance isn't readable through a canvas it isn't placed on. This adds a guest-token row to the role and capability matrix in `SECURITY.md`, as dashboard guest tokens have; grants through a canvas for regular viewers stay out of scope (§8).
- **Reports and thumbnails.** A canvas screenshot report or thumbnail captures the layout, so it renders the canvas page headlessly, as dashboard ones do. Data-only reports on a placement read the placement's data route and need no browser.
- **Export and import.** A canvas bundle holds the canvas and the persisted instances it places, by UUID, so importing updates them in place, alongside today's asset bundles. A snapshot (§11) is the self-contained alternative.
</details>
### New or Changed Public Interfaces
<details>
<summary>Full list</summary>
- **REST** (`/api/v1/canvas/`): CRUD with editors and viewers (create may seed a definition or import a snapshot); `GET|PATCH /{id_or_uuid}/definition`, `GET /{id_or_uuid}/definition/changes?since=N`; `GET /schema` (JSON Schemas for the definition, operations and grid placement, plus every registered widget's behavior and UI metadata); `POST /{uuid}/placement/{placement_id}/data`; `GET /{uuid}/snapshot`; `POST /{uuid}/permalink`, opened at `/canvas/p/<key>/`.
- **Operations:** `add`, `set_props`, `patch_props`, `place`, `move`, `remove`, `set_scope`, `set_settings`, `promote`, `inline`.
- **MCP:** `get_canvas`, `apply_canvas_ops`, alongside SIP-231's §13 tools.
- **`superset-core`:** `superset_core.canvas.GridPlacement` and `InstanceResolver` (looks up persisted instances); the container and scope fields this SIP adds to SIP-231's `WidgetBehavior` and `WidgetUi`.
- **`@apache-superset/core`:** canvas context for renderers (placement id, colors, timestamps, refresh key, children and grid for containers); `getActiveCanvas` / `onDidChangeActiveCanvas`. Renderers register in SIP-231's registry, and the canvas page implements SIP-231's `useWidgetData`, `useRuntimeFilters` and `emitFilter` for its placements.
- **Extensions to SIP-231** (see "Changes this SIP asks of SIP-231"): the `x-runtime` keyword, `runtime_props` on data requests, and the `emitRuntimeProps` / `useRuntimeProps` hooks (§5).
- **Realtime:** `entity.changed` with `entity_type: "canvas"`.
- **Security:** `Canvas` permission resource (`can_read`, `can_write`); a `canvas` guest-token resource type (M2).
- **Configuration:** `CANVAS` feature flag, `SUBJECTS_RELATED_TYPES_CANVASES`.
- **Models:** `canvases`, `canvas_ops`, `canvas_editors`, `canvas_viewers`; permalinks use the existing key-value store.
- **Unchanged:** dashboards, `position_json`, `json_metadata`, charts and their APIs.
</details>
### Changes this SIP asks of SIP-231
These need SIP-231's agreement; the rest of this SIP builds on SIP-231 as written.
- **Runtime choices.** `x-runtime` in the documented keyword list, and `runtime_props` beside `runtime_filters` on the instance and placement data routes, with `emitRuntimeProps` and `useRuntimeProps` beside `emitFilter` and `useRuntimeFilters` (§5).
- **One flag.** Both SIPs share `CANVAS` (§8), and it turns on by default only when both reach parity, rather than at SIP-231's M2 alone.
### New dependencies
None.
### Migration Plan and Compatibility
Nothing existing changes until deployments opt in; all new tables, routes and permissions are additive.
1. **M1: Canvas.** Behind the `CANVAS` flag, shared with SIP-231: the document, operations including `promote` and `inline`, concurrency, scopes, the listener bus with SIP-231's hooks, permalinks, snapshot export, rendering, the MCP tools, and core containers. Ships alongside SIP-231 M1.
2. **M2: Parity.** Widgets for the chart and filter types dashboards use; export and import (a canvas bundle with its persisted instances); embedding; canvas reports and thumbnails (§12); direct editing on the page. Reached together with SIP-231's M2, after which the shared flag turns on by default.
3. **M3: Opt-in conversion.** "Convert to canvas" copies a dashboard into a canvas and leaves the original; a conversion report lists anything not carried over. Layout maps one to one (`ROW` widths ×2 to `colSpan`, `COLUMN` to `group`, `TABS`/`TAB` to `tabs`/`tab`), native filters become filter widgets with their scope as `auto` or an override, and `json_metadata` settings map to `settings`.
4. **M4: Canvas by default.** New dashboards are canvases; bulk conversion for the rest.
5. **M5: Retirement.** Dashboard URLs redirect; the old format and renderer are removed in a major release, recorded in `UPDATING.md`.
Each phase is gated on its milestone, not a date.
### Rejected Alternatives
<details>
<summary>Alternatives considered</summary>
- **Building on `dashboards.position_json`.** It keeps whole-document writes, client-side grammar and a dashboard-specific component set; conversion is a one-way copy instead.
- **UUID placement ids.** They cost agents a third more per edit for no reliability gain, and nothing needs them to be global: an inline instance's identity is already the canvas plus its placement.
- **Server-generated ids only.** No references between ops in one request, and retried adds duplicate.
- **A nested tree, or `parents` arrays.** A flat map with `children` gives each placement one address and one source of structure.
- **Whole-document `PUT`, JSON Patch over the document, or key-value drafts of the canvas.** Named ops validate intent and make overlap detection precise; JSON Patch is kept for what it suits, a single instance's props (`patch_props`).
- **Whole-document revision checks, per-placement locks, or CRDT/OT.** A revision plus a touch log merges independent edits with a small, server-side model.
- **Filter scope as explicit target lists, or owned by the filter widget's props.** Scope belongs to placements (SIP-231 §9), and tree-derived defaults follow the layout without upkeep; explicit lists remain available as `custom`.
- **Upgrading the document on read in code.** Format changes use Alembic migrations, so the table never holds mixed versions; widget props, which change on each widget's own schedule, use SIP-231's in-memory migrators.
- **Declaring filter dependencies for cascading.** Scope already says which filters reach a filter, so a second dependency graph could only disagree with it.
- **Filter state in the URL itself, or in the document.** URLs outgrow browser limits with a few multi-select filters, and state in the document would change the canvas for every viewer; a permalink key keeps both small.
- **Persisted instances by default.** Every text block and container would be a row with its own permissions, and a canvas couldn't travel as one document; promote covers sharing.
- **Query results in snapshots.** Opening a snapshot would show data under the exporter's access and row-level security, not the reader's.
- **Customizations as runtime filters.** A group-by changes what is grouped, which an additive filter can't express; letting callers send arbitrary group-bys would let a viewer reshape the query. `x-runtime` limits choices to what the stored instance offers.
- **Validating the whole document on every write.** A tightened widget rule would then block unrelated edits.
</details>
### Prototype
The [`msyavuz/feat/canvas`](https://github.com/apache/superset/tree/msyavuz/feat/canvas) branch implements this proposal on top of the SIP-231 prototype's widget framework: the document, readable placement ids, inline placements and `set_props`, widget behavior and UI declarations with the core containers, scopes, concurrency and live updates, the REST API and MCP tools behind `CANVAS`, and a read-only canvas page that renders the core containers and markdown.
The prototype's widget framework is the SIP-231 prototype, so it differs from SIP-231 in names: widgets register with `@widget(widget_type=…)` rather than `id`, the API is `/api/v1/widgets` with `Chart` permissions rather than `/api/v1/widget` with `Widget`, its schema routes are `/types` and `/type/<id>/control-schema`, its discovery tool is `list_widget_types` rather than `list_widgets`, and `CANVAS` gates its widget API and tools as well as the canvas. Renderers register through a canvas-only `canvas.registerWidgetRenderer` without schema versions, and receive props, instance ids and filter values as props (`filters`, `setFilterValue` and so on) instead of through hooks. `add` and `set_props` validate props against the schema only.
Not yet built:
- **Promote, inline and snapshot export** need the `WidgetInstance` table.
- **Persisted placements** are modelled and validated through `InstanceResolver`, but no resolver is registered until the `WidgetInstance` table exists, so adding one is refused.
- **The placement data endpoint** needs `QueryMixin` and the server-side query pipeline.
- **Chart and filter renderers** need `useWidgetData` and field options; until then chart and `filter.select` placements render a placeholder, so filters can't yet be driven from the page.
- **Permalinks**, SIP-231's hooks and renderer registry, per-placement datasources for the bus, and `x-runtime` / `runtime_props` for customizations.
- **`patch_props`**, `get_widget_info`, `get_widget_field_options`, and validation beyond the props schema (datasource access, columns and metrics, queries).
### Open questions
1. **Changes needed from SIP-231.** Runtime choices for customizations (`x-runtime`, `runtime_props`) and one shared flag; see "Changes this SIP asks of SIP-231".
2. **Unfinished instances.** Every write is validated in full, so a chart can't be built up field by field on a canvas. Either writes must carry complete props (agents and the property panel send whole, valid props), or a placement may hold an incomplete instance with a visible "not configured" state that renders no data.
3. **Previewing before it's live.** Every accepted op is visible to all viewers, so an agent or person can't look at a configuration's data before others see it. Options: a preview data route that takes props without storing them, or accepting that iteration happens live (see 4).
4. **Live layout edits.** Every accepted op is the stored canvas, so placing a widget can push others down for every viewer at once. An edit mode would batch a person's changes before others see them, at the cost of the "no Save" model agents rely on.
5. **Op log visibility.** Viewers can read operations from before they had access through `…/definition/changes`. Options: filter the log by the viewer's access date, or return only revisions and have clients refetch.
6. **History and undo.** The op log records every change with its author, but keeps only 500 revisions and canvases aren't versioned. Decide whether history comes from the op log, from row versioning, or both, and whether undo is an inverse op.
7. **Filters across datasources.** A filter reaches only placements on its own datasource. A later, additive field on scope overrides could map across datasources, either by column name or by explicit column mapping.
8. **Conversion from dashboards.** Which dashboard features that a canvas doesn't model (legacy filter scopes, `expanded_slices`, `async_mode`) block a dashboard from converting, and which are only listed in the conversion report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-232] Proposal for Canvas, the document that places widgets [superset]
Posted by "msyavuz (via GitHub)" <gi...@apache.org>.
msyavuz commented on issue #44989:
URL: https://github.com/apache/superset/issues/44989#issuecomment-6342465280
Hey @rusackas, essentially yes drafts are single-principal and per user. We discussed this extensively for a few weeks and we landed on something really close to dashboards edit mode. Multiplayer canvases doesn't seem too big of a use case for us to support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-232] Proposal for Canvas, the document that places widgets [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #44989:
URL: https://github.com/apache/superset/issues/44989#issuecomment-6298822941
Just wanted to leave a note of thanks for tackling all of this for now. :)
I need to read through this all again, from top to bottom. It's a beast of a SIP, and a critical one. This is, I believe the future of Superset, and this Future Superset is the future of BI & dashboard if we play our cards well here :)
I think this largely supports where I want to go with some subsequent SIPs, but I'll bite my tongue about those.
I don't see any blockers here... I'll re-review, but I'm about ready to give my +1 on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-232] Proposal for Canvas, the document that places widgets [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #44989:
URL: https://github.com/apache/superset/issues/44989#issuecomment-6304829845
Heya @msyavuz, really like the per-gesture operations model here. One thought/question about the "Direct editing: drag and drop (M2)" part, where it says "other editors work in their own drafts, so nothing moves under the pointer." This sounds like drafts would be permanently single-principal, is that accurate? I'm wondering if we should leave wiggle room for a draft to optionally be shared by more than one principal later. This would allow us to broadcast each accepted op to the draft's other live viewers/co-editors over SIP-227's realtime channel when it lands, using the same per-op revision check you already have for conflicts. Nothing to build for M1, but hoping to make sure the token-binding doesn't quietly rule out real-time multi-user editing as a follow-up SIP (something I hope we can get to eventually).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP] Proposal for Canvas, the document that places widgets [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on issue #44989:
URL: https://github.com/apache/superset/issues/44989#issuecomment-6002495426
Thanks @msyavuz for this writeup! A few thoughts/questions:
- There's a mention of Live updates happening through polling every 5 seconds. However, for SIP-231 I'm already moving primarily towards an async+realtime direction as proposed in [SIP-227](https://github.com/apache/superset/issues/44174), which would make it possible to completely skip polling and have instantaneous updates. As Canvas + Widgets is a massive change in itself, I feel now is a great opportunity to clean up design debt and just standardize on fully async and realtime operations going forward.
- If you are going to be working on Canvas, I feel it makes sense to have a shared feature branch for this and SIP-231, as then I can directly start implementing widgets on Canvas (as discussed, the "Widget only" Explore view will likely just be Canvas with a single Widget and no layouts).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP] Proposal for Canvas, the document that places widgets [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on issue #44989:
URL: https://github.com/apache/superset/issues/44989#issuecomment-6004350179
Thanks @msyavuz! I've updated the SIP-231 body to align with this proposal and the feedback on #44875. A few points from the SIP-231 side:
- **Naming:** persisted instances live in a `widgets` table (model `Widget`, the same way `slices` holds `Slice` rows), with `widget_editors`/`widget_viewers`, and the `widget_type` column references the registered widget class. REST is `/api/v1/widget/{uuid}` (plus `/data`), and widget type metadata lives under `/api/v1/widget_type/{widget_type}/…`. Could you update the `WidgetInstance` table and `widget_id` references here? For consistency, the inline placement key could be `widgetType` too.
- **`WidgetBehavior` and `WidgetUi`:** SIP-231 notes that this SIP adds the container, layout and filter-scope fields, so the list in §4 can stay here.
- **Drafts are optional (open questions 2 and 3):** a complete, valid instance can be written directly, so `add`/`set_props`/`patch_props` with full validation work as proposed. Drafts cover the two gaps: an unfinished instance is built up in a draft rather than on the canvas until it validates, and the draft preview data route shows its data before anyone else sees it. Committing a draft of an inline instance is just a `set_props`. Whether edits should be grouped by an explicit commit or persisted live with version history is still open, so SIP-231 lists it as an open question.
- **Runtime choices (`x-runtime`, `runtime_props`):** I'd like M1 to stick to additive runtime filters. Customizations let viewers reshape queries, so they need safeguards first, such as a permission to change runtime choices. Could this move from "Changes this SIP asks of SIP-231" to a later phase? SIP-231 lists it as an open question for now.
- **Flag:** agreed on one shared `CANVAS` flag that turns on by default only when both reach parity. One implementation detail: the widget REST API is registered unconditionally and answers 404 while the flag is off, so its FAB permissions exist and roles can be prepared before the flag is turned on, while the MCP tools are removed. Could the canvas API follow the same pattern?
- **Event bus:** following Michael's feedback on #44875, the typed hooks (`useRuntimeFilters`, `emitFilter`) sit on a generic bus that accepts any event type, so extension widgets can emit their own events. It would be great if the canvas bus kept the event kind open-ended too.
- **Async and realtime:** SIP-231 data serving is now async-first through the Global Task Framework with realtime delivery (SIP-227), in line with my earlier comment about replacing the 5-second polling.
I'll open a shared feature branch with an umbrella draft PR shortly, so we can both start PRing into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org