You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "RahulBisht001 (via GitHub)" <gi...@apache.org> on 2026/01/25 05:05:43 UTC
[I] Git-backed Version Control for Datasets in Apache Superset [superset]
RahulBisht001 opened a new issue, #37427:
URL: https://github.com/apache/superset/issues/37427
# SIP: Git-backed Version Control for Datasets in Apache Superset
## 1. Motivation
Apache Superset is widely used as a self-service BI and data exploration tool. While charts and dashboards are visually inspectable and often reviewed collaboratively, **datasets—especially SQL-based virtual datasets—form the critical foundation** of all downstream analytics.
Currently, Superset stores dataset definitions (including SQL queries) in its metadata database. This creates several problems:
* **No native versioning**: Changes to SQL queries overwrite previous definitions, making it difficult or impossible to recover prior logic.
* **Poor traceability**: Teams cannot easily answer *who changed what, when, and why*.
* **Risk of accidental data loss**: SQL logic can be lost due to mistakes, overwrites, or instance-level failures.
* **Lack of review workflow**: Dataset changes cannot be reviewed using standard engineering practices like pull requests.
In modern data teams, SQL is treated as code. However, Superset currently treats it as mutable application state.
This SIP proposes introducing **optional Git-backed version control for Superset datasets**, enabling teams to manage dataset SQL definitions as code.
---
## 2. Problem Statement
There is no built-in mechanism in Apache Superset to:
* Version datasets or dataset SQL definitions
* Track historical changes to datasets
* Enforce review or approval workflows for dataset changes
* Restore previous dataset definitions after accidental modification
This gap becomes more severe as Superset adoption grows in production and regulated environments.
---
## 3. Proposed Solution (High Level)
Introduce an **optional Git integration layer** for dataset management in Apache Superset.
Key idea:
> Treat datasets (especially SQL-based virtual datasets) as version-controlled artifacts stored in a Git repository.
Superset remains the UI and execution layer, while Git becomes the **source of truth** for dataset definitions.
---
## 4. Scope of Versioning
Initial scope (intentionally limited):
* **SQL-based virtual datasets**
* Dataset metadata:
* Dataset name
* Database connection reference
* SQL query
* Schema (optional)
* Description / owner
Out of scope (for initial version):
* Dashboards and charts
* Native table metadata
* Row-level security rules
---
## 5. Architecture Overview
### 5.1 Git Repository Structure (Example)
```text
superset-datasets/
├── sales/
│ ├── daily_revenue.sql
│ ├── daily_revenue.yaml
├── marketing/
│ ├── campaign_performance.sql
│ ├── campaign_performance.yaml
```
* `.sql` files contain dataset queries
* `.yaml` (or `.json`) contains Superset metadata
---
### 5.2 Superset Integration Flow
1. Organization configures a Git provider (GitHub / GitLab / Bitbucket)
2. Superset instance is linked to a repository (read-only or read-write)
3. Each dataset can optionally be marked as **Git-backed**
4. Dataset lifecycle:
* Create / update dataset → change is written to Git
* Commit message is required
* Superset syncs dataset from repository
---
## 6. Change Management Models
### Model A: Superset → Git (Push-based)
* User edits dataset in Superset UI
* Superset requires:
* Commit message
* Branch (optional)
* Superset commits changes to Git
Best for teams starting with Superset-first workflows.
---
### Model B: Git → Superset (Pull-based)
* Dataset SQL edited directly in Git
* Superset periodically syncs (or via webhook)
* Changes are reflected in Superset UI
Best for engineering-heavy teams.
---
## 7. Governance and Controls
Optional enforcement policies:
* Require Git-backed datasets for production workspaces
* Disable direct UI editing (read-only mode)
* Enforce pull-request-based approvals
* Map Superset users to Git identities
---
## 8. Benefits
### 8.1 For Users
* Never lose SQL logic
* Clear change history
* Ability to roll back
* Confidence in dataset correctness
### 8.2 For Organizations
* Improved data governance
* Auditability and compliance
* Standardized dataset review process
* Reduced production incidents
### 8.3 For Superset Ecosystem
* Aligns Superset with modern Analytics Engineering practices
* Encourages GitOps-style workflows
* Makes Superset more enterprise-ready
---
## 9. Backward Compatibility
* Feature is **opt-in**
* Existing datasets continue to work unchanged
* No breaking changes to metadata storage
---
## 10. Implementation Considerations
* Handling merge conflicts
* Secure storage of Git credentials
* Multi-branch support
* Performance impact of sync operations
These concerns can be addressed incrementally.
---
## 11. Future Extensions
* Chart and dashboard versioning
* Dataset lineage visualization
* CI checks for SQL validity
* Integration with dbt and semantic layers
---
## 12. Conclusion
This SIP proposes a focused, opt-in Git integration for dataset versioning in Apache Superset. By treating datasets as code, Superset can significantly improve reliability, collaboration, and trust in analytics workflows.
This feature fills a real gap for production-grade usage and aligns Superset with industry best practices.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-203] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-4858087471
You can already do this via the APIs and CLIs, which you can do today. https://preset.io/blog/version-control-superset-charts-dashboards-superset/
The new DB-based versioning is landing in 7.0.
Trying to think of what the remaining use case is here, and if/why it should be an addition to Superset's core codebase, rather than some form of Extension.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-203] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "JZ6 (via GitHub)" <gi...@apache.org>.
JZ6 commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-4835050170
i think this git backed version control would be a good compliment to the database version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-203] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "sebastianvillarroel (via GitHub)" <gi...@apache.org>.
sebastianvillarroel commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-4169901525
Great SIP! As someone who has worked with Looker and several other BI tools, this is a gap I've thought about a lot. Versioning dataset definitions is a real step forward.
One thing worth considering as a follow-up: knowing *what* changed in a dataset doesn't tell you *what broke downstream*. The propagation of a change is still manual and requires knowing where a dataset is used: charts, dashboards, and models built on top of a modified virtual dataset can silently drift out of sync, and Git history alone doesn't surface that.
Is there a vision for how Superset might eventually tackle change propagation, or is that seen as the domain of external lineage tooling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-203] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "jokester (via GitHub)" <gi...@apache.org>.
jokester commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-4362751833
A way to validate new config, maybe a command to run all queries in it, would be very useful. But to make it implement-able I imagine the data/semantics layers are better decoupled from others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "Shivanigundlapalli (via GitHub)" <gi...@apache.org>.
Shivanigundlapalli commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-3840142376
Hi
I’d like to work on this issue.
Please let me know if it’s okay to proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "andy1xx8 (via GitHub)" <gi...@apache.org>.
andy1xx8 commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-3797561953
Support this featur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-203] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-3960744653
I think as for version control as a SIP, it would be beneficial to consider ALL of Superset's objects (dashboards, charts, etc). Personally, I'd rather see a versioning/rollback scheme within Superset itself, which relates to another "soft delete" feature, and has complications around people rolling back to unsupported formats of objects from older versions. This needs a lot of brainstorming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-3910858909
Anyone can open a PR... no need to ask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-203] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed issue #37427: [SIP-203] Git-backed Version Control for Datasets in Apache Superset
URL: https://github.com/apache/superset/issues/37427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-203] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "sebastianvillarroel (via GitHub)" <gi...@apache.org>.
sebastianvillarroel commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-5009545016
> As already stated, SIP #39492 (DB-based versioning, landing in 7.0) supersedes this, it never went to DISCUSS, and export-API/CLI git workflows cover the remaining use case. Closing as superseded, but thank you!
This is a solid proposal and a long overdue capability for Superset. The SQLAlchemy-Continuum approach for scalar fields plus JSON snapshots for child collections is a pragmatic architecture given the constraints you've documented.
Open Question 3 caught my attention: *"Should dependent charts be notified or validated? What is the user experience when a restored dataset column no longer matches a chart's query?"* This is the harder problem downstream of versioning — knowing a dataset changed is useful, but understanding what broke as a result requires traversing the full dependency graph across datasets, charts, and dashboards. Git history alone doesn't surface that, and neither does database-level version capture.
I've been working on this specific problem with [Puxti](https://getpuxti.com) — a tool that detects semantic changes in data definitions and propagates them downstream as reviewable PRs. The version endpoints you're adding (`GET /versions/`, `POST /restore`) look like useful primitives to hook into. I will be having a look and maybe could share my findings?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "tzercin (via GitHub)" <gi...@apache.org>.
tzercin commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-3798157116
This is a widely discussed topic, please see this article for further information.
https://preset.io/blog/version-control-superset-charts-dashboards-superset/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-203] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-4677137426
https://github.com/apache/superset/issues/39492 seems to effectively supersede this, and it's passing the vote, whereas this hasn't been brought up for discussion yet. Planning to close this unless there's a good counterargument :D
Definitely appreciate the feature and motive - I'm still excited about versioning too :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-203] Git-backed Version Control for Datasets in Apache Superset [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #37427:
URL: https://github.com/apache/superset/issues/37427#issuecomment-4898911636
As already stated, SIP #39492 (DB-based versioning, landing in 7.0) supersedes this, it never went to DISCUSS, and export-API/CLI git workflows cover the remaining use case. Closing as superseded, but thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org