You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mistercrunch (via GitHub)" <gi...@apache.org> on 2025/07/28 04:50:01 UTC
[PR] feat: migrate examples from Python to YAML format with enhanced CLI [superset]
mistercrunch opened a new pull request, #34346:
URL: https://github.com/apache/superset/pull/34346
## Summary
Migrates Superset's example data system from Python-based scripts to YAML configuration files, providing a cleaner, more maintainable approach to managing example datasets, charts, and dashboards.
## Key Changes
### 🔄 Migration to YAML Format
- Converted 9 Python example modules to YAML configurations
- Exported existing examples from database and added as YAML files:
- 11 dashboards (USA Births Names, World Bank's Data, etc.)
- 115 charts
- 25 datasets
- Moved test-specific fixtures to `tests/fixtures/examples/`
- Removed theme_id from dashboard exports for compatibility
### 🎛️ New CLI Structure
- **New command group**: `superset examples` with subcommands:
- `load` - Load example data (replaces `load-examples`)
- `clear-old` - Remove old Python-based examples
- `clear` - Placeholder for future YAML clearing
- `reload` - Clear and reload in one command
- **Backwards compatibility**: `superset load-examples` still works with deprecation warning
- **Safety mechanism**: Detects old examples and preserves them to avoid data loss
### 🐛 Fixes
- Fixed JSON data loading - examples can now load `.json.gz` files from CDN
- Fixed Docker compose configuration for isolated development
- Fixed webpack WebSocket configuration for different ports
### 📊 Enhanced Logging
- Import operations now log what's being created vs updated:
- "Creating new dashboard: Sales Dashboard"
- "Updating existing chart: World's Population"
- Provides clear visibility into the import process
### 🏗️ Architecture Improvements
- Moved import logging to individual import functions (DRY principle)
- Non-destructive migration approach - no user data is deleted
- Deterministic UUID generation for consistent example data
## Testing
- Tested migration from old Python examples to new YAML format
- Verified safety mechanism prevents accidental data overwrites
- Confirmed backwards compatibility with deprecated command
- All pre-commit checks pass
## Documentation
- Updated installation docs to use new CLI commands
- Added deprecation notice to UPDATING.md
- Updated development documentation
## Breaking Changes
None - the old `load-examples` command continues to work with a deprecation warning.
## Migration Path
For users with existing Python-based examples:
1. Run `superset examples clear-old --confirm` to remove old examples
2. Run `superset examples load` to load new YAML-based examples
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: migrate examples from Python to YAML format with enhanced CLI [superset]
Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on PR #34346:
URL: https://github.com/apache/superset/pull/34346#issuecomment-3125457412
Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment `/korbit-review`.
> Your admin can change your review schedule in the [Korbit Console](https://app.korbit.ai/settings)
[](ecbd74c9-5975-4944-94a4-473d786273fb)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: migrate examples from Python to YAML format with enhanced CLI [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #34346:
URL: https://github.com/apache/superset/pull/34346#discussion_r2243429786
##########
UPDATING.md:
##########
@@ -23,6 +23,8 @@ This file documents any backwards-incompatible changes in Superset and
assists people when migrating to a new version.
## Next
+- [XXXXX](https://github.com/apache/superset/pull/XXXXX) The examples system has been migrated from Python-based scripts to YAML configuration files. The CLI command `superset load-examples` has been deprecated in favor of `superset examples load`. The old command still works but will show a deprecation warning. Additional example management commands are available under `superset examples` including `clear-old` and `reload`. If you have old Python-based examples loaded, the new YAML-based examples will not load automatically to preserve your existing data. To migrate to the new examples, run `superset examples clear-old --confirm` followed by `superset examples load`.
Review Comment:
```suggestion
- [34346](https://github.com/apache/superset/pull/34346) The examples system has been migrated from Python-based scripts to YAML configuration files. The CLI command `superset load-examples` has been deprecated in favor of `superset examples load`. The old command still works but will show a deprecation warning. Additional example management commands are available under `superset examples` including `clear-old` and `reload`. If you have old Python-based examples loaded, the new YAML-based examples will not load automatically to preserve your existing data. To migrate to the new examples, run `superset examples clear-old --confirm` followed by `superset examples load`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: migrate examples from Python to YAML format with enhanced CLI [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #34346:
URL: https://github.com/apache/superset/pull/34346#issuecomment-3143162238
This is harder than it seemed at first ... Cypress tests have hard-coded ids, somehow this works even with auto-numbers ... Loading order must be deterministic enough to support hundreds of test-runs per day. Loading yaml the ids don't match and we don't have uuid-support on our endpoints, at least not yet.
Solution to move forward would be to add uuid-support for a bunch of endpoints, or moving away from Cypress. Both are worthwhile but beyond the scope of what I'm willing to do here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: migrate examples from Python to YAML format with enhanced CLI [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #34346:
URL: https://github.com/apache/superset/pull/34346#issuecomment-3137915698
Yeah economics of software development are changing ... this effort probably not worth it if it's going to take someone weeks, but as a background process with AI it's a no-brainer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: migrate examples from Python to YAML format with enhanced CLI [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #34346:
URL: https://github.com/apache/superset/pull/34346#issuecomment-3137263888
Oh man... good AI use case. I've been wanting to do this for years. Happy to stamp when it passes CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: migrate examples from Python to YAML format with enhanced CLI [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch closed pull request #34346: feat: migrate examples from Python to YAML format with enhanced CLI
URL: https://github.com/apache/superset/pull/34346
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org