You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@superset.apache.org by ma...@apache.org on 2025/08/23 23:17:40 UTC
(superset) 01/01: feat: Integrate Superset Showtime GitHub Actions workflows
This is an automated email from the ASF dual-hosted git repository.
maximebeauchemin pushed a commit to branch showtime_gha
in repository https://gitbox.apache.org/repos/asf/superset.git
commit 0dfc43e1f076e9233dd71dfa979a5865c2a5ca9d
Author: Maxime Beauchemin <ma...@gmail.com>
AuthorDate: Sat Aug 23 16:02:36 2025 -0700
feat: Integrate Superset Showtime GitHub Actions workflows
Replaces legacy ephemeral environment system with the new Superset Showtime tool (https://github.com/mistercrunch/superset-showtime), providing automated PR environment management using Docker containers deployed to AWS ECS.
## Key Changes
### New Showtime Workflows
- `showtime-trigger.yml`: Handles all PR events (labeled, synchronize, closed) with intelligent build detection
- `showtime-cleanup.yml`: Scheduled cleanup every 6 hours with manual trigger support
### Legacy Workflow Deprecation
- Added deprecation notices to `ephemeral-env.yml` and `ephemeral-env-pr-close.yml`
- Clear migration guidance from "testenv-up" to "🎪 trigger-start" labels
- Both systems coexist during transition period
### Architecture Improvements
- **Simplified Logic**: GitHub Actions just triggers, showtime CLI handles all decisions
- **Smart Building**: Only builds Docker images when code changes, not label changes
- **DuckDB Support**: Configured with `LOAD_EXAMPLES_DUCKDB=true` build arg
- **Environment Variables**: Requires `SUPERSET__SQLALCHEMY_EXAMPLES_URI` in ECS (showtime CLI responsibility)
## Testing
Simple 3-step test on PR 34831:
1. Set label: `gh pr edit 34831 --add-label "🎪 trigger-start"`
2. Run command: `gh workflow run showtime-trigger.yml --ref showtime_gha --field pr_number=34831`
3. Expected outcome: Environment accessible with DuckDB examples
## Benefits
- **Reliability**: Dedicated Python package vs complex YAML logic
- **Maintainability**: All environment logic centralized in showtime CLI
- **Efficiency**: Conditional builds and smart resource management
- **Testability**: Manual workflow_dispatch for development and debugging
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <no...@anthropic.com>
---
.github/workflows/ephemeral-env-pr-close.yml | 10 +-
.github/workflows/ephemeral-env.yml | 17 +++-
.github/workflows/showtime-cleanup.yml | 51 ++++++++++
.github/workflows/showtime-trigger.yml | 135 +++++++++++++++++++++++++++
4 files changed, 208 insertions(+), 5 deletions(-)
diff --git a/.github/workflows/ephemeral-env-pr-close.yml b/.github/workflows/ephemeral-env-pr-close.yml
index 60cf75dac8..fe65c6fa64 100644
--- a/.github/workflows/ephemeral-env-pr-close.yml
+++ b/.github/workflows/ephemeral-env-pr-close.yml
@@ -1,4 +1,10 @@
-name: Cleanup ephemeral envs (PR close)
+name: Cleanup ephemeral envs (PR close) [DEPRECATED]
+
+# ⚠️ DEPRECATION NOTICE ⚠️
+# This workflow is deprecated and will be removed in a future version.
+# The new Superset Showtime workflow handles cleanup automatically.
+# See .github/workflows/showtime.yml and showtime-cleanup.yml for replacements.
+# Migration guide: https://github.com/mistercrunch/superset-showtime
on:
pull_request_target:
@@ -71,5 +77,5 @@ jobs:
issue_number: ${{ github.event.number }},
owner: context.repo.owner,
repo: context.repo.repo,
- body: 'Ephemeral environment shutdown and build artifacts deleted.'
+ body: '⚠️ **DEPRECATED WORKFLOW** - Ephemeral environment shutdown and build artifacts deleted. Please migrate to the new Superset Showtime system for future PRs.'
})
diff --git a/.github/workflows/ephemeral-env.yml b/.github/workflows/ephemeral-env.yml
index af1dfaff80..48f3669886 100644
--- a/.github/workflows/ephemeral-env.yml
+++ b/.github/workflows/ephemeral-env.yml
@@ -1,4 +1,12 @@
-name: Ephemeral env workflow
+name: Ephemeral env workflow [DEPRECATED]
+
+# ⚠️ DEPRECATION NOTICE ⚠️
+# This workflow is deprecated and will be removed in a future version.
+# Please use the new Superset Showtime workflow instead:
+# - Use label "🎪 trigger-start" instead of "testenv-up"
+# - Showtime provides better reliability and easier management
+# - See .github/workflows/showtime.yml for the replacement
+# - Migration guide: https://github.com/mistercrunch/superset-showtime
# Example manual trigger:
# gh workflow run ephemeral-env.yml --ref fix_ephemerals --field label_name="testenv-up" --field issue_number=666
@@ -126,8 +134,11 @@ jobs:
throw new Error("Issue number is not available.");
}
- const body = `@${user} Processing your ephemeral environment request [here](${workflowUrl}).` +
- ` Action: **${action}**.` +
+ const body = `⚠️ **DEPRECATED WORKFLOW** ⚠️\n\n@${user} This workflow is deprecated! Please use the new **Superset Showtime** system instead:\n\n` +
+ `- Replace "testenv-up" label with "🎪 trigger-start"\n` +
+ `- Better reliability and easier management\n` +
+ `- See https://github.com/mistercrunch/superset-showtime for details\n\n` +
+ `Processing your ephemeral environment request [here](${workflowUrl}). Action: **${action}**.` +
` More information on [how to use or configure ephemeral environments]` +
`(https://superset.apache.org/docs/contributing/howtos/#github-ephemeral-environments)`;
diff --git a/.github/workflows/showtime-cleanup.yml b/.github/workflows/showtime-cleanup.yml
new file mode 100644
index 0000000000..8cbec01c0c
--- /dev/null
+++ b/.github/workflows/showtime-cleanup.yml
@@ -0,0 +1,51 @@
+name: 🎪 Showtime Cleanup
+
+# Scheduled cleanup of expired environments
+on:
+ schedule:
+ - cron: '0 */6 * * *' # Every 6 hours
+
+ # Manual trigger for testing
+ workflow_dispatch:
+ inputs:
+ max_age_hours:
+ description: 'Maximum age in hours before cleanup'
+ required: false
+ default: '48'
+ type: string
+ pattern: '^[0-9]+$'
+
+# Common environment variables
+env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+ AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+ AWS_REGION: ${{ vars.AWS_REGION || 'us-west-2' }}
+ GITHUB_ORG: ${{ github.repository_owner }}
+ GITHUB_REPO: ${{ github.event.repository.name }}
+
+jobs:
+ cleanup-expired:
+ name: Clean up expired showtime environments
+ runs-on: ubuntu-latest
+
+ permissions:
+ contents: read
+ pull-requests: write
+
+ steps:
+ - name: Install Superset Showtime
+ run: pip install superset-showtime
+
+ - name: Cleanup expired environments
+ run: |
+ MAX_AGE="${{ github.event.inputs.max_age_hours || '48' }}"
+
+ # Validate max_age is numeric
+ if [[ ! "$MAX_AGE" =~ ^[0-9]+$ ]]; then
+ echo "❌ Invalid max_age_hours format: $MAX_AGE (must be numeric)"
+ exit 1
+ fi
+
+ echo "Cleaning up environments older than ${MAX_AGE}h"
+ python -m showtime cleanup --older-than "${MAX_AGE}h"
diff --git a/.github/workflows/showtime-trigger.yml b/.github/workflows/showtime-trigger.yml
new file mode 100644
index 0000000000..db03936ab5
--- /dev/null
+++ b/.github/workflows/showtime-trigger.yml
@@ -0,0 +1,135 @@
+name: 🎪 Superset Showtime
+
+# Ultra-simple: just sync on any PR state change
+on:
+ pull_request_target:
+ types: [labeled, unlabeled, synchronize, closed]
+
+ # Manual testing
+ workflow_dispatch:
+ inputs:
+ pr_number:
+ description: 'PR number to sync'
+ required: true
+ type: number
+ sha:
+ description: 'Specific SHA to deploy (optional, defaults to latest)'
+ required: false
+ type: string
+ pattern: '^[a-f0-9]{40}$'
+
+# Common environment variables for all jobs
+env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+ AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+ AWS_REGION: us-west-2
+ GITHUB_ORG: ${{ github.repository_owner }}
+ GITHUB_REPO: ${{ github.event.repository.name }}
+ GITHUB_ACTOR: ${{ github.actor }}
+ DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+
+jobs:
+ sync:
+ name: 🎪 Sync PR to desired state
+ runs-on: ubuntu-latest
+
+ permissions:
+ contents: read
+ pull-requests: write
+
+ steps:
+ - name: Install Superset Showtime
+ run: pip install superset-showtime
+
+ - name: Check what actions are needed
+ id: check
+ run: |
+ # Bulletproof PR number extraction
+ if [[ -n "${{ github.event.pull_request.number }}" ]]; then
+ PR_NUM="${{ github.event.pull_request.number }}"
+ elif [[ -n "${{ github.event.inputs.pr_number }}" ]]; then
+ PR_NUM="${{ github.event.inputs.pr_number }}"
+ else
+ echo "❌ No PR number found in event or inputs"
+ exit 1
+ fi
+
+ echo "Using PR number: $PR_NUM"
+ OUTPUT=$(python -m showtime sync $PR_NUM --check-only)
+ echo "$OUTPUT"
+
+ # Extract outputs for conditional steps
+ ACTION=$(echo "$OUTPUT" | grep "action_needed=" | cut -d'=' -f2)
+ BUILD=$(echo "$OUTPUT" | grep "build_needed=" | cut -d'=' -f2)
+ DEPLOY=$(echo "$OUTPUT" | grep "deploy_needed=" | cut -d'=' -f2)
+
+ echo "action_needed=$ACTION" >> $GITHUB_OUTPUT
+ echo "build_needed=$BUILD" >> $GITHUB_OUTPUT
+ echo "deploy_needed=$DEPLOY" >> $GITHUB_OUTPUT
+ echo "pr_number=$PR_NUM" >> $GITHUB_OUTPUT
+
+ - name: Setup Docker Environment
+ if: steps.check.outputs.build_needed == 'true'
+ uses: ./.github/actions/setup-docker
+ with:
+ dockerhub-user: ${{ env.DOCKERHUB_USER }}
+ dockerhub-token: ${{ env.DOCKERHUB_TOKEN }}
+ build: "true"
+
+ - name: Setup supersetbot
+ if: steps.check.outputs.build_needed == 'true'
+ uses: ./.github/actions/setup-supersetbot/
+
+ - name: Get PR SHA for build
+ id: sha
+ if: steps.check.outputs.build_needed == 'true'
+ run: |
+ # Simple SHA handling - use input or default to PR head
+ if [[ -n "${{ github.event.inputs.sha }}" ]]; then
+ SHA="${{ github.event.inputs.sha }}"
+ echo "Using override SHA: $SHA"
+ elif [[ -n "${{ github.event.pull_request.head.sha }}" ]]; then
+ SHA="${{ github.event.pull_request.head.sha }}"
+ echo "Using PR head SHA: $SHA"
+ else
+ echo "❌ No SHA available from PR head or inputs"
+ exit 1
+ fi
+
+ # Validate SHA format (40-char hex)
+ if [[ ! "$SHA" =~ ^[a-f0-9]{40}$ ]]; then
+ echo "❌ Invalid SHA format: $SHA"
+ exit 1
+ fi
+
+ echo "build_sha=$SHA" >> $GITHUB_OUTPUT
+
+ - name: Checkout PR code
+ if: steps.check.outputs.build_needed == 'true'
+ uses: actions/checkout@v4
+ with:
+ ref: ${{ steps.sha.outputs.build_sha }}
+ persist-credentials: false
+
+ - name: Build Docker image
+ if: steps.check.outputs.build_needed == 'true'
+ run: |
+ supersetbot docker \
+ --push \
+ --load \
+ --preset ci \
+ --platform linux/amd64 \
+ --context-ref "${{ steps.sha.outputs.build_sha }}" \
+ --extra-flags "--build-arg INCLUDE_CHROMIUM=false --build-arg LOAD_EXAMPLES_DUCKDB=true"
+
+ - name: Execute sync deployment
+ if: steps.check.outputs.deploy_needed == 'true'
+ run: |
+ PR_NUM="${{ steps.check.outputs.pr_number }}"
+ if [[ -n "${{ github.event.inputs.sha }}" ]]; then
+ python -m showtime sync $PR_NUM --deploy --sha "${{ github.event.inputs.sha }}"
+ else
+ python -m showtime sync $PR_NUM --deploy
+ fi