You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@superset.apache.org by ha...@apache.org on 2026/09/09 15:24:44 UTC
(superset) branch master updated: chore(docker): speed up image builds and the compose dev loop (#43916)
This is an automated email from the ASF dual-hosted git repository.
hainenber pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/superset.git
The following commit(s) were added to refs/heads/master by this push:
new d1ad6aeda63 chore(docker): speed up image builds and the compose dev loop (#43916)
d1ad6aeda63 is described below
commit d1ad6aeda6332dad652b33b46ae6f82e2c201826
Author: Joe Li <jo...@preset.io>
AuthorDate: Wed Sep 9 08:24:37 2026 -0700
chore(docker): speed up image builds and the compose dev loop (#43916)
Co-authored-by: Claude Fable 5.1 <no...@anthropic.com>
---
Dockerfile | 81 +++++++++++++++-------
UPDATING.md | 2 +
docker-compose-light.yml | 1 +
docker-compose.yml | 3 +
docker/.env | 6 ++
docker/docker-init.sh | 36 +++++++++-
docs/admin_docs/installation/docker-compose.mdx | 4 +-
.../contributing/development-setup.md | 7 +-
8 files changed, 110 insertions(+), 30 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index 0c0c7a9c849..d0c370e2ca1 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -231,20 +231,6 @@ RUN /app/docker/apt-install.sh \
# The database file will be created at runtime when examples are loaded from Parquet files
RUN mkdir -p /app/data && chown -R superset:superset /app/data
-# Copy compiled things from previous stages
-COPY --from=superset-node /app/superset/static/assets superset/static/assets
-# Copy service.worker.js optionall as it doesn't exist when DEV_MODE=true
-COPY --from=superset-node /app/superset/static/service-worker.j[s] superset/static/service-worker.js
-
-# TODO, when the next version comes out, use --exclude superset/translations
-COPY superset superset
-# TODO in the meantime, remove the .po files
-RUN rm superset/translations/*/*/*.po
-
-# Merging translations from backend and frontend stages
-COPY --from=superset-node /app/superset/translations superset/translations
-COPY --from=python-translation-compiler /app/translations_mo superset/translations
-
# --- Realtime WebSocket server (part of the official image) ---------------
# The realtime transport (superset-websocket) is a Node service, bundled by
# esbuild into a single self-contained file. Copy the Node runtime plus that
@@ -267,7 +253,10 @@ EXPOSE ${SUPERSET_PORT}
######################################################################
FROM python-common AS lean
-# Install Python dependencies using docker/pip-install.sh
+# Install Python dependencies using docker/pip-install.sh.
+# Requirements are installed *before* the application source is copied
+# below so that source-only changes don't bust this (slow, network-bound)
+# cache layer or defeat --cache-from.
COPY requirements/base.txt requirements/
# Copy superset-core package needed for editable install in base.txt
@@ -275,9 +264,27 @@ COPY superset-core superset-core
RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
/app/docker/pip-install.sh --requires-build-essential -r requirements/base.txt
-# Install the superset package
+
+# Copy compiled frontend assets and application source now that
+# dependencies have been resolved and cached above.
+COPY --from=superset-node /app/superset/static/assets superset/static/assets
+# Copy service.worker.js optionally as it doesn't exist when DEV_MODE=true
+COPY --from=superset-node /app/superset/static/service-worker.j[s] superset/static/service-worker.js
+
+# TODO, when the next version comes out, use --exclude superset/translations
+COPY superset superset
+# TODO in the meantime, remove the .po files
+RUN rm superset/translations/*/*/*.po
+
+# Merging translations from backend and frontend stages
+COPY --from=superset-node /app/superset/translations superset/translations
+COPY --from=python-translation-compiler /app/translations_mo superset/translations
+
+# Install the superset package itself. --no-deps because its dependencies
+# were already installed from requirements/base.txt above, so this layer
+# stays fast even though the source copy above changes on every edit.
RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
- uv pip install -e .
+ uv pip install -e . --no-deps
RUN python -m compileall /app/superset
USER superset
@@ -293,22 +300,46 @@ RUN /app/docker/apt-install.sh \
pkg-config \
default-libmysqlclient-dev
-# Copy development requirements and install them
+# Copy development requirements and install them *before* the application
+# source is copied below, so source-only edits don't bust this cache layer.
COPY requirements/*.txt requirements/
# Copy local packages needed for editable installs in development.txt
COPY superset-core superset-core
COPY superset-extensions-cli superset-extensions-cli
-# Install Python dependencies using docker/pip-install.sh
+# requirements/development.txt is generated by `uv pip compile` and embeds
+# `-e .` (an editable install of this same package) as its first line. That
+# self-reference needs the full superset/ source tree, which hasn't been
+# copied in yet at this point, so it's stripped here; the real editable
+# install of `.` runs below, once the source is present.
RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
- /app/docker/pip-install.sh --requires-build-essential -r requirements/development.txt
-# Install the superset package
-RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
- uv pip install -e .
+ grep -vxF -- "-e ." requirements/development.txt > requirements/development-deps.txt && \
+ /app/docker/pip-install.sh --requires-build-essential -r requirements/development-deps.txt
-RUN uv pip install .[postgres]
-RUN python -m compileall /app/superset
+# Copy compiled frontend assets and application source now that
+# dependencies have been resolved and cached above.
+COPY --from=superset-node /app/superset/static/assets superset/static/assets
+# Copy service.worker.js optionally as it doesn't exist when DEV_MODE=true
+COPY --from=superset-node /app/superset/static/service-worker.j[s] superset/static/service-worker.js
+
+# TODO, when the next version comes out, use --exclude superset/translations
+COPY superset superset
+# TODO in the meantime, remove the .po files
+RUN rm superset/translations/*/*/*.po
+
+# Merging translations from backend and frontend stages
+COPY --from=superset-node /app/superset/translations superset/translations
+COPY --from=python-translation-compiler /app/translations_mo superset/translations
+
+# Install the superset package together with its postgres extra, using the
+# same uv cache mount as the requirements install above. --no-deps because
+# all dependencies (including the postgres extra's psycopg2-binary) are
+# already installed from requirements/development.txt above.
+# NOTE: source is bind-mounted over /app/superset in DEV_MODE, so a
+# compileall pass here would be wasted work; unlike `lean`, `dev` skips it.
+RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
+ uv pip install -e .[postgres] --no-deps
USER superset
diff --git a/UPDATING.md b/UPDATING.md
index 4c998c267db..a163ad48216 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -1233,6 +1233,8 @@ Custom time ranges that use the "Now" or "Today" anchor (for the Start, End, or
Charts and dashboards using these anchors will compute a different (correct) timestamp after upgrading; if a chart's filters or drill-downs were tuned to compensate for the old offset, review them after upgrading.
+- [43916](https://github.com/apache/superset/pull/43916): The `docker-compose` dev loop now skips re-running `superset load_examples` on every `docker compose up` once the example data and dashboards are present in the databases (set `SUPERSET_FORCE_LOAD_EXAMPLES=yes` to reload them anyway), and the `superset-node` service now defaults `DISABLE_TS_CHECKER=true` like `docker-compose-light.yml` already did, skipping webpack's TypeScript type-checking pass in dev by default.
+
## 6.1.0
### ClickHouse minimum driver version bump
diff --git a/docker-compose-light.yml b/docker-compose-light.yml
index 3be24237eb9..52bb23c5b51 100644
--- a/docker-compose-light.yml
+++ b/docker-compose-light.yml
@@ -138,6 +138,7 @@ services:
condition: service_started
volumes: *superset-volumes
environment:
+ SUPERSET_FORCE_LOAD_EXAMPLES: "${SUPERSET_FORCE_LOAD_EXAMPLES:-}"
DATABASE_HOST: db-light
DATABASE_DB: superset_light
POSTGRES_DB: superset_light
diff --git a/docker-compose.yml b/docker-compose.yml
index d3a79aeef90..3d3ff242e19 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -183,6 +183,8 @@ services:
condition: service_started
user: *superset-user
volumes: *superset-volumes
+ environment:
+ SUPERSET_FORCE_LOAD_EXAMPLES: "${SUPERSET_FORCE_LOAD_EXAMPLES:-}"
healthcheck:
disable: true
@@ -202,6 +204,7 @@ services:
BUILD_SUPERSET_FRONTEND_IN_DOCKER: true
NPM_RUN_PRUNE: false
SCARF_ANALYTICS: "${SCARF_ANALYTICS:-}"
+ DISABLE_TS_CHECKER: "${DISABLE_TS_CHECKER:-true}"
# configuring the dev-server to use the host.docker.internal to connect to the backend
superset: "http://superset:8088"
# Webpack dev server must bind to 0.0.0.0 to be accessible from outside the container
diff --git a/docker/.env b/docker/.env
index 7cd78cf2903..25fcc776b1d 100644
--- a/docker/.env
+++ b/docker/.env
@@ -73,6 +73,12 @@ SUPERSET_ENV=development
# Swagger UI is opt-in (off by default); enable it for local development.
SUPERSET_ENABLE_SWAGGER_UI=true
SUPERSET_LOAD_EXAMPLES=yes
+# Once the example data and dashboards are present in the databases,
+# `docker-init.sh` skips `superset load_examples` on later runs. Set to "yes"
+# (or run `SUPERSET_FORCE_LOAD_EXAMPLES=yes docker compose up`) to reload the
+# examples anyway, e.g. after changing the example datasets or after a partial
+# load.
+#SUPERSET_FORCE_LOAD_EXAMPLES=no
CYPRESS_CONFIG=false
SUPERSET_PORT=8088
MAPBOX_API_KEY=''
diff --git a/docker/docker-init.sh b/docker/docker-init.sh
index e4b25b5b187..e2cfa617470 100755
--- a/docker/docker-init.sh
+++ b/docker/docker-init.sh
@@ -66,14 +66,44 @@ echo_step "3" "Starting" "Setting up roles and perms"
superset init
echo_step "3" "Complete" "Setting up roles and perms"
+# Loading examples parses and inserts every example dataset, chart and
+# dashboard and is one of the slowest steps of `docker compose up`. Rather
+# than trusting a marker file (which goes stale as soon as the database volume
+# is recreated), ask the databases themselves: when both the example data and
+# the dashboards imported from it are present, the previous load completed and
+# there is nothing left to redo. Any failure here (missing tables, unreachable
+# database, import error) simply reports "not loaded" so the full load runs.
+examples_already_loaded() {
+ python - <<'PY' 2>/dev/null
+import sys
+
+from superset.app import create_app
+from superset.sql.parse import Table
+
+app = create_app()
+with app.app_context():
+ from superset import db
+ from superset.models.dashboard import Dashboard
+ from superset.utils.database import get_example_database
+
+ has_dashboard = (
+ db.session.query(Dashboard).filter_by(slug="world_health").first() is not None
+ )
+ has_data = get_example_database().has_table(Table("wb_health_population"))
+ sys.exit(0 if has_dashboard and has_data else 1)
+PY
+}
+
if [ "$SUPERSET_LOAD_EXAMPLES" = "yes" ]; then
- # Load some data to play with
echo_step "4" "Starting" "Loading examples"
-
- # If Cypress run which consumes superset_test_config – load required data for tests
+ # Cypress runs always load, since they need a distinct set of test data
+ # (`--load-test-data`) in a separate database. Set
+ # SUPERSET_FORCE_LOAD_EXAMPLES=yes to reload the examples regardless.
if [ "$CYPRESS_CONFIG" == "true" ]; then
superset load_examples --load-test-data
+ elif [ "$SUPERSET_FORCE_LOAD_EXAMPLES" != "yes" ] && examples_already_loaded; then
+ echo "Examples already loaded, skipping (set SUPERSET_FORCE_LOAD_EXAMPLES=yes to reload them)"
else
superset load_examples
fi
diff --git a/docs/admin_docs/installation/docker-compose.mdx b/docs/admin_docs/installation/docker-compose.mdx
index 17d1433feac..d7c987755ff 100644
--- a/docs/admin_docs/installation/docker-compose.mdx
+++ b/docs/admin_docs/installation/docker-compose.mdx
@@ -196,7 +196,9 @@ One important variable is `SUPERSET_LOAD_EXAMPLES` which determines whether the
container will populate example data and visualizations into the metadata database. These examples
are helpful for learning and testing out Superset but unnecessary for experienced users and
production deployments. The loading process can sometimes take a few minutes and a good amount of
-CPU, so you may want to disable it on a resource-constrained device.
+CPU, so you may want to disable it on a resource-constrained device. Once the example data and
+dashboards are present in the databases, later `superset_init` runs skip loading them; run
+`SUPERSET_FORCE_LOAD_EXAMPLES=yes docker compose up` to reload the examples anyway.
For more advanced or dynamic configurations that are typically managed in a `superset_config.py` file
located in your `PYTHONPATH`, note that it can be done by providing a
diff --git a/docs/developer_docs/contributing/development-setup.md b/docs/developer_docs/contributing/development-setup.md
index 2992aeed042..ec3eb44ff86 100644
--- a/docs/developer_docs/contributing/development-setup.md
+++ b/docs/developer_docs/contributing/development-setup.md
@@ -99,11 +99,16 @@ Affecting the Docker build process:
- **INCLUDE_CHROMIUM (default=false):** whether to include the Chromium headless browser in the build
- **BUILD_TRANSLATIONS(default=false):** whether to compile the translations from the .po files available
- **SUPERSET_LOAD_EXAMPLES (default=yes):** whether to load the examples into the database upon startup,
- save some precious time on startup by `SUPERSET_LOAD_EXAMPLES=no docker compose up`
+ save some precious time on startup by `SUPERSET_LOAD_EXAMPLES=no docker compose up`. Once the example
+ data and dashboards are present in the databases, later `docker compose up` runs skip loading
+ them; run `SUPERSET_FORCE_LOAD_EXAMPLES=yes docker compose up` to reload the examples anyway.
- **SUPERSET_LOG_LEVEL (default=info)**: Can be set to debug, info, warning, error, critical
for more verbose logging
- **SUPERSET_DEBUG_ENABLED (default=false)**: Enable Werkzeug debugger with interactive console.
Set to `true` for debugging: `SUPERSET_DEBUG_ENABLED=true docker compose up`
+- **DISABLE_TS_CHECKER (default=true)**: whether the `superset-node` webpack dev server skips
+ TypeScript type-checking, which speeds up rebuilds and saves several GB of memory. Set to
+ `false` to have webpack surface type errors during development.
For more env vars that affect your configuration, see this
[superset_config.py](https://github.com/apache/superset/blob/master/docker/pythonpath_dev/superset_config.py)