You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mistercrunch (via GitHub)" <gi...@apache.org> on 2025/01/29 02:57:50 UTC
[PR] feat: improve docker images to support running in userspace [superset]
mistercrunch opened a new pull request, #32017:
URL: https://github.com/apache/superset/pull/32017
While digging into fixing ephemeral envs, I realized that running docker-compose as `superset` didn't work out of the box. `docker-compose.yml` sets `root` as the user which goes around the issue while running `compose`. When switching to user `superset`, there were some permission issues around excecuting certain scripts and such.
Clearly it's better/more-secure to run the app as a non-root user. So here I'm making some changes and simplifying things, paying attention to a few layers, and avoiding doing any sorts of installation of anything while starting the app as the `superset` user in the tree of entrypoint scripts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #32017:
URL: https://github.com/apache/superset/pull/32017#issuecomment-2620566710
/testenv up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #32017:
URL: https://github.com/apache/superset/pull/32017#issuecomment-2620994921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32017:
URL: https://github.com/apache/superset/pull/32017#issuecomment-2620568022
@mistercrunch Processing your ephemeral environment request [here](https://github.com/apache/superset/actions/runs/13023951433).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #32017:
URL: https://github.com/apache/superset/pull/32017#discussion_r1933148120
##########
Dockerfile:
##########
@@ -76,8 +76,7 @@ COPY superset-frontend /app/superset-frontend
FROM superset-node-ci AS superset-node
# Build the frontend if not in dev mode
-RUN --mount=type=cache,target=/app/superset-frontend/.temp_cache \
- --mount=type=cache,target=/root/.npm \
+RUN --mount=type=cache,target=$/root/.npm \
Review Comment:
Here caching the cache mount was preventing the docker-compose-level mount from hooking up the cache to the host. Getting a nice perf boost by letting the host mount do what it's supposed to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #32017:
URL: https://github.com/apache/superset/pull/32017#discussion_r1933149737
##########
Dockerfile:
##########
@@ -100,21 +99,14 @@ RUN if [ "$BUILD_TRANSLATIONS" = "true" ]; then \
# Base python layer
######################################################################
FROM python:${PY_VER} AS python-base
-ARG BUILD_TRANSLATIONS="false" # Include translations in the final build
-ENV BUILD_TRANSLATIONS=${BUILD_TRANSLATIONS}
-ARG DEV_MODE="false" # Skip frontend build in dev mode
-ENV DEV_MODE=${DEV_MODE}
-
-ENV LANG=C.UTF-8 \
Review Comment:
LANG/ LC_ALL are not needed anymore as of 3.9 I believe. Other ENVs we don't need in this layer. Turns out ENV don't carry through a layer with a different base and led to some confusion on my part. As it was ENV would not carry through to the final layers, but they do when located in python-common
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: eph env + improve docker images to run in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #32017:
URL: https://github.com/apache/superset/pull/32017#issuecomment-2640705934
Can you confirm you have issues when using `localhost:9000` if so I can't recreate here. Also noting that there may be some false-negative log messages around the `/ws` endpoint. I haven't had the time to look into those, but should get to the bottom of it/tackled them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #32017:
URL: https://github.com/apache/superset/pull/32017#issuecomment-2620993181
OMG a working ephemeral env! ^^^^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #32017:
URL: https://github.com/apache/superset/pull/32017#discussion_r1933218659
##########
docker/docker-bootstrap.sh:
##########
@@ -34,7 +37,8 @@ if [ "$CYPRESS_CONFIG" == "true" ]; then
export SUPERSET__SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://superset:superset@db:5432/superset_cypress
PORT=8081
fi
-if [[ "$DATABASE_DIALECT" == postgres* ]] ; then
+if [[ "$DATABASE_DIALECT" == postgres* ]] && [ "$(whoami)" = "root" ]; then
Review Comment:
We need to keep this for the use case of other docker-compose files that mount the docker/ folder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #32017:
URL: https://github.com/apache/superset/pull/32017#discussion_r1933197642
##########
Dockerfile:
##########
@@ -238,12 +236,13 @@ RUN /app/docker/apt-install.sh \
# Copy development requirements and install them
COPY requirements/*.txt requirements/
# Install Python dependencies using docker/pip-install.sh
-RUN --mount=type=cache,target=/root/.cache/uv \
+RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
/app/docker/pip-install.sh --requires-build-essential -r requirements/development.txt
# Install the superset package
-RUN --mount=type=cache,target=/root/.cache/uv \
+RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
uv pip install .
+RUN uv pip install .[postgres]
Review Comment:
going back to installing postgres client as part of the build process for the `dev` and `ci` layer (still not packaging the client on `lean`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #32017:
URL: https://github.com/apache/superset/pull/32017#discussion_r1933160021
##########
Dockerfile:
##########
@@ -159,13 +138,20 @@ RUN if [ "$BUILD_TRANSLATIONS" = "true" ]; then \
# Python APP common layer
######################################################################
FROM python-base AS python-common
+
+ENV SUPERSET_HOME="/app/superset_home" \
Review Comment:
these now carry through to the 3 final images
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32017:
URL: https://github.com/apache/superset/pull/32017#issuecomment-2620595733
@mistercrunch Ephemeral environment spinning up at http://35.93.23.171:8080. Credentials are `admin`/`admin`. Please allow several minutes for bootstrapping and startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #32017:
URL: https://github.com/apache/superset/pull/32017#discussion_r1933148430
##########
Dockerfile:
##########
@@ -100,21 +99,14 @@ RUN if [ "$BUILD_TRANSLATIONS" = "true" ]; then \
# Base python layer
######################################################################
FROM python:${PY_VER} AS python-base
-ARG BUILD_TRANSLATIONS="false" # Include translations in the final build
Review Comment:
Moving this section to the python-common layer that's used for the app itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #32017:
URL: https://github.com/apache/superset/pull/32017#discussion_r1933216960
##########
docker/docker-bootstrap.sh:
##########
@@ -18,9 +18,12 @@
set -eo pipefail
+# Ensure the temporary directory exists
+mkdir -p /tmp/superset_temp
+
# Make python interactive
if [ "$DEV_MODE" == "true" ]; then
- if command -v uv > /dev/null 2>&1; then
+ if [ "$(whoami)" = "root" ] && command -v uv > /dev/null 2>&1; then
Review Comment:
only doing this when `root` as `superset` doesn't have the rights to touch the python packaging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: eph env + improve docker images to run in userspace [superset]
Posted by "drummerwolli (via GitHub)" <gi...@apache.org>.
drummerwolli commented on PR #32017:
URL: https://github.com/apache/superset/pull/32017#issuecomment-2640477791
@mistercrunch can you check [this issue](https://github.com/apache/superset/issues/31890)? I have problems running the `docker compose up --build` command from the [development instructions](https://superset.apache.org/docs/contributing/development#docker-compose-recommended) on a fresh clean machine, and if i try without the changes from this PR (the commit just before), it works. I'm not too deep into Docker to understand exactly what's going on, but assuming you changed some caching here, it sounds very suspicious that this could be related. thanks for checking!
cc @sfirke who helped on the investigation as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: eph env + improve docker images to run in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch merged PR #32017:
URL: https://github.com/apache/superset/pull/32017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #32017:
URL: https://github.com/apache/superset/pull/32017#discussion_r1933150455
##########
Dockerfile:
##########
@@ -125,27 +117,14 @@ RUN pip install --no-cache-dir --upgrade uv
RUN uv venv /app/.venv
ENV PATH="/app/.venv/bin:${PATH}"
-# Install Playwright and optionally setup headless browsers
-ARG INCLUDE_CHROMIUM="true"
-ARG INCLUDE_FIREFOX="false"
-RUN --mount=type=cache,target=/root/.cache/uv\
Review Comment:
Moving this section to the python-common layer that's used for the app itself as the python/backend translation builder layer does not need this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: improve docker images to support running in userspace [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #32017:
URL: https://github.com/apache/superset/pull/32017#discussion_r1933148430
##########
Dockerfile:
##########
@@ -100,21 +99,14 @@ RUN if [ "$BUILD_TRANSLATIONS" = "true" ]; then \
# Base python layer
######################################################################
FROM python:${PY_VER} AS python-base
-ARG BUILD_TRANSLATIONS="false" # Include translations in the final build
Review Comment:
Moving this section to the python-common layer that's used for the app itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org