You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/09/26 06:33:14 UTC
[GitHub] [superset] pwang2 opened a new pull request, #21583: misc: optimize docker build by better utilizing build cache
pwang2 opened a new pull request, #21583:
URL: https://github.com/apache/superset/pull/21583
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
Using common docker build practice to utilizing docker build cache better.
New docker image build
1. 394.79 MB -> 348.48 MB
2. 3-5 minutes saving for initial build by not copying/deleting unneeded file, especial node_modules.
3. merged the first stage to the lean image. still being able to cache the pip requirements installation.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
```bash
podman build -t superset --no-cache --target lean -f ./Dockerfile
```
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] EugeneTorap commented on pull request #21583: misc: optimize docker build by better utilizing build cache
Posted by GitBox <gi...@apache.org>.
EugeneTorap commented on PR #21583:
URL: https://github.com/apache/superset/pull/21583#issuecomment-1265053987
@pwang2 Thank you so much for this PR!
The changes should speeds up docker build CI by 3-5 minutes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21583: build: optimize docker build by better utilizing build cache
Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21583:
URL: https://github.com/apache/superset/pull/21583#discussion_r989165929
##########
Dockerfile:
##########
@@ -15,60 +15,34 @@
# limitations under the License.
#
-######################################################################
-# PY stage that simply does a pip install on our requirements
-######################################################################
-ARG PY_VER=3.8.13-slim
-FROM python:${PY_VER} AS superset-py
-
-RUN mkdir /app \
- && apt-get update -y \
- && apt-get install -y --no-install-recommends \
- build-essential \
- default-libmysqlclient-dev \
- libpq-dev \
- libsasl2-dev \
- libecpg-dev \
- && rm -rf /var/lib/apt/lists/*
-
-# First, we just wanna install requirements, which will allow us to utilize the cache
-# in order to only build if and only if requirements change
-COPY ./requirements/*.txt /app/requirements/
-COPY setup.py MANIFEST.in README.md /app/
-COPY superset-frontend/package.json /app/superset-frontend/
-RUN cd /app \
- && mkdir -p superset/static \
- && touch superset/static/version_info.json \
- && pip install --no-cache -r requirements/local.txt
-
-
######################################################################
# Node stage to deal with static asset construction
######################################################################
+ARG PY_VER=3.8.13-slim
Review Comment:
Good to know! Thanks for the explaination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21583: build: optimize docker build by better utilizing build cache
Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21583:
URL: https://github.com/apache/superset/pull/21583#discussion_r989165929
##########
Dockerfile:
##########
@@ -15,60 +15,34 @@
# limitations under the License.
#
-######################################################################
-# PY stage that simply does a pip install on our requirements
-######################################################################
-ARG PY_VER=3.8.13-slim
-FROM python:${PY_VER} AS superset-py
-
-RUN mkdir /app \
- && apt-get update -y \
- && apt-get install -y --no-install-recommends \
- build-essential \
- default-libmysqlclient-dev \
- libpq-dev \
- libsasl2-dev \
- libecpg-dev \
- && rm -rf /var/lib/apt/lists/*
-
-# First, we just wanna install requirements, which will allow us to utilize the cache
-# in order to only build if and only if requirements change
-COPY ./requirements/*.txt /app/requirements/
-COPY setup.py MANIFEST.in README.md /app/
-COPY superset-frontend/package.json /app/superset-frontend/
-RUN cd /app \
- && mkdir -p superset/static \
- && touch superset/static/version_info.json \
- && pip install --no-cache -r requirements/local.txt
-
-
######################################################################
# Node stage to deal with static asset construction
######################################################################
+ARG PY_VER=3.8.13-slim
Review Comment:
Good to know! Thanks for the explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] pwang2 commented on a diff in pull request #21583: misc: optimize docker build by better utilizing build cache
Posted by GitBox <gi...@apache.org>.
pwang2 commented on code in PR #21583:
URL: https://github.com/apache/superset/pull/21583#discussion_r988480069
##########
Dockerfile:
##########
@@ -15,60 +15,34 @@
# limitations under the License.
#
-######################################################################
-# PY stage that simply does a pip install on our requirements
-######################################################################
-ARG PY_VER=3.8.13-slim
-FROM python:${PY_VER} AS superset-py
-
-RUN mkdir /app \
- && apt-get update -y \
- && apt-get install -y --no-install-recommends \
- build-essential \
- default-libmysqlclient-dev \
- libpq-dev \
- libsasl2-dev \
- libecpg-dev \
- && rm -rf /var/lib/apt/lists/*
-
-# First, we just wanna install requirements, which will allow us to utilize the cache
-# in order to only build if and only if requirements change
-COPY ./requirements/*.txt /app/requirements/
-COPY setup.py MANIFEST.in README.md /app/
-COPY superset-frontend/package.json /app/superset-frontend/
-RUN cd /app \
- && mkdir -p superset/static \
- && touch superset/static/version_info.json \
- && pip install --no-cache -r requirements/local.txt
-
-
######################################################################
# Node stage to deal with static asset construction
######################################################################
+ARG PY_VER=3.8.13-slim
FROM node:16-slim AS superset-node
ARG NPM_BUILD_CMD="build"
ENV BUILD_CMD=${NPM_BUILD_CMD}
# NPM ci first, as to NOT invalidate previous steps except for when package.json changes
RUN mkdir -p /app/superset-frontend
-RUN mkdir -p /app/superset/assets
+
COPY ./docker/frontend-mem-nag.sh /
-COPY ./superset-frontend /app/superset-frontend
-RUN /frontend-mem-nag.sh \
- && cd /app/superset-frontend \
- && npm ci
+RUN /frontend-mem-nag.sh
-# This seems to be the most expensive step
-RUN cd /app/superset-frontend \
- && npm run ${BUILD_CMD} \
- && rm -rf node_modules
+WORKDIR /app/superset-frontend/
+COPY superset-frontend/package*.json .
Review Comment:
Thanks, @EugeneTorap , I tested, without / works seems okay. but it looks more obvious when adding slash here. :)
<img width="252" alt="image" src="https://user-images.githubusercontent.com/1592014/194195452-43d02154-cab0-4dec-acf5-eea3f9464f70.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] EugeneTorap commented on a diff in pull request #21583: misc: optimize docker build by better utilizing build cache
Posted by GitBox <gi...@apache.org>.
EugeneTorap commented on code in PR #21583:
URL: https://github.com/apache/superset/pull/21583#discussion_r985475636
##########
Dockerfile:
##########
@@ -15,60 +15,34 @@
# limitations under the License.
#
-######################################################################
-# PY stage that simply does a pip install on our requirements
-######################################################################
-ARG PY_VER=3.8.13-slim
-FROM python:${PY_VER} AS superset-py
-
-RUN mkdir /app \
- && apt-get update -y \
- && apt-get install -y --no-install-recommends \
- build-essential \
- default-libmysqlclient-dev \
- libpq-dev \
- libsasl2-dev \
- libecpg-dev \
- && rm -rf /var/lib/apt/lists/*
-
-# First, we just wanna install requirements, which will allow us to utilize the cache
-# in order to only build if and only if requirements change
-COPY ./requirements/*.txt /app/requirements/
-COPY setup.py MANIFEST.in README.md /app/
-COPY superset-frontend/package.json /app/superset-frontend/
-RUN cd /app \
- && mkdir -p superset/static \
- && touch superset/static/version_info.json \
- && pip install --no-cache -r requirements/local.txt
-
-
######################################################################
# Node stage to deal with static asset construction
######################################################################
+ARG PY_VER=3.8.13-slim
FROM node:16-slim AS superset-node
ARG NPM_BUILD_CMD="build"
ENV BUILD_CMD=${NPM_BUILD_CMD}
# NPM ci first, as to NOT invalidate previous steps except for when package.json changes
RUN mkdir -p /app/superset-frontend
-RUN mkdir -p /app/superset/assets
+
COPY ./docker/frontend-mem-nag.sh /
-COPY ./superset-frontend /app/superset-frontend
-RUN /frontend-mem-nag.sh \
- && cd /app/superset-frontend \
- && npm ci
+RUN /frontend-mem-nag.sh
-# This seems to be the most expensive step
-RUN cd /app/superset-frontend \
- && npm run ${BUILD_CMD} \
- && rm -rf node_modules
+WORKDIR /app/superset-frontend/
+COPY superset-frontend/package*.json .
Review Comment:
**When using COPY with more than one source file, the destination must be a directory and end with a /**
```suggestion
COPY superset-frontend/package*.json ./
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] pwang2 commented on a diff in pull request #21583: build: optimize docker build by better utilizing build cache
Posted by GitBox <gi...@apache.org>.
pwang2 commented on code in PR #21583:
URL: https://github.com/apache/superset/pull/21583#discussion_r989157053
##########
Dockerfile:
##########
@@ -15,60 +15,34 @@
# limitations under the License.
#
-######################################################################
-# PY stage that simply does a pip install on our requirements
-######################################################################
-ARG PY_VER=3.8.13-slim
-FROM python:${PY_VER} AS superset-py
-
-RUN mkdir /app \
- && apt-get update -y \
- && apt-get install -y --no-install-recommends \
- build-essential \
- default-libmysqlclient-dev \
- libpq-dev \
- libsasl2-dev \
- libecpg-dev \
- && rm -rf /var/lib/apt/lists/*
-
-# First, we just wanna install requirements, which will allow us to utilize the cache
-# in order to only build if and only if requirements change
-COPY ./requirements/*.txt /app/requirements/
-COPY setup.py MANIFEST.in README.md /app/
-COPY superset-frontend/package.json /app/superset-frontend/
-RUN cd /app \
- && mkdir -p superset/static \
- && touch superset/static/version_info.json \
- && pip install --no-cache -r requirements/local.txt
-
-
######################################################################
# Node stage to deal with static asset construction
######################################################################
+ARG PY_VER=3.8.13-slim
Review Comment:
This is some not well documented in docker document. in a multistage dockerfile, ARG used in FROM need to be added before the first FROM.
<img width="1131" alt="image" src="https://user-images.githubusercontent.com/1592014/194346601-22c56a35-26c2-45cc-9dc7-a3c608d4a0f2.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] zhaoyongjie merged pull request #21583: build: optimize docker build by better utilizing build cache
Posted by GitBox <gi...@apache.org>.
zhaoyongjie merged PR #21583:
URL: https://github.com/apache/superset/pull/21583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] pwang2 commented on pull request #21583: misc: optimize docker build by better utilizing build cache
Posted by GitBox <gi...@apache.org>.
pwang2 commented on PR #21583:
URL: https://github.com/apache/superset/pull/21583#issuecomment-1269199250
Receiving a lot of GitHub Actions failure which I believe should nothing to do with the change here. Can anyone help me figure out if anything missing from my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21583: build: optimize docker build by better utilizing build cache
Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21583:
URL: https://github.com/apache/superset/pull/21583#discussion_r988661323
##########
Dockerfile:
##########
@@ -85,16 +59,24 @@ RUN mkdir -p ${PYTHONPATH} \
&& apt-get install -y --no-install-recommends \
build-essential \
default-libmysqlclient-dev \
+ libsasl2-dev \
libsasl2-modules-gssapi-mit \
libpq-dev \
libecpg-dev \
&& rm -rf /var/lib/apt/lists/*
-COPY --from=superset-py /usr/local/lib/python3.8/site-packages/ /usr/local/lib/python3.8/site-packages/
-# Copying site-packages doesn't move the CLIs, so let's copy them one by one
-COPY --from=superset-py /usr/local/bin/gunicorn /usr/local/bin/celery /usr/local/bin/flask /usr/bin/
+COPY ./requirements/*.txt /app/requirements/
+COPY setup.py MANIFEST.in README.md /app/
+
+# setup.py uses the version information in package.json
+COPY superset-frontend/package.json /app/superset-frontend/
Review Comment:
IMO setup.py to read package.json isn't a good idea but exists a long time in Superset, might change this logic in the separated PR.
##########
Dockerfile:
##########
@@ -15,60 +15,34 @@
# limitations under the License.
#
-######################################################################
-# PY stage that simply does a pip install on our requirements
-######################################################################
-ARG PY_VER=3.8.13-slim
-FROM python:${PY_VER} AS superset-py
-
-RUN mkdir /app \
- && apt-get update -y \
- && apt-get install -y --no-install-recommends \
- build-essential \
- default-libmysqlclient-dev \
- libpq-dev \
- libsasl2-dev \
- libecpg-dev \
- && rm -rf /var/lib/apt/lists/*
-
-# First, we just wanna install requirements, which will allow us to utilize the cache
-# in order to only build if and only if requirements change
-COPY ./requirements/*.txt /app/requirements/
-COPY setup.py MANIFEST.in README.md /app/
-COPY superset-frontend/package.json /app/superset-frontend/
-RUN cd /app \
- && mkdir -p superset/static \
- && touch superset/static/version_info.json \
- && pip install --no-cache -r requirements/local.txt
-
-
######################################################################
# Node stage to deal with static asset construction
######################################################################
+ARG PY_VER=3.8.13-slim
Review Comment:
The section is the Node image so I think `PY_VER` is useless here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org