You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/11 15:45:58 UTC
[GitHub] [airflow] potiuk opened a new pull request #21522: Prepare to switch to debian bullseye
potiuk opened a new pull request #21522:
URL: https://github.com/apache/airflow/pull/21522
In order to switch smoothly to Debian Bullseye we need to run a
few more tests and see if the performance impact we observe is
consistent (the tests with Public Runners seems to be killed more
frequently there).
This requires some preparatory work - namely add the capability
of overriding the DEBIAN_VERSION via PR - because otherwise
Build Images workflow that runs from main will build buster images
until the change is actually merged.
This change introduce this capability without switching to
Bullseye yet - it will unblock the actual PR that will make the
change from failing in PRs though.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#discussion_r804975863
##########
File path: Dockerfile
##########
@@ -348,20 +352,22 @@ SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "nounset", "-o", "n
ARG AIRFLOW_UID
LABEL org.apache.airflow.distro="debian" \
- org.apache.airflow.distro.version="buster" \
+ org.apache.airflow.distro.version="${DEBIAN_VERSION}" \
org.apache.airflow.module="airflow" \
org.apache.airflow.component="airflow" \
org.apache.airflow.image="airflow" \
org.apache.airflow.uid="${AIRFLOW_UID}"
+ARG DEBIAN_VERSION
ARG PYTHON_BASE_IMAGE
ARG AIRFLOW_PIP_VERSION
ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} \
# Make sure noninteractive debian install is used and language variables set
DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \
LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 \
- AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
+ AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
+ DEBIAN_VERSION=${DEBIAN_VERSION}
Review comment:
```suggestion
AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION}
```
Not needed anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #21522:
URL: https://github.com/apache/airflow/pull/21522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#discussion_r805066143
##########
File path: docs/docker-stack/docker-examples/customizing/github-different-repository.sh
##########
@@ -21,9 +21,12 @@ set -euo pipefail
AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START build]
+export DEBIAN_VERSION="buster"
+
docker build . \
--pull \
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-buster" \
+ --build-arg DEBIAN_VERSION="${DEBIAN_VERSION}" \
+ --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-${DEBIAN_VERSION}" \
Review comment:
Do we need to even show PYTHON_BASE_IMAGE and DEBIAN_VERSION in these examples?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#issuecomment-1036773704
It seems a bit slimmer now ( the only "buster/bullseye" distinction is in the build scripts now - but not in the image).
Testing the "main" build here as well: https://github.com/potiuk/airflow/actions/runs/1832053186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#discussion_r804977181
##########
File path: Dockerfile
##########
@@ -67,10 +68,13 @@ FROM ${PYTHON_BASE_IMAGE} as airflow-build-image
# xtrace - to show commands executed)
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "nounset", "-o", "nolog", "-c"]
+ARG DEBIAN_VERSION
ARG PYTHON_BASE_IMAGE
+
ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} \
DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \
- LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8
+ LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 \
+ DEBIAN_VERSION=${DEBIAN_VERSION}
Review comment:
Actually yeah...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#issuecomment-1036810364
And of course a typo: https://github.com/potiuk/airflow/actions/runs/1832116945 . Good I checked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#issuecomment-1036636835
Preparation complete!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#issuecomment-1036368994
Testing "main" build here: https://github.com/potiuk/airflow/actions/runs/1830182817 . Looks good so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#issuecomment-1036610757
All shoud be addressed @mik-laj !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#discussion_r805069355
##########
File path: docs/docker-stack/docker-examples/customizing/github-different-repository.sh
##########
@@ -21,9 +21,12 @@ set -euo pipefail
AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START build]
+export DEBIAN_VERSION="buster"
+
docker build . \
--pull \
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-buster" \
+ --build-arg DEBIAN_VERSION="${DEBIAN_VERSION}" \
+ --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-${DEBIAN_VERSION}" \
Review comment:
Actually - not any more - since we do not make it a variable any more, the DEBIAN_VERSION can be completely skipped
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#issuecomment-1036853770
All Green !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#issuecomment-1036749994
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #21522: Prepare to switch to debian bullseye
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #21522:
URL: https://github.com/apache/airflow/pull/21522#discussion_r804969760
##########
File path: .github/workflows/build-images.yml
##########
@@ -273,6 +276,9 @@ jobs:
scripts/ci/libraries/_initialization.sh | \
awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
echo "DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}" >> $GITHUB_ENV
+ DEBIAN_VERSION=$(grep "export DEBIAN_VERSION" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
Review comment:
```suggestion
cut -d "=" -f 3 | sed s'/["}]//g')
```
A bit simpler, but the result should be the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org