You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/06 15:14:15 UTC
[GitHub] [airflow] potiuk opened a new pull request, #25565: Separate instruction to install OS dependencies in images
potiuk opened a new pull request, #25565:
URL: https://github.com/apache/airflow/pull/25565
This change will allow to experiment with other base images
(for example CentOS that our users highly demand) but also
it has a few nice simplifications and improvements along
the way:
* no more runtime parameters for CI image (they only make
sense for PROD image)
* no more support for Buster image (it is end of life in August)
* Dockerfile has now less embedded default values (most of them
moved to inlined bash script)
* configuration for yarn sources is removed (we do not need yarn
any more in our images)
* additional pure-dev dependencies in CI image are passed through
ADDITIONAL_DEV_DEPS
* dev installation does not remove installation cache, making
the CI image slightly bigger but easier for devel use - to install new
dependencies (no need for `apt-get update` before installation)
* latest patchlevels of various tools we use for CI were bumped
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
mik-laj commented on PR #25565:
URL: https://github.com/apache/airflow/pull/25565#issuecomment-1207488662
CC: @sfc-gh-mkmak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #25565:
URL: https://github.com/apache/airflow/pull/25565#discussion_r939846169
##########
tests/charts/helm_template_generator.py:
##########
@@ -102,7 +102,7 @@ def validate_k8s_object(instance, kubernetes_version):
def render_chart(
- name="RELEASE-NAME",
+ name="release-name",
Review Comment:
This based on previous PR https://github.com/apache/airflow/pull/25582. I rebased in on top to see if it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a diff in pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
mik-laj commented on code in PR #25565:
URL: https://github.com/apache/airflow/pull/25565#discussion_r939723985
##########
docs/docker-stack/build-arg-ref.rst:
##########
@@ -130,33 +130,33 @@ for examples of using those arguments.
| ``ADDITIONAL_PYTHON_DEPS`` | | Optional python packages to extend |
| | | the image with some extra dependencies. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``DEV_APT_COMMAND`` | (see Dockerfile) | Dev apt command executed before dev deps |
+| ``DEV_APT_COMMAND`` | | Dev apt command executed before dev deps |
| | | are installed in the Build image. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_DEV_APT_COMMAND`` | | Additional Dev apt command executed |
| | | before dev dep are installed |
| | | in the Build image. Should start with |
| | | ``&&``. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``DEV_APT_DEPS`` | (see Dockerfile) | Dev APT dependencies installed |
-| | | in the Build image. |
+| ``DEV_APT_DEPS`` | Empty - install default de dependencies | Dev APT dependencies installed |
Review Comment:
de?
##########
docs/docker-stack/build-arg-ref.rst:
##########
@@ -130,33 +130,33 @@ for examples of using those arguments.
| ``ADDITIONAL_PYTHON_DEPS`` | | Optional python packages to extend |
| | | the image with some extra dependencies. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``DEV_APT_COMMAND`` | (see Dockerfile) | Dev apt command executed before dev deps |
+| ``DEV_APT_COMMAND`` | | Dev apt command executed before dev deps |
| | | are installed in the Build image. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_DEV_APT_COMMAND`` | | Additional Dev apt command executed |
| | | before dev dep are installed |
| | | in the Build image. Should start with |
| | | ``&&``. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``DEV_APT_DEPS`` | (see Dockerfile) | Dev APT dependencies installed |
-| | | in the Build image. |
+| ``DEV_APT_DEPS`` | Empty - install default de dependencies | Dev APT dependencies installed |
+| | (see ``install_os_dependencies.sh``) | in the Build image. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_DEV_APT_DEPS`` | | Additional apt dev dependencies |
| | | installed in the Build image. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_DEV_APT_ENV`` | | Additional env variables defined |
| | | when installing dev deps. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``RUNTIME_APT_COMMAND`` | (see Dockerfile) | Runtime apt command executed before deps |
-| | | are installed in the Main image. |
+| ``RUNTIME_APT_COMMAND`` | | Runtime apt command executed before deps |
+| | | are installed in the ``main`` stage. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_RUNTIME_APT_COMMAND`` | | Additional Runtime apt command executed |
| | | before runtime dep are installed |
-| | | in the Main image. Should start with |
+| | | in the ``main`` stage. Should start with |
| | | ``&&``. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``RUNTIME_APT_DEPS`` | (see Dockerfile) | Runtime APT dependencies installed |
-| | | in the Main image. |
+| ``RUNTIME_APT_DEPS`` | Empty - install default de dependencies | Runtime APT dependencies installed |
Review Comment:
de?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25565:
URL: https://github.com/apache/airflow/pull/25565#issuecomment-1213276288
More comments needed :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25565:
URL: https://github.com/apache/airflow/pull/25565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a diff in pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
mik-laj commented on code in PR #25565:
URL: https://github.com/apache/airflow/pull/25565#discussion_r939723985
##########
docs/docker-stack/build-arg-ref.rst:
##########
@@ -130,33 +130,33 @@ for examples of using those arguments.
| ``ADDITIONAL_PYTHON_DEPS`` | | Optional python packages to extend |
| | | the image with some extra dependencies. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``DEV_APT_COMMAND`` | (see Dockerfile) | Dev apt command executed before dev deps |
+| ``DEV_APT_COMMAND`` | | Dev apt command executed before dev deps |
| | | are installed in the Build image. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_DEV_APT_COMMAND`` | | Additional Dev apt command executed |
| | | before dev dep are installed |
| | | in the Build image. Should start with |
| | | ``&&``. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``DEV_APT_DEPS`` | (see Dockerfile) | Dev APT dependencies installed |
-| | | in the Build image. |
+| ``DEV_APT_DEPS`` | Empty - install default de dependencies | Dev APT dependencies installed |
Review Comment:
```suggestion
| ``DEV_APT_DEPS`` | Empty - install default the dependencies | Dev APT dependencies installed |
```
##########
docs/docker-stack/build-arg-ref.rst:
##########
@@ -130,33 +130,33 @@ for examples of using those arguments.
| ``ADDITIONAL_PYTHON_DEPS`` | | Optional python packages to extend |
| | | the image with some extra dependencies. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``DEV_APT_COMMAND`` | (see Dockerfile) | Dev apt command executed before dev deps |
+| ``DEV_APT_COMMAND`` | | Dev apt command executed before dev deps |
| | | are installed in the Build image. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_DEV_APT_COMMAND`` | | Additional Dev apt command executed |
| | | before dev dep are installed |
| | | in the Build image. Should start with |
| | | ``&&``. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``DEV_APT_DEPS`` | (see Dockerfile) | Dev APT dependencies installed |
-| | | in the Build image. |
+| ``DEV_APT_DEPS`` | Empty - install default de dependencies | Dev APT dependencies installed |
Review Comment:
```suggestion
| ``DEV_APT_DEPS`` | Empty - install default the dependencies | Dev APT dependencies installed |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25565:
URL: https://github.com/apache/airflow/pull/25565#issuecomment-1207237719
I need to get https://github.com/apache/airflow/pull/25567 to main and rebase for the CI to succeed (We are overiding default parameters in the CI image in the main version of breeze.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25565:
URL: https://github.com/apache/airflow/pull/25565#issuecomment-1221540395
@mik-laj ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25565:
URL: https://github.com/apache/airflow/pull/25565#issuecomment-1221176137
Anyone :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25565:
URL: https://github.com/apache/airflow/pull/25565#issuecomment-1207233034
This is a refactoring for installation of `apt' dependencies in our Dockerfile and it will be really useful to explore a possibility of building a CentOS image which has been proposed by the Snowflake team in https://github.com/apache/airflow/pull/24825
cc: @mik-laj @sfc-gh-mkmak - it's not yet adding CentOS but it should make it easy for us to be able to choose Debian/CentOS when building the image. Once this one is merged, I will attempt to incorporate what you've done at Snowflake and when I succceed I will make a propsal at the devlist to introduce an experimental CentOS support (once I know if we can easily have 1-1 parity for our PROD images).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a diff in pull request #25565: Separate instruction to install OS dependencies in images
Posted by GitBox <gi...@apache.org>.
mik-laj commented on code in PR #25565:
URL: https://github.com/apache/airflow/pull/25565#discussion_r939724329
##########
tests/charts/helm_template_generator.py:
##########
@@ -102,7 +102,7 @@ def validate_k8s_object(instance, kubernetes_version):
def render_chart(
- name="RELEASE-NAME",
+ name="release-name",
Review Comment:
Can you tell little more about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org