You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/27 16:03:16 UTC
[GitHub] [airflow] potiuk opened a new pull request #11176: Better customizable build process
potiuk opened a new pull request #11176:
URL: https://github.com/apache/airflow/pull/11176
**NOTE TO REVIEWER** Please review only the last commit as it depends on #11174
This is the third (and not last) part of making the Production
image more corporate-environment friendly. It's been prepared
for the request of one of the big Airflow user (company) that
has rather strict security requirements when it comes to
preparing and building images. They are committed to
synchronizing with the progress of Apache Airflow 2.0 development
and making the image customizable so that they can build it using
only sources controlled by them internally was one of the important
requirements for them.
This change adds the possibilty of customizing various steps in
the build process:
* adding custom scripts to be run before installation of both
build image and runtime image. This allows for example to
add installing custom GPG keys, and adding custom sources.
* customizing the way NodeJS and Yarn are installed in the
build image segment - as they might rely on their own way
of installation.
* adding extra packages to be installed during both build and
dev segment build steps. This is crucial to achieve the same
size optimizations as the original image.
* defining additional environment variables (for example
environment variables that indicate acceptance of the EULAs
in case of installing proprietary packages that require
EULA acceptance - both in the build image and runtime image
(again the goal is to keep the image optimized for size)
The image build process remains the same when no customization
options are specified, but having those options increases
flexibility of the image build process in corporate environments.
This is part of #11171.
This change also fixes some of the issues opened and raised by
other users of the Dockerfile.
Fixes: #10730
Fixes: #10555
Fixes: #10856
Input from those issues has been taken into account when this
change was designed so that the cases described in those issues
could be implemented. Example from one of the issue landed as
an example way of building highly customized Airflow Image
using those customization options.
Depends on #11174
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #11176: More customizable build process for Docker images
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #11176:
URL: https://github.com/apache/airflow/pull/11176
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11176: More customizable build process
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11176:
URL: https://github.com/apache/airflow/pull/11176#issuecomment-700610889
Another round of changes brought from our big-corporate-customer experience. Would love to merge it soon to get better customizability in the image building - on the way to truly production-ready Docker Image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11176: Better customizable build process
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11176:
URL: https://github.com/apache/airflow/pull/11176#issuecomment-699653848
cc: @jaketf
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11176: More customizable build process
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11176:
URL: https://github.com/apache/airflow/pull/11176#discussion_r496714351
##########
File path: IMAGES.rst
##########
@@ -255,15 +255,87 @@ The image is primarily optimised for size of the final image, but also for speed
It first pre-installs them from the right github branch and only after that final airflow installation is
done from either local sources or remote location (PIP or github repository).
-Manually building the images
-----------------------------
+Customizing the image
+.....................
+Customizing the image is an alternative way of adding your own dependencies to the image.
+
+The easiest way to build the image image is to use ``breeze`` script, but you can also build such customized
+image by running appropriately crafted docker build in which you specify all the ``build-args``
+that you need to add to customize it. You can read about all the args and ways you can build the image
+in the `<#ci-image-build-arguments>`_ chapter below.
+
+Here just a few examples are presented which should give you general understanding of what you can customize.
+
+This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and
+additional apt dev and runtime dependencies.
+
+.. code-block:: bash
+
+ docker build . -f Dockerfile.ci \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
+ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
+ --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \
+ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
+ --build-arg AIRFLOW_SOURCES_FROM="empty" \
+ --build-arg AIRFLOW_SOURCES_TO="/empty" \
+ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc"
+ --build-arg ADDITIONAL_PYTHON_DEPS="pandas"
+ --build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++"
+ --build-arg ADDITIONAL_RUNTIME_APT_DEPS="default-jre-headless"
+ --tag my-image
+
+
+the same image can be built using ``breeze`` (it supports auto-completion of the options):
+
+.. code-block:: bash
+
+ ./breeze build-image -f Dockerfile.ci \
+ --production-image --python 3.7 --install-airflow-version=1.10.12 \
+ --additional-extras=jdbc --additional-python-deps="pandas" \
+ --additional-dev-apt-deps="gcc g++" --additional-runtime-apt-deps="default-jre-headless"
You can build the default production image with standard ``docker build`` command but they will only build
default versions of the image and will not use the dockerhub versions of images as cache.
-CI images
-.........
+You can customize more aspects of the image - such as additional commands executed before apt dependencies
+are installed, or adding extra sources to install your dependencies from. You can see all the arguments
+described below but here is an example of rather complex command to customize the image
+based on example in `this comment <https://github.com/apache/airflow/issues/8605#issuecomment-690065621>`_:
+
+.. code-block:: bash
+
+ docker build . -f Dockerfile.ci \
+ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+ --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
+ --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
+ --build-arg AIRFLOW_INSTALL_VERSION="==1.10.12" \
+ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
+ --build-arg AIRFLOW_SOURCES_FROM="empty" \
+ --build-arg AIRFLOW_SOURCES_TO="/empty" \
+ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="slack" \
+ --build-arg ADDITIONAL_PYTHON_DEPS="apache-airflow-backport-providers-odbc \
+ apache-airflow-backport-providers-odbc \
Review comment:
```suggestion
```
It is duplicated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org