[airflow] 02/11: Improve image building documentation for new users (#17409)

[jh...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

jhtimmins pushed a commit to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit f2da764a3d50ec7fe5ea2e3497b21dce5f0219a0
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Aug 4 23:07:22 2021 +0200

    Improve image building documentation for new users (#17409)
    
    * Improve image building documentation for new users
    
    This PR improves documentation for building images of airflow,
    specifically targetting users who do not have big experience with
    building the images. It shows examples on how custom image building
    can be easily used to upgrade provider packages as well as how
    image building can be easily integrated in quick-start using
    docker-compose.
    
    (cherry picked from commit 4ee4199f6f0107f39726fc3551d1bf20b8b1283c)
---
 docs/apache-airflow-providers/index.rst                     |  5 +++++
 docs/apache-airflow/start/docker-compose.yaml               |  6 +++++-
 docs/apache-airflow/start/docker.rst                        | 12 ++++++++++++
 docs/docker-stack/build.rst                                 | 13 +++++++++++++
 .../docker-examples/extending/add-apt-packages/Dockerfile   |  2 +-
 .../extending/add-build-essential-extend/Dockerfile         |  2 +-
 .../extending/{embedding-dags => add-providers}/Dockerfile  |  7 +++----
 .../docker-examples/extending/add-pypi-packages/Dockerfile  |  2 +-
 .../docker-examples/extending/embedding-dags/Dockerfile     |  2 +-
 .../docker-examples/extending/writable-directory/Dockerfile |  2 +-
 10 files changed, 43 insertions(+), 10 deletions(-)

diff --git a/docs/apache-airflow-providers/index.rst b/docs/apache-airflow-providers/index.rst
index 7329b7f..71c5132 100644
--- a/docs/apache-airflow-providers/index.rst
+++ b/docs/apache-airflow-providers/index.rst
@@ -21,6 +21,8 @@ Provider packages
 
 .. contents:: :local:
 
+.. _providers:community-maintained-providers:
+
 Community maintained providers
 ''''''''''''''''''''''''''''''
 
@@ -31,6 +33,9 @@ Those provider packages are separated per-provider (for example ``amazon``, ``go
 etc.). Those packages are available as ``apache-airflow-providers`` packages - separately per each provider
 (for example there is an ``apache-airflow-providers-amazon`` or ``apache-airflow-providers-google`` package).
 
+The full list of community managed providers is available at
+`Providers Index <https://airflow.apache.org/docs/#providers-packages-docs-apache-airflow-providers-index-html>`_.
+
 You can install those provider packages separately in order to interface with a given service. For those
 providers that have corresponding extras, the provider packages (latest version from PyPI) are installed
 automatically when Airflow is installed with the extra.
diff --git a/docs/apache-airflow/start/docker-compose.yaml b/docs/apache-airflow/start/docker-compose.yaml
index 5a301cf..06991e7 100644
--- a/docs/apache-airflow/start/docker-compose.yaml
+++ b/docs/apache-airflow/start/docker-compose.yaml
@@ -44,7 +44,11 @@
 version: '3'
 x-airflow-common:
   &airflow-common
+  # In order to add custom dependencies or upgrade provider packages you can use your extended image.
+  # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
+  # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
   image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:|version|}
+  # build: .
   environment:
     &airflow-common-env
     AIRFLOW__CORE__EXECUTOR: CeleryExecutor
@@ -60,7 +64,7 @@ x-airflow-common:
     - ./dags:/opt/airflow/dags
     - ./logs:/opt/airflow/logs
     - ./plugins:/opt/airflow/plugins
-  user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
+  user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}"
   depends_on:
     redis:
       condition: service_healthy
diff --git a/docs/apache-airflow/start/docker.rst b/docs/apache-airflow/start/docker.rst
index 77c9333..747f87c 100644
--- a/docs/apache-airflow/start/docker.rst
+++ b/docs/apache-airflow/start/docker.rst
@@ -68,6 +68,18 @@ If you need install a new Python library or system library, you can :doc:`build
 .. _initializing_docker_compose_environment:
 
 
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use an extended image, containing some additional dependencies - for
+example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
+by placing a custom Dockerfile alongside your `docker-compose.yaml`. Then you can use `docker-compose build` command to build your image (you need to
+do it only once). You can also add the `--build` flag to your `docker-compose` commands to rebuild the images
+on-the-fly when you run other `docker-compose` commands.
+
+Examples of how you can extend the image with custom providers, python packages,
+apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
+
 Initializing Environment
 ========================
 
diff --git a/docs/docker-stack/build.rst b/docs/docker-stack/build.rst
index 7d89f2f..c469a35 100644
--- a/docs/docker-stack/build.rst
+++ b/docs/docker-stack/build.rst
@@ -250,6 +250,19 @@ You should be aware, about a few things:
 Examples of image extending
 ---------------------------
 
+Example of upgrading Airflow Provider packages
+..............................................
+
+The :ref:`Airflow Providers <providers:community-maintained-providers>` are released independently of core
+Airflow and sometimes you might want to upgrade specific providers only to fix some problems or
+use features available in that provider version. Here is an example of how you can do it
+
+.. exampleinclude:: docker-examples/extending/add-providers/Dockerfile
+    :language: Dockerfile
+    :start-after: [START Dockerfile]
+    :end-before: [END Dockerfile]
+
+
 Example of adding ``apt`` package
 .................................
 
diff --git a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
index 62de197..f11e87a 100644
--- a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
@@ -15,7 +15,7 @@
 
 # This is an example Dockerfile. It is not intended for PRODUCTION use
 # [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
 USER root
 RUN apt-get update \
   && apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
index b34fdc9..47ac51f 100644
--- a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
@@ -15,7 +15,7 @@
 
 # This is an example Dockerfile. It is not intended for PRODUCTION use
 # [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
 USER root
 RUN apt-get update \
   && apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
similarity index 90%
copy from docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
copy to docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
index c849697..cdf7a42 100644
--- a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
@@ -15,8 +15,7 @@
 
 # This is an example Dockerfile. It is not intended for PRODUCTION use
 # [START Dockerfile]
-FROM apache/airflow
-
-COPY --chown=airflow:root test_dag.py /opt/airflow/dags
-
+FROM apache/airflow:2.1.2
+RUN pip install --no-cache-dir apache-airflow-providers-docker==2.1.0
 # [END Dockerfile]
+
diff --git a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
index cc2559f..310f84c 100644
--- a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
@@ -15,6 +15,6 @@
 
 # This is an example Dockerfile. It is not intended for PRODUCTION use
 # [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
 RUN pip install --no-cache-dir lxml
 # [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
index c849697..48701aa 100644
--- a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
@@ -15,7 +15,7 @@
 
 # This is an example Dockerfile. It is not intended for PRODUCTION use
 # [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
 
 COPY --chown=airflow:root test_dag.py /opt/airflow/dags
 
diff --git a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
index ba07f68..8fbb98d 100644
--- a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
@@ -15,7 +15,7 @@
 
 # This is an example Dockerfile. It is not intended for PRODUCTION use
 # [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
 RUN umask 0002; \
     mkdir -p ~/writeable-directory
 # [END Dockerfile]