[airflow] 02/23: Specific use-case: adding packages via requirements.txt in compose (#29598)

[ep...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 46f9cb90d61c07f56e2ad9f9e4b95c6194f96b92
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Feb 19 19:09:03 2023 +0100

    Specific use-case: adding packages via requirements.txt in compose (#29598)
    
    The users of docker compose do not realise that docker compose
    has built-in mechanism for rebuilding the image automatically when
    needed and do not realise that this does not make their workflows
    more complex. Apparently just describing how you can use custom
    images is either scary or not clear enough, as users keep on for
    dynamic way of adding the packages (which makes no sense if you
    use what docker-compose provides out-of-the-box) so it is
    worthwile to specifically mention "requirements.txt" and
    explicitly add all the steps needed to make requirements.txt and
    custom Docker image part of your regular workflow.
    
    (cherry picked from commit a21c17bc07c1eeb733eca889a02396fab401b215)
---
 docs/apache-airflow/howto/docker-compose/index.rst | 34 ++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/docs/apache-airflow/howto/docker-compose/index.rst b/docs/apache-airflow/howto/docker-compose/index.rst
index 6e75ef8416..e4d169c0ea 100644
--- a/docs/apache-airflow/howto/docker-compose/index.rst
+++ b/docs/apache-airflow/howto/docker-compose/index.rst
@@ -271,6 +271,40 @@ to rebuild the images on-the-fly when you run other ``docker compose`` commands.
 Examples of how you can extend the image with custom providers, python packages,
 apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
 
+Special case - adding dependencies via requirements.txt file
+============================================================
+
+Usual case for custom images, is when you want to add a set of requirements to it - usually stored in
+``requirements.txt`` file. For development, you might be tempted to add it dynamically when you are
+starting the original airflow image, but this has a number of side effects (for example your containers
+will start much slower - each additional dependency will further delay your containers start up time).
+Also it is completely unnecessary, because docker compose has the development workflow built-in.
+You can - following the previous chapter, automatically build and use your custom image when you
+iterate with docker compose locally. Specifically when you want to add your own requirement file,
+you should do those steps:
+
+1) Comment out the ``image: ...`` line and remove comment from the ``build: .`` line in the
+   ``docker-compose.yaml`` file. The relevant part of the docker-compose file of yours should look similar
+   to (use correct image tag):
+
+```
+#image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.5.1}
+build: .
+```
+
+2) Create ``Dockerfile`` in the same folder your ``docker-compose.yaml`` file is with content similar to:
+
+```
+FROM apache/airflow:2.5.1
+ADD requirements.txt .
+RUN pip install -r requirements.txt
+```
+
+3) Place ``requirements.txt`` file in the same directory.
+
+Run ``docker compose build`` to build the image, or add ``--build`` flag to ``docker compose up`` or
+``docker compose run`` commands to build the image automatically as needed.
+
 Networking
 ==========