You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/02/17 17:51:59 UTC
[GitHub] [airflow] potiuk opened a new pull request, #29598: Specific use-case: adding packages via requirements.txt in compose
potiuk opened a new pull request, #29598:
URL: https://github.com/apache/airflow/pull/29598
The users of docker compose do not realise that docker compose has built-in mechanism for rebuilding the image automatically when needed and do not realise that this does not make their workflows more complex. Apparently just describing how you can use custom images is either scary or not clear enough, as users keep on for dynamic way of adding the packages (which makes no sense if you use what docker-compose provides out-of-the-box) so it is worthwile to specifically mention "requirements.txt" and explicitly add all the steps needed to make requirements.txt and custom Docker image part of your regular workflow.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #29598: Specific use-case: adding packages via requirements.txt in compose
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #29598:
URL: https://github.com/apache/airflow/pull/29598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29598: Specific use-case: adding packages via requirements.txt in compose
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29598:
URL: https://github.com/apache/airflow/pull/29598#issuecomment-1436044348
All comments reflected and docs building fixed. Merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29598: Specific use-case: adding packages via requirements.txt in compose
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29598:
URL: https://github.com/apache/airflow/pull/29598#issuecomment-1435032956
Looks like we need to be even more explicit about that one: https://github.com/apache/airflow/discussions/29597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on a diff in pull request #29598: Specific use-case: adding packages via requirements.txt in compose
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #29598:
URL: https://github.com/apache/airflow/pull/29598#discussion_r1111110631
##########
docs/apache-airflow/howto/docker-compose/index.rst:
##########
@@ -271,6 +271,39 @@ to rebuild the images on-the-fly when you run other ``docker compose`` commands.
Examples of how you can extend the image with custom providers, python packages,
apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
+Special case - adding dependencies via requirements.txt file
+============================================================
+
+Usual case for custom images, is when you want to add a set of requirements to it - usually stored in
+``requirements.txt`` file. For development, you might be tempted to add it dynamically when you are
+starting the original airflow image, but this has a number of side effects (for example your containers
+will start much slower - the slower you have a number of requirements). Also it is completely unnecessary,
Review Comment:
```suggestion
will start much slower - each additional dependency will further delay your containers start up time). Also it is completely unnecessary,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on a diff in pull request #29598: Specific use-case: adding packages via requirements.txt in compose
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #29598:
URL: https://github.com/apache/airflow/pull/29598#discussion_r1111110631
##########
docs/apache-airflow/howto/docker-compose/index.rst:
##########
@@ -271,6 +271,39 @@ to rebuild the images on-the-fly when you run other ``docker compose`` commands.
Examples of how you can extend the image with custom providers, python packages,
apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
+Special case - adding dependencies via requirements.txt file
+============================================================
+
+Usual case for custom images, is when you want to add a set of requirements to it - usually stored in
+``requirements.txt`` file. For development, you might be tempted to add it dynamically when you are
+starting the original airflow image, but this has a number of side effects (for example your containers
+will start much slower - the slower you have a number of requirements). Also it is completely unnecessary,
Review Comment:
```suggestion
will start much slower - each additional dependency will further delay your containers start up). Also it is completely unnecessary,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on a diff in pull request #29598: Specific use-case: adding packages via requirements.txt in compose
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #29598:
URL: https://github.com/apache/airflow/pull/29598#discussion_r1111110631
##########
docs/apache-airflow/howto/docker-compose/index.rst:
##########
@@ -271,6 +271,39 @@ to rebuild the images on-the-fly when you run other ``docker compose`` commands.
Examples of how you can extend the image with custom providers, python packages,
apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
+Special case - adding dependencies via requirements.txt file
+============================================================
+
+Usual case for custom images, is when you want to add a set of requirements to it - usually stored in
+``requirements.txt`` file. For development, you might be tempted to add it dynamically when you are
+starting the original airflow image, but this has a number of side effects (for example your containers
+will start much slower - the slower you have a number of requirements). Also it is completely unnecessary,
Review Comment:
```suggestion
will start much slower - each additional dependency will further delay your containers start up time). Also it is completely unnecessary,
```
0r something like this to keep the same structure:
”the more requirements you have, the slower it will get”
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on a diff in pull request #29598: Specific use-case: adding packages via requirements.txt in compose
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #29598:
URL: https://github.com/apache/airflow/pull/29598#discussion_r1111110631
##########
docs/apache-airflow/howto/docker-compose/index.rst:
##########
@@ -271,6 +271,39 @@ to rebuild the images on-the-fly when you run other ``docker compose`` commands.
Examples of how you can extend the image with custom providers, python packages,
apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
+Special case - adding dependencies via requirements.txt file
+============================================================
+
+Usual case for custom images, is when you want to add a set of requirements to it - usually stored in
+``requirements.txt`` file. For development, you might be tempted to add it dynamically when you are
+starting the original airflow image, but this has a number of side effects (for example your containers
+will start much slower - the slower you have a number of requirements). Also it is completely unnecessary,
Review Comment:
```suggestion
will start much slower - each additional dependency will further delay your containers start up time). Also it is completely unnecessary,
```
0r something like this to keep the original structure:
”the more requirements you have, the slower it will get”
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org