You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/16 08:52:33 UTC
[GitHub] [airflow] potiuk opened a new pull request #22304: Stronger language about Docker Compose customizability
potiuk opened a new pull request #22304:
URL: https://github.com/apache/airflow/pull/22304
Despite our warnings, our users continue treating the Docker
Compose that we exposed as something that should be easy to
extend and customize for their own needs, yet they continue
to struggle with some basic behaviour of containers, Docker Compose
and how they interact. This results in vast space of potential
problems as Docker Compose gives the user a false premise of
something that "just works" where it requires quite a deep
understanding on how it works.
When you get things wrong with Docker Compose, you often end up
with extremely confusing messages, that might suggest that the
problem is with Airflow, but really the problem is with how users
interact with their custom Docker images, registries, pulling,
networking, mounting volumes and plenty other things.
While this is the same with Kubernetes and Helm Chart, Helm Chart makes
it infinitely easier to customize in declarative way (this is what
our values.yaml does) and anything that has not been foreseen by Helm
Chart developers is "hard" by definition.
Docker Compose makes no such distinction. You really can't make Docker
Compose customizable by configuration, and any customization in it
requires modifying the compose file and for people who do not know
what they are doing will eventually lead to errors that they are not
able to diagnose and leads to creation of "Airlfow isssues", where they
should be brought to "Docker Compose" issues.
Example of that is here: https://github.com/apache/airflow/discussions/22301
where there are at least two issues that are not reproducible without
knowing in detail what the user has done, how the image was build
and distributed, and how the docker-compose installation interacted
with them. This leads to a terrible distraction for supporting
users of Airflow as the issues are really Docker Compose issues and
Airflow maintainers should not be involved in solving those.
This PR adds a bit stronger language and statement about the scope
and customizability of the Quick Start Docker Compose of ours. Not
only mentioning "Lack of Production Readiness" but also the
responsibility of the user to understand and diagnose docker compose
errors on their own and setting expectations that issues with Docker
Compose running should be directed elsewhere.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22304:
URL: https://github.com/apache/airflow/pull/22304#issuecomment-1068885495
Hey everyone - here is my take on stronger language that we should IMHO put on the Docker-Compose Quick Start.
I've been invovled in a number of issues raised by our users and honestly I think the "premise" of docker-compose that you can "just use it" and modify and you will be good, without really understanding of the "intricacies" of it does much more harm than good - giving the users false expectations on "how easy" it is to get airflow running with docker-compose.
Yeah. Easy to start, Extremely difficult to diagnose and solve problems if you do not know exactly what you are doing. I think we should be very explicit that if user chooses docker-compose based on our quick start - the moment they touch the file, they are mostly on their own.
I'd love to hear other's thoughts about it, but this is the experience I have with users who think that our "docker-compose" is on our website, means that we will help them to solve their "learn how docker-compose works" problems.
Many times the comments I saw from the users was `But I am using the "official" docker-compose. It shoudl work!". I think this is false expectations and this PR is another attempt to be even clearer about the expectations users should have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #22304:
URL: https://github.com/apache/airflow/pull/22304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22304:
URL: https://github.com/apache/airflow/pull/22304#discussion_r827769919
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -22,6 +22,26 @@ Running Airflow in Docker
This quick-start guide will allow you to quickly start Airflow with :doc:`CeleryExecutor </executor/celery>` in Docker. This is the fastest way to start Airflow.
+Customizing the quick-start Docker Compose
+==========================================
+
+DO NOT attempt to customize images and the Docker Compose if you do not know exactly what you are doing,
+know Docker Compose or do not will to debug and resolve problems on your own. If you do not
+know Docker Compose and expect Airflow to **just work** beyond following precisely the quick-start,
+then please use other ways of running Airflow - for example :doc:`/start/local` for testing and trying
+and :doc:`Official Airflow Community Helm Chart<helm-chart:index>` for production purposes.
+
+Even if many users think of Docker Compose as "ready to use", it is really a developer tool, that requires
+the user to know very well how docker images, containers docker compose networking, volumes, naming, image
Review comment:
```suggestion
the user to know very well how docker images, containers, docker compose networking, volumes, naming, image
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #22304:
URL: https://github.com/apache/airflow/pull/22304#discussion_r829354931
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -22,6 +22,26 @@ Running Airflow in Docker
This quick-start guide will allow you to quickly start Airflow with :doc:`CeleryExecutor </executor/celery>` in Docker. This is the fastest way to start Airflow.
+Customizing the quick-start Docker Compose
+==========================================
+
+DO NOT attempt to customize images and the Docker Compose if you do not know exactly what you are doing,
+know Docker Compose or do not will to debug and resolve problems on your own. If you do not
Review comment:
```suggestion
do not know Docker Compose, or are not prepared to debug and resolve problems on your own. If you do not
```
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -22,6 +22,26 @@ Running Airflow in Docker
This quick-start guide will allow you to quickly start Airflow with :doc:`CeleryExecutor </executor/celery>` in Docker. This is the fastest way to start Airflow.
Review comment:
If docker compose is really intended as a dev tool, maybe this line should be reworked to reflect that. I'm not sure exactly what the language should be. But maybe something like
```suggestion
This quick-start guide will allow you to quickly get Airflow up and running with :doc:`CeleryExecutor </executor/celery>` in Docker. For running airflow in production, we recommend running on kubernetes with the official helm chart.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #22304:
URL: https://github.com/apache/airflow/pull/22304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #22304:
URL: https://github.com/apache/airflow/pull/22304#discussion_r829354931
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -22,6 +22,26 @@ Running Airflow in Docker
This quick-start guide will allow you to quickly start Airflow with :doc:`CeleryExecutor </executor/celery>` in Docker. This is the fastest way to start Airflow.
+Customizing the quick-start Docker Compose
+==========================================
+
+DO NOT attempt to customize images and the Docker Compose if you do not know exactly what you are doing,
+know Docker Compose or do not will to debug and resolve problems on your own. If you do not
Review comment:
```suggestion
do not know Docker Compose, or are not prepared to debug and resolve problems on your own. If you do not
```
##########
File path: docs/apache-airflow/start/docker.rst
##########
@@ -22,6 +22,26 @@ Running Airflow in Docker
This quick-start guide will allow you to quickly start Airflow with :doc:`CeleryExecutor </executor/celery>` in Docker. This is the fastest way to start Airflow.
Review comment:
If docker compose is really intended as a dev tool, maybe this line should be reworked to reflect that. I'm not sure exactly what the language should be. But maybe something like
```suggestion
This quick-start guide will allow you to quickly get Airflow up and running with :doc:`CeleryExecutor </executor/celery>` in Docker. For running airflow in production, we recommend running on kubernetes with the official helm chart.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22304:
URL: https://github.com/apache/airflow/pull/22304#issuecomment-1070824420
Another example https://github.com/apache/airflow/discussions/22322. Users have much bigger expectations when seeing our docker compose and I think we should be more explicit in telling them their expectations cannot be fulfilled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #22304:
URL: https://github.com/apache/airflow/pull/22304#issuecomment-1068885495
Hey everyone - here is my take on stronger language that we should IMHO put on the Docker-Compose Quick Start.
I've been involved in a number of issues raised by our users and honestly I think the "premise" of docker-compose that you can "just use it" and modify and you will be good, without really understanding of the "intricacies" of it does much more harm than good - giving the users false expectations on "how easy" it is to get airflow running with docker-compose.
Yeah. Easy to start, Extremely difficult to diagnose and solve problems if you do not know exactly what you are doing. I think we should be very explicit that if user chooses docker-compose based on our quick start - the moment they touch the file, they are mostly on their own.
I'd love to hear other's thoughts about it, but this is the experience I have with users who think that our "docker-compose" is on our website, means that we will help them to solve their "learn how docker-compose works" problems.
Many times the comments I saw from the users was `But I am using the "official" docker-compose. It shoudl work!". I think this is false expectations and this PR is another attempt to be even clearer about the expectations users should have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22304:
URL: https://github.com/apache/airflow/pull/22304#issuecomment-1070824420
Another example https://github.com/apache/airflow/discussions/22322. Users have much bigger expectations when seeing our docker compose and I think we should be more explicit in telling them their expectations cannot be fulfilled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #22304:
URL: https://github.com/apache/airflow/pull/22304#issuecomment-1071179987
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22304: Stronger language about Docker Compose customizability
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #22304:
URL: https://github.com/apache/airflow/pull/22304#issuecomment-1071179987
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org