You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/25 08:14:37 UTC
[GitHub] [airflow] david-woelfle opened a new issue #16041: Explain why example docker-compose.yml file is not suited for production
david-woelfle opened a new issue #16041:
URL: https://github.com/apache/airflow/issues/16041
Hi everyone,
thank you so much for your outstanding work on this project! While going through the docs I found the section on [Running Airflow in Docker](https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html), where a [docker-compose.yml](https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml) file is provided. In the header of the this file it is stated:
> WARNING: This configuration is for local development. Do not use it in a production deployment.
Could you explain why this is the case? I.e. why this setting should not be used in production and also what must be done/changed if someone would like to run an Airflow environment in production based on this configuration example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847658496
+1 to this explanation.
For the record, we are using docker-compose in production for almost 100 DAGs and our deployment is very very stable and good-performing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847675720
Publishing something that is labelled as production ready is a lot of effort, and the maintenance effort required by the community to maintain it is much bigger. That's why we have to be very careful with labeling something as "production ready" when we officially publish it as community. We have to be prepared to support all kind of users, respond to their issues and fix them and possibly add new features continuously.
Just look how much time it took to graduate Helm chart (3 people worked full time for last ~ 2 months I believe to get it to the state where we could label it as "officially ready". It took me few months to release first version of Production Docker Images and then over a year to iterate on it and update and respond to issues and handle many cases that were initially unforeseen but users raised them and we responded and added them. And only now I think we are close to make it the image released as "Official Docker image" #10107. Just a project to get it there https://github.com/apache/airflow/projects/3 has 35 issues in "Done" state and two more are needed to complete it.
Different users have different expectations, configurations, databases, executors, deployments, scalability requirements etc. etc. What works for you @JavierLopezT - might not work for 100 other users and they might have different expectations. Don't forget how opinionated you are in the way how you run YOUR deployment and how those opinions my be different for many other people.
When we label something as "production-ready" we should be ready to respond to such issues. We need to have automated tests covering regressions in case anything changes, we need to have formal release process ready, we need to be able to analyse and diagnose and fix problems when they arise.
The current Docker-compose is by far not production ready. It is more of a "quick-start" if you want to try Airflow. No more, no less. There are already a number of issues "The docker compose does not work with LocalExecutor" or "The docker compose does not work with MySQL" raised . And yes - it does not, and this is by design. It is not supposed to. It's not production-ready. This is not it's purpose.
There are other issues that are already created around that:
* More examples of Docker Compose: #16031
* Add production ready docker-compose files: #8605
I think what @mik-laj proposed as having a wizard that generates docker-compose based on the expectations is a good start to go into "production-ready docker-compose" direction. But we have a loooong way to get there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] david-woelfle commented on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
david-woelfle commented on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847683714
Thank you @potiuk for the detailed answer! Now it makes much more sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847685310
Crystal clear @potiuk It's OK for you if I open an MR with a summary of your words for including it in the docker-compose file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847655897
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847721984
This file is also missing a few things that make this docker-compose unsafe for production:
- [CPU/memory resource limit](https://github.com/compose-spec/compose-spec/blob/002f33d2fa78425cd31f096444280cc4c9daf445/schema/compose-spec.json#L518-L519)- Each. the container has access to all system resources
- SSL - The connection to the container should be encrypted by Traefik / other proxy, or we should configure SSL in the webserver ([webserver] web_server_ssl_*).
- Containers use a local file system, but we should use volumes in a production environment.
We should also mention the possible ways of deploying DAGs, eg Git Sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847675720
Publishing something that is labelled as production ready is a lot of effort, and the maintenance effort required by the community to maintain it is much bigger. That's why we have to be very careful with labeling something as "production ready" when we officially publish it as community. We have to be prepared to support all kind of users, respond to their issues and fix them and possibly add new features continuously.
Just look how much time it took to graduate Helm chart (3 people worked full time for last ~ 2 months I believe to get it to the state where we could label it as "officially ready"). It took me few months to release first version of Production Docker Images and then over a year to iterate on it and update and respond to issues and handle many cases that were initially unforeseen but users raised them and we responded and added them. And only now I think we are close to make it the image released as "Official Docker image" #10107. Just a project to get it there https://github.com/apache/airflow/projects/3 has 35 issues in "Done" state and two more are needed to complete it.
Different users have different expectations, configurations, databases, executors, deployments, scalability requirements etc. etc. What works for you @JavierLopezT - might not work for 100 other users and they might have different expectations. Don't forget how opinionated you are in the way how you run YOUR deployment and how those opinions my be different for many other people.
When we label something as "production-ready" we should be ready to respond to such issues. We need to have automated tests covering regressions in case anything changes, we need to have formal release process ready, we need to be able to analyse and diagnose and fix problems when they arise.
The current Docker-compose is by far not production ready. It is more of a "quick-start" if you want to try Airflow. No more, no less. There are already a number of issues "The docker compose does not work with LocalExecutor" or "The docker compose does not work with MySQL" raised . And yes - it does not, and this is by design. It is not supposed to. It's not production-ready. This is not it's purpose.
There are other issues that are already created around that:
* More examples of Docker Compose: #16031
* Add production ready docker-compose files: #8605
I think what @mik-laj proposed as having a wizard that generates docker-compose based on the expectations is a good start to go into "production-ready docker-compose" direction. But we have a loooong way to get there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847711499
Hmm i think it would have to be a general description rather than specific to docker-compose we have a couple of those 'not production ready' things in Airflow code and i am not sure even where to put it.
But maybe we could indeed make some words about it in README or smth. I am not sure what others think about it. Anyone from the @apache/airflow-committers have an opinion ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847721984
This file is also missing a few things that make this docker-compose unsafe for production:
- [CPU/memory resource limit](https://github.com/compose-spec/compose-spec/blob/002f33d2fa78425cd31f096444280cc4c9daf445/schema/compose-spec.json#L518-L519)- Each. the container has access to all system resources
- SSL - The connection to the container should be encrypted by Traefik / other proxy, or we should configure SSL in the webserver ([webserver] web_server_ssl_*).
- Containers use a local file system, but we should use volumes in a production environment.
We should also mention the possible ways of deploying DAGs, eg Git Sync.
I. I also recommend the last one. discussions on Slack, where I explained the assumptions of this guide.
https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1621801810231000?thread_ts=1621711385.211600&cid=CCQ7EGB1P
> This example docker file has been prepared for the most popular configuration. I've only tested it with CeleryExecuttor. As for other executor configurations, I think that is beyond the scope of this article. The purpose of this article was to facilitate the launch of Airflow by someone who is unfamiliar with Airflow on its first run, so that they can test and check how Airflow works.
> One way to do this is to limit all the actions that the user takes. Now, to start Airflow they just need to run two simple commands:
> ```
> curl...
> docker-compose up
> ```
> You don't need to select a database engine, executor, or set other configuration options. You don't even need to know what it is to run Airflow.
> We should prepare separate guides on how to configure Docker-compose in other configurations . I even started [working on a tool](https://github.com/apache/airflow/issues/8605#issuecomment-759469443) that would allow us to generate several Docker-compose filesets based on user-supplied options, but I gave up work when Polidea was acquired by Snowflake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16041:
URL: https://github.com/apache/airflow/issues/16041#issuecomment-847711499
Hmm i think it would have to be a general description rather than specific to docker-compose. We have a couple of those 'not production ready' things in Airflow code and i am not sure even where to put it.
But maybe we could indeed make some words about it in README or smth. I am not sure what others think about it. Anyone from the @apache/airflow-committers have an opinion ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #16041: Explain why example docker-compose.yml file is not suited for production
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #16041:
URL: https://github.com/apache/airflow/issues/16041
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org