[GitHub] [airflow] potiuk commented on issue #24794: Using _PIP_ADDITIONAL_REQUIREMENTS in official docker-compose

[GitBox <gi...@apache.org>]

potiuk commented on issue #24794:
URL: https://github.com/apache/airflow/issues/24794#issuecomment-1172978715

   The `_PIP_ADDITIONAL_REQUIREMENTS` is strongly discouraged as development only feature. 
   
   https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements
   
   > Installing requirements this way is a very convenient method of running Airflow, very useful for testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in production environment. We have deliberately chose to make it a development/test dependency and we print a warning, whenever it is used. There is an inherent security-related issue with using this method in production. Installing the requirements this way can happen at literally any time - when your containers get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen literally any time. This opens you up to a serious vulnerability where your production environment might be brought down by a single dependency being removed from PyPI - or even dependency of your dependency. This means that you put your production service availability in hands of 3rd-party developers. At any time, any moment including weekends and holidays those 3rd party 
 developers might bring your production Airflow instance down, without you even knowing it. This is a serious vulnerability that is similar to the infamous [leftpad](https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/) problem. You can fully protect against this case by building your own, immutable custom image, where the dependencies are baked in. You have been warned.
   
   And you are using it wit hdocker-compose which is developer-only, quick-start feature:
   
   https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#production-readiness
   
   > DO NOT expect the Docker Compose below will be enough to run production-ready Docker Compose Airflow installation using it. This is truly quick-start docker-compose for you to get Airflow up and running locally and get your hands dirty with Airflow. Configuring a Docker-Compose installation that is ready for production requires an intrinsic knowledge of Docker Compose, a lot of customization and possibly even writing the Docker Compose file that will suit your needs from the scratch. It’s probably OK if you want to run Docker Compose-based deployment, but short of becoming a Docker Compose expert, it’s highly unlikely you will get robust deployment with it.
   
   > If you want to get an easy to configure Docker-based deployment that Airflow Community develops, supports and can provide support with deployment, you should consider using Kubernetes and deploying Airflow using [Official Airflow Community Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html).
   
   
   So, if you ask, if any of those work out of the box, the answer is - "depends". If you are not able to diagnose and fix problems coming from using this "development features" they you simply should not do it. Those options are not  for users who open issues when they find some errors. but for users who can understand diagnose, and fix problems they see when they use it. The comments quoted above mean that if you do not know exactly what you are doing and use those features, you are absolutely on your own to do the investigations, understand, and fix the problems it causes.
   
   You shoudl embrace it and deal with the consequences of it.
   
   This also basically means that anything you find not working should - at best - be turned in a PR when you find a fix. Slightly better approach will be to document in detail what are your finding as a developer (not user) that you already made to make it works. Posting logs, hypothesis on what does not work and what you have done to fix it, are about the only good approaach you can take there. The other one is not to use those "development-only" features and focus on our Helm Chart (https://airflow.apache.org/docs/helm-chart/stable/index.html) and using your own custom/extrended  images (https://airflow.apache.org/docs/docker-stack/build.html) rather than taking time of others without understading the "developer stack" for Airflow. 
   
   Which I hertily invite you to.
   
   * If you are developer and you are able to analyse and provide fixes to development features - please by all means do - but come with solutions or discussions and not "bugs" if you find a problem that you cannot solve by reading and following the source code
   * if you are a user and want to raise a "bug" or "feature request" against the user-facing features - and you have a reproduction case, logs etc. - feel free to do so as well
   
   Converting it into a discussion. Bugs should not  be raised against deveiloper features.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org