[GitHub] [airflow] potiuk commented on pull request #11529: Behaviour to install all airflow providers added

[GitBox <gi...@apache.org>]

potiuk commented on pull request #11529:
URL: https://github.com/apache/airflow/pull/11529#issuecomment-708572246


   Hey @ashb @kaxil  @turbaszek, others. I looked how to solve the installation problem described in #11489 and failing Kubernetes builds (caused likely by the provider split) and I believe i found the best approach. I think we want to keep this:
   
   * in any setup where we use packages, we want to install separately airflow and providers
   * in any setup where we use sources, we want to install airflow + providers together for development convenience
   
   The proposal I have is a variable INSTALL_PROVIDERS_FROM_SOURCES. When this flag is "true", airflow will install providers from sources, when it is missing or anything else but "true", it will install only airflow core.  By default in Breeze and when building images from sources I set this variable to "true" (but it can be set to false by --skip-installing-airflow-providers flag so that you can also install a "bare" airflow in Breeze or prepare a "bare" image without any providers easily. Together with in-progess #11464 (I will have to add conditional dependencies there and rebase on top of this) it will have exactly the desired effect:
   
   * when installing airflow with Breeze from sources (both prod and CI), we continue installing all providers from sources like we have in 1.10. Extras will not cause installing of apache-airflow-providers-* as those dependencies will be disabled in this case. You can also disable installing all providers by ``-skip-install0ing-airlfow-providers``
   
   * when installing airflow with Breeze  from PyPI or GitHub, only the providers required by "extras" will be installed
   
   * when installing airflow locally with "." (without -e) only bare Airflow will be installed, provders will only be available if you happen to be in the airflow directory or if you install provider packages manually. You can also set INSTALL_PROVIDER_SOURCES="true" before installation and then all providers will be installed as in 1.10.
   
   * when installing airflow locally with -e '.'  - providers will be automatically installed and available. Since 'airflow' is taken direclty from sources, providers are there and they will be importable.
   
   
   I tested all the above scenarios and I think it makes perfect sense. Let me know what you think.
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org