[GitHub] [airflow] kaxil commented on pull request #11526: Adds automated installation of dependent packages

[GitBox <gi...@apache.org>]

kaxil commented on pull request #11526:
URL: https://github.com/apache/airflow/pull/11526#issuecomment-709247235


   > Hey @kaxil @ash - as mentioned on slack - I need to get that one merged in order to solve the Docker building problem that is now causing Kubernetes failures. Simply - when installing airflow[EXTRAS] I need to install also all the providers. This has not been possible yet because we do not have those packages in PyPI. This one is needed to fix #11490
   > 
   > This PR will likely fail in a few cases because Airflow will not find the right packages in PyPI to install:
   > 
   > * when installing Docker (#11490)
   > * when installing Airflow locally from sources (#11489) with editable (likely not) or just regular local install with 'pip install .'
   > 
   > Now the question is how we approach it:
   > 
   > 1. We can publish both airflow and provider's packages in PyPI. Though the result will be that the providers will be installed in the version from PyPI.
   > 2. We can try to implements some workaround to be able to install provider packages in Docker without having PyPI. This is possible, I can prepare packages before building docker, place them in 'docker-context-files' and install them from there using ``--find-links` option of PyPI which will install the packages from there. This is what I am currently leaning towards - in which case I will not need to have anything in PyPI (which for alphas might be a good idea).
   > 3. We could also just publish the providers in PyPI and do not install Airflow.
   > 
   > Another point is to solve the local installation (#11489) but I think this one will have to be installed differently because you for sure want to have local providers installed. This I am going to likely solve by a custom approach to not exclude packages in setup.py when installing locally (providing that I will be able to reliably detect the case that I am installing airflow from sources and not building the package which should be possible).
   > 
   > But I have to double check that one.
   
   I like (2)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org