[GitHub] [airflow] turbaszek commented on issue #13364: Add virtual env creation in Celery workers

[GitBox <gi...@apache.org>]

turbaszek commented on issue #13364:
URL: https://github.com/apache/airflow/issues/13364#issuecomment-753446898


   > Once getting to a point when you want to create cluster for different types of python tasks and you've multiple teams working on the same cluster, you need to start splitting into different python packages the business login code to allow better versioning control and avoid the need of restarting the workers when deploying new util code.
   
   In my opinion if you start to have multiple teams working on the same Airflow deployment then virtualenv solves the problem only on Python level which in my opinion is not sufficient. I personally would be in favour of worker/queue per env/team (as @ashb suggested). This seems to be more reliable approach (managing deps on deployment level) than managing dependencies via application (Airflow) which main task is scheduling and executing tasks.
   
   But as @potiuk said - we should move this discussion to devlist to gather wider feedback.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org