[GitHub] [airflow] potiuk commented on issue #26930: Isolated plugins / operators / hooks

[GitBox <gi...@apache.org>]

potiuk commented on issue #26930:
URL: https://github.com/apache/airflow/issues/26930#issuecomment-1271612179

   This would be a complete change in how plugin mechanism work, because plugins are essentially executed in the same interpreter as the "main part" of the process. For example plugins create Python objects that then are used to instantiate views in the UI, or plugins create Timetable objects that are used by scheduler to calculate when to schedule the next dag run. if we were to isolate plugins and run them in separate process, we would have to execute them separately, have separate virtualenv for plugins and serialize the objects back and forth remotely between independently runniing python interpreters. This could be done with either mutliprocessing (with different python executable) or even have a separate plugin server that would have to be used for such purpose and reused between different components).
   
   It's not impossible, of course but it is rather complex when it comes to deployment and requires very, very careful thinking not only about the APIs and interfaces used but also about performance implications. This is a complex change that would require  [Airflow Improvement Proposal](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals) to be drafted, discussed and finalized in the [devlist](https://lists.apache.org/list.html?dev@airflow.apache.org], reviewing and interating over the docs, reaching consensus and passing successful voting. If you are up to the task (just be aware that it will likely take months just to discuss it so this is a marathon, not a sprint) - feel free to start it.
   
   Just a watch-out: it's not needed in 9x% of cases where airflow is used so this is a niche case, that's why we have not invested into it. So there is a risk that we might even decide that we do not want to implement it, if from the discussion it will come out that either complexity of the implementation, or number of changes, or potential backwards incompatibilities or performance implications will be too much of a problem. Simply the gains to achieve might be much lower than cost involved in maintenance and risks of breking existing installation. So any proposal there must take into account and discuss all the aspects that might lead to higher cost of maintenance, complexity, performance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org