You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/07 08:32:42 UTC
[GitHub] [airflow] jdu opened a new issue, #26930: Isolated plugins / operators / hooks
jdu opened a new issue, #26930:
URL: https://github.com/apache/airflow/issues/26930
### Description
Because of the complexity of some of the plugins that are starting to come out, i'd like to propose a long-term goal / feature of finding a way to isolate plugins to prevent dependency issues, and stabilize integration points and problems over the longer term.
We're starting to run into issues where plugins and extensions to airflow have really complex dependency trees which cause upgrades and installs to become much more complex as well as major versions can break previously working plugins triggering rewrites as well as version checking to decide which interface to work against.
I know i'm light on details, but doing something similar to other tools where plugins are isolated in a separate process / environment, like LSP implementations in an editor, or how Sublime handles python-based extensions (I think they run as a background service when needed), where they are spawned as a secondary process and integrate with airflow through gRPC or something similar.
### Use case/motivation
This would make developing plugins and extensions easier as there can be a contractual interface that developers work against that can insulate them from changes in airflows code and classes. We're right now dealing with an issue in upgrading to 2.4.x where a plugin we rely on heavily has been a bit of a horror show on 2.4.x
a) Because it's own dependencies now conflict with airflows and some provider packages
b) Because the interface it worked against changed (albeit the interface it worked against was probably not the correct one in the first place).
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26930: Isolated plugins / operators / hooks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26930:
URL: https://github.com/apache/airflow/issues/26930#issuecomment-1271612587
converting it into discussion - as this is the stage it is in at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #26930: Isolated plugins / operators / hooks
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26930:
URL: https://github.com/apache/airflow/issues/26930#issuecomment-1271289366
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26930: Isolated plugins / operators / hooks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26930:
URL: https://github.com/apache/airflow/issues/26930#issuecomment-1271612179
This would be a complete change in how plugin mechanism work, because plugins are essentially executed in the same interpreter as the "main part" of the process. For example plugins create Python objects that then are used to instantiate views in the UI, or plugins create Timetable objects that are used by scheduler to calculate when to schedule the next dag run. if we were to isolate plugins and run them in separate process, we would have to execute them separately, have separate virtualenv for plugins and serialize the objects back and forth remotely between independently runniing python interpreters. This could be done with either mutliprocessing (with different python executable) or even have a separate plugin server that would have to be used for such purpose and reused between different components).
It's not impossible, of course but it is rather complex when it comes to deployment and requires very, very careful thinking not only about the APIs and interfaces used but also about performance implications. This is a complex change that would require [Airflow Improvement Proposal](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals) to be drafted, discussed and finalized in the [devlist](https://lists.apache.org/list.html?dev@airflow.apache.org], reviewing and interating over the docs, reaching consensus and passing successful voting. If you are up to the task (just be aware that it will likely take months just to discuss it so this is a marathon, not a sprint) - feel free to start it.
Just a watch-out: it's not needed in 9x% of cases where airflow is used so this is a niche case, that's why we have not invested into it. So there is a risk that we might even decide that we do not want to implement it, if from the discussion it will come out that either complexity of the implementation, or number of changes, or potential backwards incompatibilities or performance implications will be too much of a problem. Simply the gains to achieve might be much lower than cost involved in maintenance and risks of breking existing installation. So any proposal there must take into account and discuss all the aspects that might lead to higher cost of maintenance, complexity, performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #26930: Isolated plugins / operators / hooks
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26930: Isolated plugins / operators / hooks
URL: https://github.com/apache/airflow/issues/26930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jdu commented on issue #26930: Isolated plugins / operators / hooks
Posted by GitBox <gi...@apache.org>.
jdu commented on issue #26930:
URL: https://github.com/apache/airflow/issues/26930#issuecomment-1271306304
As an additional benefit this would free it up so that devs could develop plugins in other languages as well which would increase the potential number of maintainers and give a boost to the plugin ecosystem around airflow potentially.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org