You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "klesh (via GitHub)" <gi...@apache.org> on 2026/09/02 03:36:30 UTC
[I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
klesh opened a new issue, #9092:
URL: https://github.com/apache/devlake/issues/9092
## Proposal: Deprecate the Python plugin framework and consolidate on Go
### Summary
Propose to deprecate the Python plugin framework (`backend/python/`, a.k.a. PyDevLake) and remove it from the codebase after a 3-month deprecation window. Going forward, all plugins should be written in Go. This reduces maintenance overhead, simplifies the Docker image and CI, and focuses contributor effort on a single, well-supported runtime.
### Background
The Python plugin framework was introduced in Feb 2023 (#3702, #4438) to lower the barrier for writing plugins. In practice it never saw adoption:
- Only **one** plugin ever shipped on it: `azuredevops` (Python).
- The `dbt` Python plugin was already removed on 2026-08-31 (#8970).
- A fully-featured Go implementation, `azuredevops_go`, now exists and is actively developed (on-premises support landed 2026-08-16, #9014).
### Evidence: Python plugins are not being maintained
| Plugin | Last meaningful commit | Note |
| --- | --- | --- |
| `azuredevops` (Python) | 2025-12-31 `feat: add environment_pattern` (#8671) | No feature/fix since; only deps refresh & runtime bumps |
| `pydevlake` framework | 2025-12-31 (same as above) | Last real fix: #8957 on 2026-07-15 (missing composite PKs) |
| `dbt` (Python) | removed 2026-08-31 (#8970) | Already gone |
In the last ~12 months, the only commits touching `backend/python/` are dependency refreshes, Poetry/uv lock bumps, a Python 3.9→3.11 runtime bump, and a single primary-key fix. Meanwhile the Go side shipped 40+ plugins and continues to receive features weekly.
### Benefits
1. **One runtime to maintain.** Eliminate the parallel PyDevLake framework, its RPC bridge, `pydevlake` package, `uv`/Poetry tooling, and Python-specific test/build scripts.
2. **Simpler Docker build.** Today `backend/Dockerfile` is based on `python:3.11-slim-bookworm`, installs python3-dev/pip/uv, copies `python/`, and runs `build.sh`. After removal the image can be a slim Go-based image — smaller, fewer CVEs, faster builds. `Dockerfile.local`'s "empty python plugins dir" workaround also goes away.
3. **Simpler CI.** Two workflows (`.github/workflows/test.yml`, `test-e2e.yml`) and Make targets (`python-dep`, `python-unit-test`, `build-python`, `build-pydevlake`) can be dropped.
4. **Clearer contributor story.** New plugin authors get one path (Go) with 40+ reference implementations instead of choosing between two frameworks where one is unmaintained.
5. **No functional loss.** The only remaining Python plugin (`azuredevops`) is superseded by `azuredevops_go`, which already has broader feature coverage (Server/on-prem support).
### Plan & Timeline
| Phase | Window | Actions |
| --- | --- | --- |
| **1. Warning** | T+0 (on issue merge) | Add deprecation notices to: `backend/python/README.md`, plugin `README.md`s, server startup log line, and `config-ui` plugin selection UI. Mark `azuredevops` (Python) as deprecated in the plugin registry; point users to `azuredevops_go`. |
| **2. Freeze** | T+1 month | Stop accepting new features/fixes for Python plugins. Only critical security fixes land. Add a CI gate rejecting new files under `backend/python/plugins/`. |
| **3. Migration assist** | T+1~T+3 months | Ship a migration guide + script to move any remaining `azuredevops` (Python) connections/scopes to `azuredevops_go`. Document data-layer equivalence. |
| **4. Removal** | T+3 months | Delete `backend/python/`, PyDevLake, the Python Dockerfile stage, `make python-*` targets, and Python CI jobs. Remove `azuredevops` (Python) from the plugin registry. |
Total deprecation window: **3 months** from the date this proposal is accepted.
### Risks & Mitigations
- **Existing users of `azuredevops` (Python):** Mitigated by `azuredevops_go` parity + migration guide in Phase 3. The Go plugin already covers cloud + on-prem.
- **External contributors with in-progress Python plugins:** The 3-month window + the warning phase gives them time to port. No private Python plugins are known to exist in the wild.
- **Reverting later:** Not needed; Go covers all current and planned use cases.
### What we need
- 👍 / 👎 from maintainers and community on deprecating Python.
- Owners to confirm `azuredevops_go` feature parity with the Python `azuredevops` plugin (esp. CI/CD environment extraction).
- Volunteers to write the Phase-3 migration guide.
---
/cc @merico-dev/devlake-maintainers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-5509355960
> At my company we have developed some python plugins (redmine, glpi, zephyr), have also added some additional features to the azure devops python plugin (allowing to extract issues and qa entities), and have made some minor improvements to the python plugin framework. We haven't submitted yet these changes to this repo but we were planning to. We are using this plugins internally and with some of our clients.
>
> I really think that having python framwork lowers the barrier for implementing plugins and even though is less performant, I prefer them since in general they are easier to maintain due to the bigger python community and in general plugins take significantly less code and ceremony (I tried re implementing some in go and the resulting code is quite verbose and a little harder to read).
>
> If we contribute these changes would you consider keeping this support?
Thanks for the feedback! We always welcome contributions. The main challenge right now is that we don't have enough maintainers actively working on this part. We're happy to keep it around if more people step up to help maintain it over the next 3 months.
I've got some ideas to improve the architecture, but working around Python makes it pretty tough. 😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "DoDiODev (via GitHub)" <gi...@apache.org>.
DoDiODev commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-6442032819
Sounds good — plugins first makes sense. When you get to the dependency update, happy to share our Pydantic v2 / SQLModel migration as a starting point (or open it as a PR after your plugins land, whichever you prefer), so we don't duplicate work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-6432201343
> 10\. [@klesh](https://github.com/klesh) thank you very much for taking the time to consider this and answer.
> I think is doable, but as I mention before, I think is easier to maintain (since python community is bigger in general, and in particular for my team), and I do think that python plugins lower the barrier for contributions to the project.
> I thought that more people from the community would step up and add similar comments, but I don't know how big is the apache devlake community, it's main technical background, and if this announcement has already been noticed by the community. I know this is very difficult, and sincerely don't want to add extra burden to the maintenance of the project.
> If you, and the rest of the pmc think that is better to not send the python plugins and the changes we have done to try to keep the python framework alive, I would totally understand and just adapt to your vision and were the project is going. Just want to know if you consider the addition of such python plugins and some small changes to the framework worth to try keeping the python adapter in place or not.
> We have in fact 3 plugins that may contribute as they currently are (redmine, zephyr & glpi) and we plan to implement a few other plugins (jmeter, datadog, xunit) that we still need to review the full scope of them to see if makes sense to contribute as well and how they fit the existing apache devlake model.
> Thank you again to consider this, spending time and effort in reading & answering, and hope that we can come up with the best for the entire community. And obviously, thank you for maintaining apache devlake which I think is a great project and value it.
> Regards
Thank you for the detailed explanation. I appreciate your passion, and if that is your plan, I think we could take another shot at Python. I will remove the deprecation, and you can start contributing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "rabelenda (via GitHub)" <gi...@apache.org>.
rabelenda commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-5509229968
At my company we have developed some python plugins (redmine, glpi, zephyr), have also added some additional features to the azure devops python plugin (allowing to extract issues and qa entities), and have made some minor improvements to the python plugin framework. We haven't submitted yet these changes to this repo but we were planning to.
I really think that having python framwork lowers the barrier for implementing plugins and even though is less performant, I prefer them since in general they are easier to maintain due to the bigger python community and in general plugins take significantly less code and ceremony (I tried re implementing some in go and the resulting code is quite verbose and a little harder to read).
If we contribute these changes would you consider keeping this support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "DoDiODev (via GitHub)" <gi...@apache.org>.
DoDiODev commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-6441636654
Thanks for reverting the deprecation (#9259) — good to see the Python side getting contributors again.
Since PyDevLake is staying, two maintenance points that will matter for new plugins:
1. **Pydantic v1 / SQLModel 0.0.8.** All three `poetry.lock` files pin `pydantic 1.10.26`, `sqlmodel 0.0.8` and, through it, `SQLAlchemy 1.4.41` (2022). Pydantic v1 does not support Python 3.14+, so it also blocks any runtime upgrade.
2. **Python 3.11 reaches end of life in October 2027.** `pydevlake` is pinned to `>=3.11,<3.12`, so moving to a supported runtime needs (1) first.
We have a Pydantic v2 / SQLModel 0.0.38 migration of `pydevlake`, the `azuredevops` plugin and the test fakeplugin (about 15 files, ~200 lines, no Go changes). It applies cleanly to current `main` and the Python unit tests pass. We'd be happy to open it as a PR.
@rabelenda — since you plan framework changes and new plugins: would you prefer the v2 migration to land **before** your contributions (so new plugins are written against v2), or after? We'd like to avoid conflicts with your work.
No rush from our side; this is only meant to coordinate the order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "rabelenda (via GitHub)" <gi...@apache.org>.
rabelenda commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-6441730389
Thank you very much to give it another shot to the python plugin ecosystem ❤️ .
I will submit pull requests for the plugins I have, so you can see more value in a short term. And then I can update those dependencies and submit a PR for that as well if you like that.
Regards
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-6334511700
@rabelenda Will it be possible to convert them to golang plugins using AI? It should be doable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "rabelenda (via GitHub)" <gi...@apache.org>.
rabelenda commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-6377999451
@klesh thank you very much for taking the time to consider this and answer.
I think is doable, but as I mention before, I think is easier to maintain (since python community is bigger in general, and in particular for my team), and I do think that python plugins lower the barrier for contributions to the project.
I thought that more people from the community would step up and add similar comments, but I don't know how big is the apache devlake community, it's main technical background, and if this announcement has already been noticed by the community. I know this is very difficult, and sincerely don't want to add extra burden to the maintenance of the project.
If you, and the rest of the pmc think that is better to not send the python plugins and the changes we have done to try to keep the python framework alive, I would totally understand and just adapt to your vision and were the project is going. Just want to know if you consider the addition of such python plugins and some small changes to the framework worth to try keeping the python adapter in place or not.
We have in fact 3 plugins that may contribute as they currently are (redmine, zephyr & glpi) and we plan to implement a few other plugins (jmeter, datadog, xunit) that we still need to review the full scope of them to see if makes sense to contribute as well and how they fit the existing apache devlake model.
Thank you again to consider this, spending time and effort in reading & answering, and hope that we can come up with the best for the entire community. And obviously, thank you for maintaining apache devlake which I think is a great project and value it.
Regards
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "rabelenda (via GitHub)" <gi...@apache.org>.
rabelenda commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-6323808766
@klesh should I try submitting a redmine & zephyr python plugins, and fixes we have developed in python framework? will it be rejected due to the notice? Or will it help avoiding/reverting this deprecation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Proposal] Deprecate Python plugin framework and consolidate on Go [devlake]
Posted by "rabelenda (via GitHub)" <gi...@apache.org>.
rabelenda commented on issue #9092:
URL: https://github.com/apache/devlake/issues/9092#issuecomment-6452906341
Here is the first pull request for the redmine plugin: https://github.com/apache/devlake/pull/9265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org