You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "CamilleTeruel (via GitHub)" <gi...@apache.org> on 2023/04/20 09:53:30 UTC
[GitHub] [incubator-devlake] CamilleTeruel opened a new issue, #4986: [Refactor][pydevlake] Refactor plugin registration
CamilleTeruel opened a new issue, #4986:
URL: https://github.com/apache/incubator-devlake/issues/4986
## What and why to refactorIssue title:
Currently, the remote plugins are registered this way:
- the route `/plugins/register` is created in `backend/server/api.go`
- `backend/python/plugins/start.sh` is executed
- this executes the `startup` command of each plugin
- the plugin registers itself by POSTing its details to `/plugins/register`
This is not ideal, because:
- this is more complicated than it needs to be
- this exposes the registration route to the public, which can be a security issue
- the `start.sh` and `build.sh` scripts have to make assumptions about how to build all plugins (e.g. that they use `poetry`)
## Describe the solution you'd like
Instead, we could add a Makefile to each plugin, with `build` and `run` targets.
The `build` target is executed by `backend/Makefile` to build each plugins.
Then at startup, each plugin is registered by using info returned by `make run plugin-info`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on issue #4986: [Refactor][pydevlake] Refactor plugin registration
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on issue #4986:
URL: https://github.com/apache/incubator-devlake/issues/4986#issuecomment-1524452225
@klesh sure I can do that, but at a high level I'm thinking of a service based architecture so we can leverage standard protocols and design patterns. Current architecture is too monolithic and all python (and Go code) run in the same container as the API server which is a problem as far as load balancing in a multi-tenant environment (unless we are going for separate deployments per client?). It also uses a custom shell-based protocol that we made up to interface Go and Python, which (IMO) is best if replaced with REST or RPC calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4986: [Refactor][pydevlake] Refactor plugin registration
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4986:
URL: https://github.com/apache/incubator-devlake/issues/4986#issuecomment-1524607693
@keon94 no problem, we are open to all possibilities.
Share your thoughts pls
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 commented on issue #4986: [Refactor][pydevlake] Refactor plugin registration
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 commented on issue #4986:
URL: https://github.com/apache/incubator-devlake/issues/4986#issuecomment-1522287716
So basically have the server make a shell call to a script that returns the plugin details, rather than the plugins calling back into the server? This can work but you also have to worry about the custom file descriptor we use and correctly propagating that from Go to Shell to Python. I don't think that's trivial. Also, If we're going to do a mass refactor, I honestly think we should just move away from cmd calls and have python run as a separate RPC service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] CamilleTeruel commented on issue #4986: [Refactor][pydevlake] Refactor plugin registration
Posted by "CamilleTeruel (via GitHub)" <gi...@apache.org>.
CamilleTeruel commented on issue #4986:
URL: https://github.com/apache/incubator-devlake/issues/4986#issuecomment-1525192023
> So basically have the server make a shell call to a script that returns the plugin details, rather than the plugins calling back into the server?
Exactly, python plugins are loaded by walking `REMOTE_PLUGIN_DIR` looking for `run.sh` executables just after loading go plugins in `loader.go`.
> This can work but you also have to worry about the custom file descriptor we use and correctly propagating that from Go to Shell to Python. I don't think that's trivial.
This still uses invoker and bridge abstraction that take care of the extra file descriptor.
> Also, If we're going to do a mass refactor, I honestly think we should just move away from cmd calls and have python run as a separate RPC service.
This is not a mass refactor. This is just changing plugin registration flow. I agree that we should move away from cmd calls. This change is a necessary first step in that direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #4986: [Refactor][pydevlake] Refactor plugin registration
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on issue #4986:
URL: https://github.com/apache/incubator-devlake/issues/4986#issuecomment-1524390833
I'm concerned from the security perspective.
@CamilleTeruel Can you draw a flow-chart or architecture diagram to show how the new mechanism works?
@keon94 Would you like to do the same if you have another approach?
No matter what we opt for in the end, the security of the new approach must be enforced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] keon94 closed issue #4986: [Refactor][pydevlake] Refactor plugin registration
Posted by "keon94 (via GitHub)" <gi...@apache.org>.
keon94 closed issue #4986: [Refactor][pydevlake] Refactor plugin registration
URL: https://github.com/apache/incubator-devlake/issues/4986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org