You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/06/02 06:57:39 UTC
[GitHub] [incubator-dolphinscheduler] gaojun2048 opened a new issue #2869: [Proposal] DolphinScheduler Plugin
gaojun2048 opened a new issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869
Hello everyone
The subject of this email is to initiate a discussion about DS Pluginization. DS Plugin related architecture adjustment is very important, so I will use Chinese and English to describe. （中文请往最后看）
## aims
The core purpose of DS Plugin is to decouple the upper core logic of DS from the specific implementation. I have been involved in the second development of presto recently. I have learned a lot of excellent design concepts about plug-in from the presto architecture. These design concepts have helped presto become a very easy to expand project. Now if you want to develop a presto-based one Your own query plugin is very simple.
We hope that DS can also have such excellent expansion capabilities, because each job type in DS can be a plug-in, which can be implemented by different people. Therefore, we hope that the development of DS Job Type will be simpler, so that in the future, more contributors will be attracted to participate. Now if I want to develop a new job is very difficult, I need to look at a lot of code, have a deep understanding of the source code of DS to know where to start. This is very unfriendly to new developers who want to participate in the DS community.
### Here are some thoughts on DS's current design:
1. The alarm collection and sending logic in the alert service is decoupled from the specific alarm methods (email, WeChat, pegs, etc.). Dolphinscheduler-alert should not focus on how to send the alarm. Sending email or WeChat should be specific The alarm implementation plugin works.
2. For workflow tasks, dolphinscheduler-server should only need to pay attention to the execution of DAG tasks, and each task should have a unified standard interface for dolphinscheduler-server to call. For example, MR task, when presto-worker executes MR task, only need to know the interface and corresponding implementation class that the task implements. The specific implementation of the MR task should be put into the plugin to be developed by the developer of this task.
3. For the front end, the front end needs to cooperate with the plug-in design. Now the front end is hard-coded, such as MR tasks and spark tasks. The front-end configurable parameters of each task node are written by the front-end personnel, which is not Conducive to plug-in. Because it is difficult for us to ask the implementer of a plugin to develop the front-end UI for this plugin.
### Plan
Based on the above considerations, I plan to refer to the use of Java SPI in presto, and also use Java SPI in DS to implement Plugin design.
#### Server
I plan to add a dolphinscheduler-spi module. All plug-in interfaces that can be implemented are put into the spi module. Core modules such as dolphinscheduler-server only write code based on the interfaces in the spi. The specific implementation of all plugins are put into a dolphinscheduler-plugin module (for example, alarm plugins are put into dolphinscheduler-alert-plugin, Job plugins are put into dolphinscheduler-job-plugin) Each plugin is in dolphinscheduler-alert-plugin Or create your own modules under the dolphinscheduler-job-plugin module. These plug-in modules need to rely on dolphinscheduler-spi and implement the interface provided in dolphinscheduler-spi.
#### front end
The plug-in implementer tells the front end through a port the parameters that need to be configured when the plug-in needs to be used. For example, the email alert plugin needs to tell the front-end page to fill in the email recipient and the cc; the nail alert needs to be filled in by the front end for the target group; for the Job plugin, the MR plugin needs the front-end configuration parameter jar package path, mr parameter, etc. . I think so about this piece. We design an enum in the plug-in that supports all the front-end control types. Then each plug-in can return the parameters that it needs to display on the front-end through a getParams interface method. Such as input, radio, selector, area, etc.; when the system starts to load the plug-in, the system will save the json generated by the params corresponding to each plug-in to the database for use by the front end. The parameters of each task type can be divided into general parameters and custom parameters. The general parameters include (node name, running flag, description, priority, etc., common parameters for each task type), and the parameters can be written by the front end. Every Job Type is the same. The custom parameter is a params json that takes this parameter from the database, and then generates UI elements (input, radio, selector...) in the front end according to the information of each parameter defined in json. **But there is a problem here. This method cannot support the form elements that require multi-level linkage, because it is more difficult to describe js events through json. Please help us find a way**.
### I hope that the design of DS Plugin can bring the following benefits:
1. Decoupling the core business logic from the specific implementation (such as decoupling the alarm logic from the specific alarm implementation method; decoupling the task execution from the specific task implementation in the DAG) can improve the scalability of the DS architecture.
2. For plug-in developers, there is no need to pay attention to how the core logic of the DS upper layer works. They only need to pay attention to the interfaces provided to them by the SPI layer, and then implement these interfaces. This will significantly reduce the difficulty of getting started with new task types.
3. The implementation layer of the plug-in can only use the interfaces and classes open in the SPI. In addition, the plug-in implementation layer should not rely on the code of other modules in the DS. If the plug-in layer wants to use a tool class in the dolphinscheduler-common module, the correct approach should be to first migrate the tool class from the dolphinscheduler-common module to the dolphinscheduler-spi. Or put the tool classes into a single module, do not put it together with the upper layer of the core business logic code that may be frequently modified, to ensure that the functional boundary of the module that can be dependent on other plugins is as small as possible. At present, I just put Some utils are put into the dolphinscheduler-spi. It can be seen that even the spi module's pom file needs to add some dependencies that must be used in utils, such as slf4j, commons-xx, poi, etc. In order to ensure that the problem of dependencies can be discovered during the compilation phase, rather than being found at runtime, I will check the dependencies in the SPI module through a maven plug-in, and require all third-party dependencies (including transitive dependencies) used in SPI All must be clearly defined and specified, and the scope is not allowed to be below the provided level, which will result in a large number of tool classes placed in the spi, there will be more dependencies in the spi, but it should not be a big problem. A little bit of help everyone express their opinions. The ultimate goal is that the code between the implementation of each different plug-in will not affect each other. For example, the implementation of MR jobs will not affect the code of Spark jobs. For reviewers of the code, it can also greatly improve the efficiency of the review.
4. As long as the code of the SPI layer is not modified, in theory, the modification of the dolphinscheduler-server layer and the code modification of the plug-in layer do not affect each other, and everyone develops based on the same set of SPI interfaces.
5. For users of DS, they can choose the plug-in they need. In special cases, they only need to compile the code of the plug-in they need to integrate the new plug-in into their existing environment.
## Progress
At present, I plan to start with the plug-in of Alert and make a try. Because the Alert module is relatively simple, it is suitable for a rapid transformation.
1. I have completed a custom maven plug-in, its role is to automatically check the spi and plug-in related dependencies, and can support the addition of <packaging>dolphinscheduler-plugin</packaging> logo in the pom file to allow DS to compile Automatically generate related META-INF/services files. For detailed information, please refer to: https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin
2. https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1
This pr is part of the code development that I carried out on the dev branch. At present, the plug-in transformation of SPI and AlertServer has been implemented.
3. At the plug-in implementation layer, the transformation of the mail alert plug-in has been completed.
4. In the dolphinscheduler-alert module, the development of plug-in configuration and loading function is completed. At present, the plug-in supports the configuration of alert.plugin.binding=xxx/pom.xml in alert.properties to specify the loading of the plug-in and the deployment and runtime on the server by configuring alert.plugin.dir=/xxx/xxx / Directory to load plugins. **Of course, the alert.plugin.dir parameter requires DS to place all plugins in the directory specified by alert.plugin.dir during package deployment. I have not done this function yet. Everyone thinks where to put the plugin directory, I suggest putting it in the ${DS_INSTALL_DIR}lib/plugin directory**.
There are many comments in the code, the main purpose is to make it easier for everyone to understand my ideas through comments.
There are some //TODO comments in the code. These are the needs that need to be further modified. At present, I have not figured out how to implement them.
## Current issues
1. The plug-in will return the parameters that it needs to be displayed by the front end through the getParams method in PluginFactory. For example, the email alert needs the front end to fill in the recipient and copy the person. Under normal circumstances, the front end does not know which alarm plug-ins are available, then the front end needs to read the list of available alarm plug-ins currently configured by the system from a place, and then the name of the plug-in (such as'mail'), each The parameters that the plug-in needs to fill are displayed on the front end. Where should this information be kept? Want to design a mysql table to save this information?
2. If at time A, there are email alert plugins and nail alert plugins in the system, the user has enabled these two plugins on the front-end interface, and has filled in the information of email recipients, nail recipients, and so on. When the time comes to time B, the user deletes the email alert plug-in from the system. Should the user fill in the recipient information on the email alert at this time? How to keep it? If it is not retained, the user adds the jar of the email alert plugin to the system again, does the user need to restart to fill in the email recipient information?
The above problems are some of the problems left by the current alarm plug-in. I hope everyone can come up with ideas together.
Of course, my strength alone cannot complete the transformation of the entire plug-in. I hope to get your support and help, give me suggestions, or complete some sub-tasks, thank you.
## Task list (not open for the time being, you need to wait for the above question to discuss the result before deciding on the task list)
At present, the following tasks need to be completed.
1. Design the database plug-in table. When the plug-in is loaded, the ID of the plug-in and the front-end display name (Chinese name/English name) are required. The parameter list that the plug-in needs to display on the front-end is saved in the database. In this way, the front-end page can dynamically know which alarm plug-ins are available in the system, what parameters each plug-in needs to fill in, and what type of form elements each parameter is.
2. I just completed the transformation of the email alarm, and we also need to modify the WeChat alarm plugin.
3. Nail alarm plug-in modification.
4. Transformation of enterprise WeChat alarm.
5. When packaging, you need to put the plugin in a special plugin directory.
6. ...
-----------------------------------------------------------------------------------------------
大家好
这个邮件主题是发起一个关于DS Plugin 化的讨论。DS Plugin相关的架构调整非常重要，所以我将用中文和英文两种语言来进行描述。
## 目标
DS Plugin的核心目的是将DS的上层核心逻辑与具体的实现解耦。我最近一直在参与presto的二次开发工作，从presto的架构中学习到了很多关于插件化的优秀的设计理念，这些设计理念帮助presto成为一个非常容易扩展的项目，现在如果你想基于presto开发一个自己的查询插件是非常简单的事情。
我们希望DS也能有这样优秀的扩展能力，因为在DS中每一个Job Type都可以是一个插件，都可以被不同的人来实现。因此我们希望DS Job Type的开发变得再简单一点，这样在未来也可吸引更多的贡献者参与进来。现在如果我想开发一个新的Job是非常困难的，我需要看大量的代码，对DS的源码有很深入的了解才能知道该从何处下手。这对那些想参与到DS社区的新的开发者来说是非常不友好的。
### 下面是我对DS目前的设计的一些思考：
1. alert服务中的告警收集与发送逻辑与具体的告警方式（邮件、微信、钉钉等）进行解耦，dolphinscheduler-alert中不应该去关注告警该怎么发，发邮件或者发微信应该是具体的告警实现插件工作。
2. 对于工作流任务来说，dolphinscheduler-server应该只需要关注DAG任务的执行，每个任务都应该有统一的标准的接口来让dolphinscheduler-server调用。比如MR任务，presto-worker执行MR任务时，只需要知道任务实现的接口和对应的实现类。MR任务的具体实现应该放到插件里由这个任务的开发者来开发。
3. 对于前端来说，前端需要配合插件化的设计，现在前端是写死的，比如MR任务，spark任务这些，每个任务节点的前端可配置参数都是由前端人员写死的，这不利于插件化。因为我们很难要求一个插件的实现者再去针对这个插件开发前端UI。
### 计划
基于以上这些思考，我计划参考presto中对Java SPI的使用，在DS中也使用Java SPI来实现Plugin的设计。
#### 服务端
我计划增加一个dolphinscheduler-spi模块，所有可以被实现的插件的接口都放到spi模块中，dolphinscheduler-server等核心模块只基于spi中的接口来编写代码。所有插件的具体实现都放到一个dolphinscheduler-plugin模块中(比如告警插件都放到dolphinscheduler-alert-plugin中，Job插件都放到dolphinscheduler-job-plugin中) 每个插件都在dolphinscheduler-alert-plugin或者dolphinscheduler-job-plugin模块下再创建自己的模块，这些插件模块需要依赖dolphinscheduler-spi，并实现dolphinscheduler-spi中提供的接口。
#### 前端
由插件实现方通过一个接口来告诉前端这个插件需要使用时需要配置的参数。比如邮件告警插件要告诉前端页面需要去填写邮件接收人，抄送人；钉钉告警需要由前端填写目标群；对于Job插件来说，MR插件需要需要前端配置参数jar包路径，mr参数等等。关于这一块我是这么想的，我们在插件里设计一个所有前端支持的控件类型enum，然后每个插件可以通过一个getParams接口方法返回自己需要前端展示的参数，每个参数需要提供前端控件类型，比如input, radio , selector ,area等等；在系统启动加载插件时，系统会将每个插件对应的params生成json保存到数据库中，待前端使用。每个任务类型的参数都可以分为通用参数和自定义参数，通用参数包括（节点名称、运行标志、描述、优先级等等每个任务类型通用的参数），通过参数可以由前端写死，每个Job Type都一样。自定义参数是从数据库中取这个参数的params json，然后根据json中定义的每个参数的信息在前端生成UI元素（input,radio,selector...）。**但这里有一个问题，这种方式无法支持那种需要多级联动的form元素，因为想通过json来描述js的事件有比较大的困难，大家帮忙想想办法**。
### 我期望通过DS Plugin的设计能够带来以下好处：
1. 将核心的业务逻辑与具体的实现解耦（比如告警逻辑与具体的告警实现方式解耦；DAG中任务的执行与具体的任务实现解耦）可以提升DS架构的扩展性。
2. 对于插件的开发者来说，不需要关注DS上层的核心逻辑是怎么运行的，他们只需要关注SPI层提供给他们的接口，然后实现这些接口即可。这将显著的减少新的任务类型的开发上手难度。
3. 插件的实现层只能使用SPI中开放出来的接口和类，除此之外插件实现层尽量不要依赖DS中其它模块的代码。如果插件层想使用dolphinscheduler-common模块中的一个工具类，那正确的做法应该是先将该工具类从dolphinscheduler-common模块中迁移到dolphinscheduler-spi中。或者将工具类统一放到一个单独的模块中，不要和上层的可能被经常修改的核心业务逻辑代码放到一起，保证这个可以被其它插件依赖的模块的功能边界尽量的小，目前我只是将一些utils放到了dolphinscheduler-spi中，可以看到即使这样spi模块的pom文件中也需要添加一些utils中必须用到的依赖，比如slf4j,commons-xx,poi等等。为了保证依赖的问题能够在编译阶段时被发现，而不是等于运行时再发现，我会通过一个maven插件对SPI模块中的依赖做检查，要求所有SPI中使用的第三方依赖（包括传递依赖）都必须明确定义并指明版本，而且scope不允许是provided级以下的，这会导致一但大量的工具类放到spi中，spi中的依赖会比较多，但应该也不是啥大问题，关于这一点大家帮忙发表下看法。最终的目标是每个不同的插件的实现之间的代码不会相互影响，比如MR作业的实现不会影响到Spark作业的代码。这对于代码的review人员来说，也可以很大程度上提高review的效率。
4. 只要SPI层的代码不被修改，理论上dolphinscheduler-server层的修改和插件层的代码修改互不影响，大家基于同一套SPI的接口来开发。
5. 对于DS的使用者来说，他们可以选择自己需要的插件，特殊情况下他们只需要编译自己需要的插件的代码即可将新的插件集成到他们已有的环境中。
## 进展
目前我计划先从Alert插件化开始，做一个尝试。因为Alert模块比较简单，适合做一个快速的改造。
1. 我完成了一个自定义maven插件，它的作用是可以自动检查spi和插件相关的依赖，并且可以支持在pom文件中添加<packaging>dolphinscheduler-plugin</packaging>标识从而让DS在编译时自动生成相关的META-INF/services文件。详细的信息可以参考 ： https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin
2. https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1
这个pr是我在dev分支上进行的一部分代码的开发，目前已经实现了SPI和AlertServer的插件化改造。
3. 在插件实现层，已经完成了邮件告警插件的改造。
4. 在dolphinscheduler-alert模块中完成了插件的配置、加载功能的开发。目前插件支持开发时通过alert.properties中配置alert.plugin.binding=xxx/pom.xml来指定加载插件和在服务器上部署运行时通过在alert.properties中配置alert.plugin.dir=/xxx/xxx/目录来加载插件。**当然，alert.plugin.dir参数需要DS在打包部署时将所有插件都放到alert.plugin.dir指定的目录下，这个功能我还没有做。大家觉得插件目录放到哪合适，我建议放到${DS_INSTALL_DIR}lib/plugin目录下**。
代码中有很多的注释，主要的目的是希望通过注释让大家了更容易了解我的想法。
代码中有一些//TODO 的注释，这些是需要进一步改造完成的需求点，目前我还没有想好该如何实现。
## 目前的问题
1. 插件会通过PluginFactory中的getParams方法返回自己需要前端展示的参数，比如邮件告警需要前端填写收件人，抄送人。正常情况下前端是不知道有哪些告警插件可以用的，那前端就需要从一个地方读取系统当前已经配置了的可用的告警插件列表，然后将插件的名称（比如’邮件‘）、每个插件需要填写的参数展示在前端。那这些信息应该保存在哪？要设计一张mysql表用来保存这些信息吗？
2. 如果在A时刻，系统里有邮件告警插件，钉钉告警插件，用户在前端界面上启用了这两个插件，并且已经填写了邮件接收人，钉钉的接收人等等信息。时间来到B时刻，用户将邮件告警插件从系统中删除了，这个时候用户在邮件告警上填写的接收人信息等该保留吗？怎么保留？如果不保留，用户再次添加邮件告警插件的jar到系统中，那用户是需要重启填写邮件接收人信息吗？
以上的问题是目前告警插件化的一些遗留的问题，希望大家能一起出出主意。
当然，我一个人的力量是无法完成整个插件化的改造的，我希望得到大家的支持和帮助，给我提提建议，或者完成一些子任务，谢谢大家。
## 任务列表(暂不开放，需要等上面的问题讨论出结果之后再决定任务列表)
目前还有如下任务是需要完成的。
1. 设计数据库插件表，在插件加载时会需要把插件的ID，前端展示名（中文名/英文名），插件需要在前端展示的参数列表保存在数据库。这样前端页面才能动态的知道系统有哪些告警插件是可以使用的，每个插件又需要填写什么参数，每个参数是什么类型的表单元素。
2. 我只简单的完成了邮件告警的改造，还需要微信告警插件改造。
3. 钉钉告警插件改造。
4. 企业微信告警的改造。
5. 打包时需要把插件放到特殊的插件目录中。
6. ...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hgaol edited a comment on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hgaol edited a comment on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637567909
Also, here is the model what I designed before, and decided to do more work on it. But recently, I have other things to do. Just FYI, hope it can help to your design.
```json
[
{
"id": "email",
"cn": {...},
"en": {
"name": "Email",
"fields": [
{
"name": "receiver",
"type": "input",
"info": "who will receive this mail",
"placeholder": "xx@xx.com; bb@xx.com"
},
{
"name": "CC",
"type": "select",
"info": "copy",
"options": [
{
"id": "op1",
"label": "option1"
},
{
"id": "op2",
"label": "option2"
}
]
},
{
"name": "text",
"type": "textarea",
"info": "input some text",
"placeholder": "input some text"
}
]
}
}
]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hgaol commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hgaol commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637564747
BTW, I've created a PR about how to develop an alert plugin #2593 . Should I close this PR...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaojun2048 edited a comment on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
gaojun2048 edited a comment on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637551538
> Good idea, in fact, I have been thinking about this problem recently. In the form of SPI, users can expand themselves. Maybe later I can assist you to complete it together
That's great . When opening the to do list, I will @ you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hgaol edited a comment on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hgaol edited a comment on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637563189
Cool, that's what I want to do. As you said, it can't solve the nested fields problems. I think it's fine for alert plugin, as for job plugin, I'm not sure if it can meet the demands.
For the current problems, here is my opinions:
1. Do we need a table to save the plugin instance information?
I think the answer is 'yes'. IMO, we need 2 new tables, one is plugin_definition, the other is plugin_instance. For plugin_definition, it has id, ch_json, en_json fields. For plugin_instance, it save the values based on the definition.
2. As mentioned above, we should save the plugin instance information. If the plugin has been removed, it should alert through log, mail, or something else. And user do not need to file the form again.
This is my opinion :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] CalvinKirs commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637533726
Good idea, in fact, I have been thinking about this problem recently. In the form of SPI, users can expand themselves. Maybe later I can assist you to complete it together
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hgaol edited a comment on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hgaol edited a comment on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637563189
Cool, that's what I want to do. As you said, it can't solve the nested fields problems. I think it's fine for alert plugin, as for job plugin, I'm not sure if it will meet the demands.
For the current problems, here is my opinions:
1. Do we need a table to save the plugin instance information?
I think the answer is 'yes'. IMO, we need 2 new tables, one is plugin_definition, the other is plugin_instance. For plugin_definition, it has id, ch_json, en_json fields. For plugin_instance, it save the values based on the definition.
2. As mentioned above, we should save the plugin instance information. If the plugin has been removed, it should alert through log, mail, or something else. And user do not need to file the form again.
This is my opinion :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] felix-thinkingdata commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-650924053
very six
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hgaol edited a comment on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hgaol edited a comment on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637567909
Also, here is the model what I designed before, and decided to do more work on it. But recently, I have other things to do. Just FYI, hope it can help to your design.
```json
[
{
"id": "email",
"cn": {...},
"en": {
"name": "Email",
"fields": [
{
"name": "receiver",
"type": "input",
"info": "who will receive this mail",
"placeholder": "xx@xx.com; bb@xx.com"
},
{
"name": "some select field",
"type": "select",
"info": "copy",
"options": [
{
"id": "op1",
"label": "option1"
},
{
"id": "op2",
"label": "option2"
}
]
},
{
"name": "text",
"type": "textarea",
"info": "input some text",
"placeholder": "input some text"
}
]
}
}
]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] zhangchunyang1024 commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
zhangchunyang1024 commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-641844360
Very Cool
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaojun2048 commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
gaojun2048 commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637574537
> dear all,
> the import discussion should happened on the mailing list, we can disscuss this topic in mail: https://lists.apache.org/thread.html/r4bbafef1b3b5943c7ba04bbe3a71b69520a5063ffaf90d6093520647%40%3Cdev.dolphinscheduler.apache.org%3E
ok!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hanfengcan commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hanfengcan commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637541068
关于前端问题，ncform和form-create两个库可以参考，都是基于json产生表单，前端只需要几个组件，然后按插件加载json。由于项目前端没有用到iview，ant，或者element，所以没办法直接使用。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaojun2048 commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
gaojun2048 commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-799067163
> That's cool, dubbo's design is using a lot of Plugin Design(also called spi). User can customize function. I want to join the big
> feature's developping work，can you show the Task list? @gaojun2048
Hi, hermeshephaestus
We have completed the plug-in of the alert module . You can get the information from this issue : [Feature] Alert Plugin Design #3049 .
Register center plugin-in and task plugin-in is doing , You can see it at this issue :
https://github.com/apache/incubator-dolphinscheduler/issues/3961.
https://github.com/apache/incubator-dolphinscheduler/issues/4878.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaojun2048 commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
gaojun2048 commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637551538
> Good idea, in fact, I have been thinking about this problem recently. In the form of SPI, users can expand themselves. Maybe later I can assist you to complete it together
That's great . When opening the todo list, I will @ you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hgaol commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hgaol commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637567909
Also, here is the model what I design before, and decided to do more work on it. But recently, I have other things to do. Just FYI, hope it can help to your design.
```json
[
{
"id": "email",
"cn": {...},
"en": {
"name": "Email",
"fields": [
{
"name": "receiver",
"type": "input",
"info": "who will receive this mail",
"placeholder": "xx@xx.com; bb@xx.com"
},
{
"name": "CC",
"type": "select",
"info": "copy",
"options": [
{
"id": "op1",
"label": "option1"
},
{
"id": "op2",
"label": "option2"
}
]
},
{
"name": "text",
"type": "textarea",
"info": "input some text",
"placeholder": "input some text"
}
]
}
}
]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaojun2048 closed issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
gaojun2048 closed issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaojun2048 commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
gaojun2048 commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637919969
> Also, here is the model what I designed before, and decided to do more work on it. But recently, I have other things to do. Just FYI, hope it can help to your design.
>
> ```json
> [
> {
> "id": "email",
> "cn": {...},
> "en": {
> "name": "Email",
> "fields": [
> {
> "name": "receiver",
> "type": "input",
> "info": "who will receive this mail",
> "placeholder": "xx@xx.com; bb@xx.com"
> },
> {
> "name": "some select field",
> "type": "select",
> "info": "copy",
> "options": [
> {
> "id": "op1",
> "label": "option1"
> },
> {
> "id": "op2",
> "label": "option2"
> }
> ]
> },
> {
> "name": "text",
> "type": "textarea",
> "info": "input some text",
> "placeholder": "input some text"
> }
> ]
> }
> }
> ]
> ```
Can we discuss this topic in mail list?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] samz406 commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
samz406 commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-648176293
awesome
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hermeshephaestus commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hermeshephaestus commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-703129572
That's cool, dubbo's design is using a lot of Plugin Design(also called spi). User can customize function. I want to join the big
feature's developping work，can you show the Task list? @gaojun2048
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hgaol edited a comment on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hgaol edited a comment on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637563189
Cool, that's what I want to do. As you said, it can't solve the nested fields problems. I think it's fine for alert plugin, as for job plugin, I'm not sure if it can meet the demands.
For the current problems, here is my opinions:
1. Do we need a table to save the plugin instance information?
I think the answer is 'yes'. IMO, we need 2 new tables, one is plugin_definition, the other is plugin_instance. For plugin_definition, it has id, ch_json, en_json, type(alert/job/...) fields. For plugin_instance, it save the values based on the definition.
2. As mentioned above, we should save the plugin instance information. If the plugin has been removed, it should alert through log, mail, or something else. And user do not need to file the form again.
This is my opinion :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637573905
dear all,
the import discussion should happened on the mailing list, we can disscuss this topic in mail: https://lists.apache.org/thread.html/r4bbafef1b3b5943c7ba04bbe3a71b69520a5063ffaf90d6093520647%40%3Cdev.dolphinscheduler.apache.org%3E
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaojun2048 commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
gaojun2048 commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637552632
> 关于前端问题，ncform和form-create两个库可以参考，都是基于json产生表单，前端只需要几个组件，然后按插件加载json。由于项目前端没有用到iview，ant，或者element，所以没办法直接使用。
Thanks for your participation, if so, our front end may have to do more work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hgaol edited a comment on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hgaol edited a comment on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637564747
BTW, I've created a PR about how to develop an alert plugin #2593 . Should I close that PR...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hgaol commented on issue #2869: [Proposal] DolphinScheduler Plugin Design
Posted by GitBox <gi...@apache.org>.
hgaol commented on issue #2869:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2869#issuecomment-637563189
Cool, that's what I want to do. As you said, it can't solve the nested fields problems. I think it's fine for alert plugin, as for job plugin, I'm not sure it will meet the demands.
For the current problems, here is my opinions:
1. Do we need a table to save the plugin instance information?
I think the answer is 'yes'. IMO, we need 2 new tables, one is plugin_definition, the other is plugin_instance. For plugin_definition, it has id, ch_json, en_json fields. For plugin_instance, it save the values based on the definition.
2. As mentioned above, we should save the plugin instance information. If the plugin has been removed, it should alert through log, mail, or something else. And user do not need to file the form again.
This is my opinion :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org