You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by JUN GAO <ga...@gmail.com> on 2020/06/02 07:04:06 UTC
[Poprosal] DolphinScheduler Plugin
Hello everyone
The subject of this email is to initiate a discussion about DS Plugin . DS
Plugin related architecture adjustment is very important, so I will use
Chinese and English to describe
The issue ID :
https://github.com/apache/incubator-dolphinscheduler/issues/2869
Full details:
（中文请往最后看）
aims
The core purpose of DS Plugin is to decouple the upper core logic of DS
from the specific implementation. I have been involved in the second
development of presto recently. I have learned a lot of excellent design
concepts about plug-in from the presto architecture. These design concepts
have helped presto become a very easy to expand project. Now if you want to
develop a presto-based one Your own query plugin is very simple.
We hope that DS can also have such excellent expansion capabilities,
because each job type in DS can be a plug-in, which can be implemented by
different people. Therefore, we hope that the development of DS Job Type
will be simpler, so that in the future, more contributors will be attracted
to participate. Now if I want to develop a new job is very difficult, I
need to look at a lot of code, have a deep understanding of the source code
of DS to know where to start. This is very unfriendly to new developers who
want to participate in the DS community.
Here are some thoughts on DS's current design:
1.
The alarm collection and sending logic in the alert service is decoupled
from the specific alarm methods (email, WeChat, pegs, etc.).
Dolphinscheduler-alert should not focus on how to send the alarm. Sending
email or WeChat should be specific The alarm implementation plugin works.
2.
For workflow tasks, dolphinscheduler-server should only need to pay
attention to the execution of DAG tasks, and each task should have a
unified standard interface for dolphinscheduler-server to call. For
example, MR task, when presto-worker executes MR task, only need to know
the interface and corresponding implementation class that the task
implements. The specific implementation of the MR task should be put into
the plugin to be developed by the developer of this task.
3.
For the front end, the front end needs to cooperate with the plug-in
design. Now the front end is hard-coded, such as MR tasks and spark tasks.
The front-end configurable parameters of each task node are written by the
front-end personnel, which is not Conducive to plug-in. Because it is
difficult for us to ask the implementer of a plugin to develop the
front-end UI for this plugin.
Plan
Based on the above considerations, I plan to refer to the use of Java SPI
in presto, and also use Java SPI in DS to implement Plugin design.
Server
I plan to add a dolphinscheduler-spi module. All plug-in interfaces that
can be implemented are put into the spi module. Core modules such as
dolphinscheduler-server only write code based on the interfaces in the spi.
The specific implementation of all plugins are put into a
dolphinscheduler-plugin module (for example, alarm plugins are put into
dolphinscheduler-alert-plugin, Job plugins are put into
dolphinscheduler-job-plugin) Each plugin is in
dolphinscheduler-alert-plugin Or create your own modules under the
dolphinscheduler-job-plugin module. These plug-in modules need to rely on
dolphinscheduler-spi and implement the interface provided in
dolphinscheduler-spi.
front end
The plug-in implementer tells the front end through a port the parameters
that need to be configured when the plug-in needs to be used. For example,
the email alert plugin needs to tell the front-end page to fill in the
email recipient and the cc; the nail alert needs to be filled in by the
front end for the target group; for the Job plugin, the MR plugin needs the
front-end configuration parameter jar package path, mr parameter, etc. . I
think so about this piece. We design an enum in the plug-in that supports
all the front-end control types. Then each plug-in can return the
parameters that it needs to display on the front-end through a getParams
interface method. Such as input, radio, selector, area, etc.; when the
system starts to load the plug-in, the system will save the json generated
by the params corresponding to each plug-in to the database for use by the
front end. The parameters of each task type can be divided into general
parameters and custom parameters. The general parameters include (node
name, running flag, description, priority, etc., common parameters for each
task type), and the parameters can be written by the front end. Every Job
Type is the same. The custom parameter is a params json that takes this
parameter from the database, and then generates UI elements (input, radio,
selector...) in the front end according to the information of each
parameter defined in json. But there is a problem here. This method cannot
support the form elements that require multi-level linkage, because it is
more difficult to describe js events through json. Please help us find a way
.
I hope that the design of DS Plugin can bring the following benefits:
1. Decoupling the core business logic from the specific implementation
(such as decoupling the alarm logic from the specific alarm implementation
method; decoupling the task execution from the specific task implementation
in the DAG) can improve the scalability of the DS architecture.
2. For plug-in developers, there is no need to pay attention to how the
core logic of the DS upper layer works. They only need to pay attention to
the interfaces provided to them by the SPI layer, and then implement these
interfaces. This will significantly reduce the difficulty of getting
started with new task types.
3. The implementation layer of the plug-in can only use the interfaces
and classes open in the SPI. In addition, the plug-in implementation layer
should not rely on the code of other modules in the DS. If the plug-in
layer wants to use a tool class in the dolphinscheduler-common module, the
correct approach should be to first migrate the tool class from the
dolphinscheduler-common module to the dolphinscheduler-spi. Or put the tool
classes into a single module, do not put it together with the upper layer
of the core business logic code that may be frequently modified, to ensure
that the functional boundary of the module that can be dependent on other
plugins is as small as possible. At present, I just put Some utils are put
into the dolphinscheduler-spi. It can be seen that even the spi module's
pom file needs to add some dependencies that must be used in utils, such as
slf4j, commons-xx, poi, etc. In order to ensure that the problem of
dependencies can be discovered during the compilation phase, rather than
being found at runtime, I will check the dependencies in the SPI module
through a maven plug-in, and require all third-party dependencies
(including transitive dependencies) used in SPI All must be clearly defined
and specified, and the scope is not allowed to be below the provided level,
which will result in a large number of tool classes placed in the spi,
there will be more dependencies in the spi, but it should not be a big
problem. A little bit of help everyone express their opinions. The ultimate
goal is that the code between the implementation of each different plug-in
will not affect each other. For example, the implementation of MR jobs will
not affect the code of Spark jobs. For reviewers of the code, it can also
greatly improve the efficiency of the review.
4. As long as the code of the SPI layer is not modified, in theory, the
modification of the dolphinscheduler-server layer and the code modification
of the plug-in layer do not affect each other, and everyone develops based
on the same set of SPI interfaces.
5. For users of DS, they can choose the plug-in they need. In special
cases, they only need to compile the code of the plug-in they need to
integrate the new plug-in into their existing environment.
Progress
At present, I plan to start with the plug-in of Alert and make a try.
Because the Alert module is relatively simple, it is suitable for a rapid
transformation.
1.
I have completed a custom maven plug-in, its role is to automatically
check the spi and plug-in related dependencies, and can support the
addition of dolphinscheduler-plugin logo in the pom file to allow DS to
compile Automatically generate related META-INF/services files. For
detailed information, please refer to:
https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin
2.
gaojun2048#1
<https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
This pr is part of the code development that I carried out on the dev
branch. At present, the plug-in transformation of SPI and AlertServer has
been implemented.
3.
At the plug-in implementation layer, the transformation of the mail
alert plug-in has been completed.
4.
In the dolphinscheduler-alert module, the development of plug-in
configuration and loading function is completed. At present, the plug-in
supports the configuration of alert.plugin.binding=xxx/pom.xml in
alert.properties to specify the loading of the plug-in and the deployment
and runtime on the server by configuring alert.plugin.dir=/xxx/xxx /
Directory to load plugins. Of course, the alert.plugin.dir parameter
requires DS to place all plugins in the directory specified by
alert.plugin.dir during package deployment. I have not done this function
yet. Everyone thinks where to put the plugin directory, I suggest putting
it in the ${DS_INSTALL_DIR}lib/plugin directory.
There are many comments in the code, the main purpose is to make it easier
for everyone to understand my ideas through comments.
There are some //TODO comments in the code. These are the needs that need
to be further modified. At present, I have not figured out how to implement
them.
Current issues
1.
The plug-in will return the parameters that it needs to be displayed by
the front end through the getParams method in PluginFactory. For example,
the email alert needs the front end to fill in the recipient and copy the
person. Under normal circumstances, the front end does not know which alarm
plug-ins are available, then the front end needs to read the list of
available alarm plug-ins currently configured by the system from a place,
and then the name of the plug-in (such as'mail'), each The parameters that
the plug-in needs to fill are displayed on the front end. Where should this
information be kept? Want to design a mysql table to save this information?
2.
If at time A, there are email alert plugins and nail alert plugins in
the system, the user has enabled these two plugins on the front-end
interface, and has filled in the information of email recipients, nail
recipients, and so on. When the time comes to time B, the user deletes the
email alert plug-in from the system. Should the user fill in the recipient
information on the email alert at this time? How to keep it? If it is not
retained, the user adds the jar of the email alert plugin to the system
again, does the user need to restart to fill in the email recipient
information?
The above problems are some of the problems left by the current alarm
plug-in. I hope everyone can come up with ideas together.
Of course, my strength alone cannot complete the transformation of the
entire plug-in. I hope to get your support and help, give me suggestions,
or complete some sub-tasks, thank you.
Task list (not open for the time being, you need to wait for the above
question to discuss the result before deciding on the task list)
At present, the following tasks need to be completed.
1.
Design the database plug-in table. When the plug-in is loaded, the ID of
the plug-in and the front-end display name (Chinese name/English name) are
required. The parameter list that the plug-in needs to display on the
front-end is saved in the database. In this way, the front-end page can
dynamically know which alarm plug-ins are available in the system, what
parameters each plug-in needs to fill in, and what type of form elements
each parameter is.
2.
I just completed the transformation of the email alarm, and we also need
to modify the WeChat alarm plugin.
3.
Nail alarm plug-in modification.
4.
Transformation of enterprise WeChat alarm.
5.
When packaging, you need to put the plugin in a special plugin directory.
6.
...
------------------------------
大家好
这个邮件主题是发起一个关于DS Plugin 化的讨论。DS Plugin相关的架构调整非常重要，所以我将用中文和英文两种语言来进行描述。
目标
DS
Plugin的核心目的是将DS的上层核心逻辑与具体的实现解耦。我最近一直在参与presto的二次开发工作，从presto的架构中学习到了很多关于插件化的优秀的设计理念，这些设计理念帮助presto成为一个非常容易扩展的项目，现在如果你想基于presto开发一个自己的查询插件是非常简单的事情。
我们希望DS也能有这样优秀的扩展能力，因为在DS中每一个Job Type都可以是一个插件，都可以被不同的人来实现。因此我们希望DS Job
Type的开发变得再简单一点，这样在未来也可吸引更多的贡献者参与进来。现在如果我想开发一个新的Job是非常困难的，我需要看大量的代码，对DS的源码有很深入的了解才能知道该从何处下手。这对那些想参与到DS社区的新的开发者来说是非常不友好的。
下面是我对DS目前的设计的一些思考：
1.
alert服务中的告警收集与发送逻辑与具体的告警方式（邮件、微信、钉钉等）进行解耦，dolphinscheduler-alert中不应该去关注告警该怎么发，发邮件或者发微信应该是具体的告警实现插件工作。
2.
对于工作流任务来说，dolphinscheduler-server应该只需要关注DAG任务的执行，每个任务都应该有统一的标准的接口来让dolphinscheduler-server调用。比如MR任务，presto-worker执行MR任务时，只需要知道任务实现的接口和对应的实现类。MR任务的具体实现应该放到插件里由这个任务的开发者来开发。
3.
对于前端来说，前端需要配合插件化的设计，现在前端是写死的，比如MR任务，spark任务这些，每个任务节点的前端可配置参数都是由前端人员写死的，这不利于插件化。因为我们很难要求一个插件的实现者再去针对这个插件开发前端UI。
计划
基于以上这些思考，我计划参考presto中对Java SPI的使用，在DS中也使用Java SPI来实现Plugin的设计。
服务端
我计划增加一个dolphinscheduler-spi模块，所有可以被实现的插件的接口都放到spi模块中，dolphinscheduler-server等核心模块只基于spi中的接口来编写代码。所有插件的具体实现都放到一个dolphinscheduler-plugin模块中(比如告警插件都放到dolphinscheduler-alert-plugin中，Job插件都放到dolphinscheduler-job-plugin中)
每个插件都在dolphinscheduler-alert-plugin或者dolphinscheduler-job-plugin模块下再创建自己的模块，这些插件模块需要依赖dolphinscheduler-spi，并实现dolphinscheduler-spi中提供的接口。
前端
由插件实现方通过一个接口来告诉前端这个插件需要使用时需要配置的参数。比如邮件告警插件要告诉前端页面需要去填写邮件接收人，抄送人；钉钉告警需要由前端填写目标群；对于Job插件来说，MR插件需要需要前端配置参数jar包路径，mr参数等等。关于这一块我是这么想的，我们在插件里设计一个所有前端支持的控件类型enum，然后每个插件可以通过一个getParams接口方法返回自己需要前端展示的参数，每个参数需要提供前端控件类型，比如input,
radio , selector
,area等等；在系统启动加载插件时，系统会将每个插件对应的params生成json保存到数据库中，待前端使用。每个任务类型的参数都可以分为通用参数和自定义参数，通用参数包括（节点名称、运行标志、描述、优先级等等每个任务类型通用的参数），通过参数可以由前端写死，每个Job
Type都一样。自定义参数是从数据库中取这个参数的params
json，然后根据json中定义的每个参数的信息在前端生成UI元素（input,radio,selector...）。
但这里有一个问题，这种方式无法支持那种需要多级联动的form元素，因为想通过json来描述js的事件有比较大的困难，大家帮忙想想办法。
我期望通过DS Plugin的设计能够带来以下好处：
1.
将核心的业务逻辑与具体的实现解耦（比如告警逻辑与具体的告警实现方式解耦；DAG中任务的执行与具体的任务实现解耦）可以提升DS架构的扩展性。
2.
对于插件的开发者来说，不需要关注DS上层的核心逻辑是怎么运行的，他们只需要关注SPI层提供给他们的接口，然后实现这些接口即可。这将显著的减少新的任务类型的开发上手难度。
3.
插件的实现层只能使用SPI中开放出来的接口和类，除此之外插件实现层尽量不要依赖DS中其它模块的代码。如果插件层想使用dolphinscheduler-common模块中的一个工具类，那正确的做法应该是先将该工具类从dolphinscheduler-common模块中迁移到dolphinscheduler-spi中。或者将工具类统一放到一个单独的模块中，不要和上层的可能被经常修改的核心业务逻辑代码放到一起，保证这个可以被其它插件依赖的模块的功能边界尽量的小，目前我只是将一些utils放到了dolphinscheduler-spi中，可以看到即使这样spi模块的pom文件中也需要添加一些utils中必须用到的依赖，比如slf4j,commons-xx,poi等等。为了保证依赖的问题能够在编译阶段时被发现，而不是等于运行时再发现，我会通过一个maven插件对SPI模块中的依赖做检查，要求所有SPI中使用的第三方依赖（包括传递依赖）都必须明确定义并指明版本，而且scope不允许是provided级以下的，这会导致一但大量的工具类放到spi中，spi中的依赖会比较多，但应该也不是啥大问题，关于这一点大家帮忙发表下看法。最终的目标是每个不同的插件的实现之间的代码不会相互影响，比如MR作业的实现不会影响到Spark作业的代码。这对于代码的review人员来说，也可以很大程度上提高review的效率。
4.
只要SPI层的代码不被修改，理论上dolphinscheduler-server层的修改和插件层的代码修改互不影响，大家基于同一套SPI的接口来开发。
5.
对于DS的使用者来说，他们可以选择自己需要的插件，特殊情况下他们只需要编译自己需要的插件的代码即可将新的插件集成到他们已有的环境中。
进展
目前我计划先从Alert插件化开始，做一个尝试。因为Alert模块比较简单，适合做一个快速的改造。
1.
我完成了一个自定义maven插件，它的作用是可以自动检查spi和插件相关的依赖，并且可以支持在pom文件中添加dolphinscheduler-plugin标识从而让DS在编译时自动生成相关的META-INF/services文件。详细的信息可以参考
： https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin
2.
gaojun2048#1
<https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
这个pr是我在dev分支上进行的一部分代码的开发，目前已经实现了SPI和AlertServer的插件化改造。
3.
在插件实现层，已经完成了邮件告警插件的改造。
4.
在dolphinscheduler-alert模块中完成了插件的配置、加载功能的开发。目前插件支持开发时通过alert.properties中配置alert.plugin.binding=xxx/pom.xml来指定加载插件和在服务器上部署运行时通过在alert.properties中配置alert.plugin.dir=/xxx/xxx/目录来加载插件。
当然，alert.plugin.dir参数需要DS在打包部署时将所有插件都放到alert.plugin.dir指定的目录下，这个功能我还没有做。大家觉得插件目录放到哪合适，我建议放到${DS_INSTALL_DIR}lib/plugin目录下
。
代码中有很多的注释，主要的目的是希望通过注释让大家了更容易了解我的想法。
代码中有一些//TODO 的注释，这些是需要进一步改造完成的需求点，目前我还没有想好该如何实现。
目前的问题
1.
插件会通过PluginFactory中的getParams方法返回自己需要前端展示的参数，比如邮件告警需要前端填写收件人，抄送人。正常情况下前端是不知道有哪些告警插件可以用的，那前端就需要从一个地方读取系统当前已经配置了的可用的告警插件列表，然后将插件的名称（比如’邮件‘）、每个插件需要填写的参数展示在前端。那这些信息应该保存在哪？要设计一张mysql表用来保存这些信息吗？
2.
如果在A时刻，系统里有邮件告警插件，钉钉告警插件，用户在前端界面上启用了这两个插件，并且已经填写了邮件接收人，钉钉的接收人等等信息。时间来到B时刻，用户将邮件告警插件从系统中删除了，这个时候用户在邮件告警上填写的接收人信息等该保留吗？怎么保留？如果不保留，用户再次添加邮件告警插件的jar到系统中，那用户是需要重启填写邮件接收人信息吗？
以上的问题是目前告警插件化的一些遗留的问题，希望大家能一起出出主意。
当然，我一个人的力量是无法完成整个插件化的改造的，我希望得到大家的支持和帮助，给我提提建议，或者完成一些子任务，谢谢大家。
任务列表(暂不开放，需要等上面的问题讨论出结果之后再决定任务列表)
目前还有如下任务是需要完成的。
1.
设计数据库插件表，在插件加载时会需要把插件的ID，前端展示名（中文名/英文名），插件需要在前端展示的参数列表保存在数据库。这样前端页面才能动态的知道系统有哪些告警插件是可以使用的，每个插件又需要填写什么参数，每个参数是什么类型的表单元素。
2.
我只简单的完成了邮件告警的改造，还需要微信告警插件改造。
3.
钉钉告警插件改造。
4.
企业微信告警的改造。
5.
打包时需要把插件放到特殊的插件目录中。
6.
...
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: [Poprosal] DolphinScheduler Plugin
Posted by JUN GAO <ga...@gmail.com>.
@dhangao@hotmail.com
Also, here is the model what I designed before, and decided to do more work
on it. But recently, I have other things to do. Just FYI, hope it can help
to your design.
[
{
"id": "email",
"cn": {...},
"en": {
"name": "Email",
"fields": [
{
"name": "receiver",
"type": "input",
"info": "who will receive this mail",
"placeholder": "xx@xx.com; bb@xx.com"
},
{
"name": "some select field",
"type": "select",
"info": "copy",
"options": [
{
"id": "op1",
"label": "option1"
},
{
"id": "op2",
"label": "option2"
}
]
},
{
"name": "text",
"type": "textarea",
"info": "input some text",
"placeholder": "input some text"
}
]
}
}
]
Can we discuss this topic in mail list?
---------------------------------------------------------------------------------------------------------------------------------------------------------
In the early days, I thought about returning the json string of the
parameter directly through the interface at the plugin layer. However,
during our use, we found that some plug-in developers added the parameter
type that the front-end ui does not support in json. So I modified the
design of this part. The plugin layer can only use the params class defined
in the spi. When the plugin is loaded in the AlertServer layer, the params
are converted into json and saved in the database.
JUN GAO <ga...@gmail.com> 于2020年6月2日周二 下午3:04写道：
> Hello everyone
> The subject of this email is to initiate a discussion about DS Plugin . DS
> Plugin related architecture adjustment is very important, so I will use
> Chinese and English to describe
>
> The issue ID :
> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>
> Full details:
>
> （中文请往最后看）
> aims
>
> The core purpose of DS Plugin is to decouple the upper core logic of DS
> from the specific implementation. I have been involved in the second
> development of presto recently. I have learned a lot of excellent design
> concepts about plug-in from the presto architecture. These design concepts
> have helped presto become a very easy to expand project. Now if you want to
> develop a presto-based one Your own query plugin is very simple.
>
> We hope that DS can also have such excellent expansion capabilities,
> because each job type in DS can be a plug-in, which can be implemented by
> different people. Therefore, we hope that the development of DS Job Type
> will be simpler, so that in the future, more contributors will be attracted
> to participate. Now if I want to develop a new job is very difficult, I
> need to look at a lot of code, have a deep understanding of the source code
> of DS to know where to start. This is very unfriendly to new developers who
> want to participate in the DS community.
> Here are some thoughts on DS's current design:
>
> 1.
>
> The alarm collection and sending logic in the alert service is
> decoupled from the specific alarm methods (email, WeChat, pegs, etc.).
> Dolphinscheduler-alert should not focus on how to send the alarm. Sending
> email or WeChat should be specific The alarm implementation plugin works.
> 2.
>
> For workflow tasks, dolphinscheduler-server should only need to pay
> attention to the execution of DAG tasks, and each task should have a
> unified standard interface for dolphinscheduler-server to call. For
> example, MR task, when presto-worker executes MR task, only need to know
> the interface and corresponding implementation class that the task
> implements. The specific implementation of the MR task should be put into
> the plugin to be developed by the developer of this task.
> 3.
>
> For the front end, the front end needs to cooperate with the plug-in
> design. Now the front end is hard-coded, such as MR tasks and spark tasks.
> The front-end configurable parameters of each task node are written by the
> front-end personnel, which is not Conducive to plug-in. Because it is
> difficult for us to ask the implementer of a plugin to develop the
> front-end UI for this plugin.
>
> Plan
>
> Based on the above considerations, I plan to refer to the use of Java SPI
> in presto, and also use Java SPI in DS to implement Plugin design.
> Server
>
> I plan to add a dolphinscheduler-spi module. All plug-in interfaces that
> can be implemented are put into the spi module. Core modules such as
> dolphinscheduler-server only write code based on the interfaces in the spi.
> The specific implementation of all plugins are put into a
> dolphinscheduler-plugin module (for example, alarm plugins are put into
> dolphinscheduler-alert-plugin, Job plugins are put into
> dolphinscheduler-job-plugin) Each plugin is in
> dolphinscheduler-alert-plugin Or create your own modules under the
> dolphinscheduler-job-plugin module. These plug-in modules need to rely on
> dolphinscheduler-spi and implement the interface provided in
> dolphinscheduler-spi.
> front end
>
> The plug-in implementer tells the front end through a port the parameters
> that need to be configured when the plug-in needs to be used. For example,
> the email alert plugin needs to tell the front-end page to fill in the
> email recipient and the cc; the nail alert needs to be filled in by the
> front end for the target group; for the Job plugin, the MR plugin needs the
> front-end configuration parameter jar package path, mr parameter, etc. . I
> think so about this piece. We design an enum in the plug-in that supports
> all the front-end control types. Then each plug-in can return the
> parameters that it needs to display on the front-end through a getParams
> interface method. Such as input, radio, selector, area, etc.; when the
> system starts to load the plug-in, the system will save the json generated
> by the params corresponding to each plug-in to the database for use by the
> front end. The parameters of each task type can be divided into general
> parameters and custom parameters. The general parameters include (node
> name, running flag, description, priority, etc., common parameters for each
> task type), and the parameters can be written by the front end. Every Job
> Type is the same. The custom parameter is a params json that takes this
> parameter from the database, and then generates UI elements (input, radio,
> selector...) in the front end according to the information of each
> parameter defined in json. But there is a problem here. This method
> cannot support the form elements that require multi-level linkage, because
> it is more difficult to describe js events through json. Please help us
> find a way.
> I hope that the design of DS Plugin can bring the following benefits:
>
> 1. Decoupling the core business logic from the specific implementation
> (such as decoupling the alarm logic from the specific alarm implementation
> method; decoupling the task execution from the specific task implementation
> in the DAG) can improve the scalability of the DS architecture.
>
> 2. For plug-in developers, there is no need to pay attention to how
> the core logic of the DS upper layer works. They only need to pay attention
> to the interfaces provided to them by the SPI layer, and then implement
> these interfaces. This will significantly reduce the difficulty of getting
> started with new task types.
>
> 3. The implementation layer of the plug-in can only use the interfaces
> and classes open in the SPI. In addition, the plug-in implementation layer
> should not rely on the code of other modules in the DS. If the plug-in
> layer wants to use a tool class in the dolphinscheduler-common module, the
> correct approach should be to first migrate the tool class from the
> dolphinscheduler-common module to the dolphinscheduler-spi. Or put the tool
> classes into a single module, do not put it together with the upper layer
> of the core business logic code that may be frequently modified, to ensure
> that the functional boundary of the module that can be dependent on other
> plugins is as small as possible. At present, I just put Some utils are put
> into the dolphinscheduler-spi. It can be seen that even the spi module's
> pom file needs to add some dependencies that must be used in utils, such as
> slf4j, commons-xx, poi, etc. In order to ensure that the problem of
> dependencies can be discovered during the compilation phase, rather than
> being found at runtime, I will check the dependencies in the SPI module
> through a maven plug-in, and require all third-party dependencies
> (including transitive dependencies) used in SPI All must be clearly defined
> and specified, and the scope is not allowed to be below the provided level,
> which will result in a large number of tool classes placed in the spi,
> there will be more dependencies in the spi, but it should not be a big
> problem. A little bit of help everyone express their opinions. The ultimate
> goal is that the code between the implementation of each different plug-in
> will not affect each other. For example, the implementation of MR jobs will
> not affect the code of Spark jobs. For reviewers of the code, it can also
> greatly improve the efficiency of the review.
>
> 4. As long as the code of the SPI layer is not modified, in theory,
> the modification of the dolphinscheduler-server layer and the code
> modification of the plug-in layer do not affect each other, and everyone
> develops based on the same set of SPI interfaces.
>
> 5. For users of DS, they can choose the plug-in they need. In special
> cases, they only need to compile the code of the plug-in they need to
> integrate the new plug-in into their existing environment.
>
> Progress
>
> At present, I plan to start with the plug-in of Alert and make a try.
> Because the Alert module is relatively simple, it is suitable for a rapid
> transformation.
>
> 1.
>
> I have completed a custom maven plug-in, its role is to automatically
> check the spi and plug-in related dependencies, and can support the
> addition of dolphinscheduler-plugin logo in the pom file to allow DS to
> compile Automatically generate related META-INF/services files. For
> detailed information, please refer to:
> https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin
> 2.
>
> gaojun2048#1
> <https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> This pr is part of the code development that I carried out on the
> dev branch. At present, the plug-in transformation of SPI and AlertServer
> has been implemented.
> 3.
>
> At the plug-in implementation layer, the transformation of the mail
> alert plug-in has been completed.
> 4.
>
> In the dolphinscheduler-alert module, the development of plug-in
> configuration and loading function is completed. At present, the plug-in
> supports the configuration of alert.plugin.binding=xxx/pom.xml in
> alert.properties to specify the loading of the plug-in and the deployment
> and runtime on the server by configuring alert.plugin.dir=/xxx/xxx /
> Directory to load plugins. Of course, the alert.plugin.dir parameter
> requires DS to place all plugins in the directory specified by
> alert.plugin.dir during package deployment. I have not done this function
> yet. Everyone thinks where to put the plugin directory, I suggest putting
> it in the ${DS_INSTALL_DIR}lib/plugin directory.
>
> There are many comments in the code, the main purpose is to make it easier
> for everyone to understand my ideas through comments.
>
> There are some //TODO comments in the code. These are the needs that need
> to be further modified. At present, I have not figured out how to implement
> them.
> Current issues
>
> 1.
>
> The plug-in will return the parameters that it needs to be displayed
> by the front end through the getParams method in PluginFactory. For
> example, the email alert needs the front end to fill in the recipient and
> copy the person. Under normal circumstances, the front end does not know
> which alarm plug-ins are available, then the front end needs to read the
> list of available alarm plug-ins currently configured by the system from a
> place, and then the name of the plug-in (such as'mail'), each The
> parameters that the plug-in needs to fill are displayed on the front end.
> Where should this information be kept? Want to design a mysql table to save
> this information?
> 2.
>
> If at time A, there are email alert plugins and nail alert plugins in
> the system, the user has enabled these two plugins on the front-end
> interface, and has filled in the information of email recipients, nail
> recipients, and so on. When the time comes to time B, the user deletes the
> email alert plug-in from the system. Should the user fill in the recipient
> information on the email alert at this time? How to keep it? If it is not
> retained, the user adds the jar of the email alert plugin to the system
> again, does the user need to restart to fill in the email recipient
> information?
>
> The above problems are some of the problems left by the current alarm
> plug-in. I hope everyone can come up with ideas together.
>
> Of course, my strength alone cannot complete the transformation of the
> entire plug-in. I hope to get your support and help, give me suggestions,
> or complete some sub-tasks, thank you.
> Task list (not open for the time being, you need to wait for the above
> question to discuss the result before deciding on the task list)
>
> At present, the following tasks need to be completed.
>
> 1.
>
> Design the database plug-in table. When the plug-in is loaded, the ID
> of the plug-in and the front-end display name (Chinese name/English name)
> are required. The parameter list that the plug-in needs to display on the
> front-end is saved in the database. In this way, the front-end page can
> dynamically know which alarm plug-ins are available in the system, what
> parameters each plug-in needs to fill in, and what type of form elements
> each parameter is.
> 2.
>
> I just completed the transformation of the email alarm, and we also
> need to modify the WeChat alarm plugin.
> 3.
>
> Nail alarm plug-in modification.
> 4.
>
> Transformation of enterprise WeChat alarm.
> 5.
>
> When packaging, you need to put the plugin in a special plugin
> directory.
>
> 6.
>
> ...
>
> ------------------------------
>
> 大家好
> 这个邮件主题是发起一个关于DS Plugin 化的讨论。DS Plugin相关的架构调整非常重要，所以我将用中文和英文两种语言来进行描述。
> 目标
>
> DS
> Plugin的核心目的是将DS的上层核心逻辑与具体的实现解耦。我最近一直在参与presto的二次开发工作，从presto的架构中学习到了很多关于插件化的优秀的设计理念，这些设计理念帮助presto成为一个非常容易扩展的项目，现在如果你想基于presto开发一个自己的查询插件是非常简单的事情。
>
> 我们希望DS也能有这样优秀的扩展能力，因为在DS中每一个Job Type都可以是一个插件，都可以被不同的人来实现。因此我们希望DS Job
> Type的开发变得再简单一点，这样在未来也可吸引更多的贡献者参与进来。现在如果我想开发一个新的Job是非常困难的，我需要看大量的代码，对DS的源码有很深入的了解才能知道该从何处下手。这对那些想参与到DS社区的新的开发者来说是非常不友好的。
> 下面是我对DS目前的设计的一些思考：
>
> 1.
>
>
> alert服务中的告警收集与发送逻辑与具体的告警方式（邮件、微信、钉钉等）进行解耦，dolphinscheduler-alert中不应该去关注告警该怎么发，发邮件或者发微信应该是具体的告警实现插件工作。
> 2.
>
>
> 对于工作流任务来说，dolphinscheduler-server应该只需要关注DAG任务的执行，每个任务都应该有统一的标准的接口来让dolphinscheduler-server调用。比如MR任务，presto-worker执行MR任务时，只需要知道任务实现的接口和对应的实现类。MR任务的具体实现应该放到插件里由这个任务的开发者来开发。
> 3.
>
>
> 对于前端来说，前端需要配合插件化的设计，现在前端是写死的，比如MR任务，spark任务这些，每个任务节点的前端可配置参数都是由前端人员写死的，这不利于插件化。因为我们很难要求一个插件的实现者再去针对这个插件开发前端UI。
>
> 计划
>
> 基于以上这些思考，我计划参考presto中对Java SPI的使用，在DS中也使用Java SPI来实现Plugin的设计。
> 服务端
>
> 我计划增加一个dolphinscheduler-spi模块，所有可以被实现的插件的接口都放到spi模块中，dolphinscheduler-server等核心模块只基于spi中的接口来编写代码。所有插件的具体实现都放到一个dolphinscheduler-plugin模块中(比如告警插件都放到dolphinscheduler-alert-plugin中，Job插件都放到dolphinscheduler-job-plugin中)
> 每个插件都在dolphinscheduler-alert-plugin或者dolphinscheduler-job-plugin模块下再创建自己的模块，这些插件模块需要依赖dolphinscheduler-spi，并实现dolphinscheduler-spi中提供的接口。
> 前端
>
> 由插件实现方通过一个接口来告诉前端这个插件需要使用时需要配置的参数。比如邮件告警插件要告诉前端页面需要去填写邮件接收人，抄送人；钉钉告警需要由前端填写目标群；对于Job插件来说，MR插件需要需要前端配置参数jar包路径，mr参数等等。关于这一块我是这么想的，我们在插件里设计一个所有前端支持的控件类型enum，然后每个插件可以通过一个getParams接口方法返回自己需要前端展示的参数，每个参数需要提供前端控件类型，比如input,
> radio , selector
> ,area等等；在系统启动加载插件时，系统会将每个插件对应的params生成json保存到数据库中，待前端使用。每个任务类型的参数都可以分为通用参数和自定义参数，通用参数包括（节点名称、运行标志、描述、优先级等等每个任务类型通用的参数），通过参数可以由前端写死，每个Job
> Type都一样。自定义参数是从数据库中取这个参数的params
> json，然后根据json中定义的每个参数的信息在前端生成UI元素（input,radio,selector...）。
> 但这里有一个问题，这种方式无法支持那种需要多级联动的form元素，因为想通过json来描述js的事件有比较大的困难，大家帮忙想想办法。
> 我期望通过DS Plugin的设计能够带来以下好处：
>
> 1.
>
> 将核心的业务逻辑与具体的实现解耦（比如告警逻辑与具体的告警实现方式解耦；DAG中任务的执行与具体的任务实现解耦）可以提升DS架构的扩展性。
> 2.
>
>
> 对于插件的开发者来说，不需要关注DS上层的核心逻辑是怎么运行的，他们只需要关注SPI层提供给他们的接口，然后实现这些接口即可。这将显著的减少新的任务类型的开发上手难度。
> 3.
>
>
> 插件的实现层只能使用SPI中开放出来的接口和类，除此之外插件实现层尽量不要依赖DS中其它模块的代码。如果插件层想使用dolphinscheduler-common模块中的一个工具类，那正确的做法应该是先将该工具类从dolphinscheduler-common模块中迁移到dolphinscheduler-spi中。或者将工具类统一放到一个单独的模块中，不要和上层的可能被经常修改的核心业务逻辑代码放到一起，保证这个可以被其它插件依赖的模块的功能边界尽量的小，目前我只是将一些utils放到了dolphinscheduler-spi中，可以看到即使这样spi模块的pom文件中也需要添加一些utils中必须用到的依赖，比如slf4j,commons-xx,poi等等。为了保证依赖的问题能够在编译阶段时被发现，而不是等于运行时再发现，我会通过一个maven插件对SPI模块中的依赖做检查，要求所有SPI中使用的第三方依赖（包括传递依赖）都必须明确定义并指明版本，而且scope不允许是provided级以下的，这会导致一但大量的工具类放到spi中，spi中的依赖会比较多，但应该也不是啥大问题，关于这一点大家帮忙发表下看法。最终的目标是每个不同的插件的实现之间的代码不会相互影响，比如MR作业的实现不会影响到Spark作业的代码。这对于代码的review人员来说，也可以很大程度上提高review的效率。
> 4.
>
>
> 只要SPI层的代码不被修改，理论上dolphinscheduler-server层的修改和插件层的代码修改互不影响，大家基于同一套SPI的接口来开发。
> 5.
>
> 对于DS的使用者来说，他们可以选择自己需要的插件，特殊情况下他们只需要编译自己需要的插件的代码即可将新的插件集成到他们已有的环境中。
>
> 进展
>
> 目前我计划先从Alert插件化开始，做一个尝试。因为Alert模块比较简单，适合做一个快速的改造。
>
> 1.
>
> 我完成了一个自定义maven插件，它的作用是可以自动检查spi和插件相关的依赖，并且可以支持在pom文件中添加dolphinscheduler-plugin标识从而让DS在编译时自动生成相关的META-INF/services文件。详细的信息可以参考
> ： https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin
> 2.
>
> gaojun2048#1
> <https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> 这个pr是我在dev分支上进行的一部分代码的开发，目前已经实现了SPI和AlertServer的插件化改造。
> 3.
>
> 在插件实现层，已经完成了邮件告警插件的改造。
> 4.
>
>
> 在dolphinscheduler-alert模块中完成了插件的配置、加载功能的开发。目前插件支持开发时通过alert.properties中配置alert.plugin.binding=xxx/pom.xml来指定加载插件和在服务器上部署运行时通过在alert.properties中配置alert.plugin.dir=/xxx/xxx/目录来加载插件。
> 当然，alert.plugin.dir参数需要DS在打包部署时将所有插件都放到alert.plugin.dir指定的目录下，这个功能我还没有做。大家觉得插件目录放到哪合适，我建议放到${DS_INSTALL_DIR}lib/plugin目录下
> 。
>
> 代码中有很多的注释，主要的目的是希望通过注释让大家了更容易了解我的想法。
>
> 代码中有一些//TODO 的注释，这些是需要进一步改造完成的需求点，目前我还没有想好该如何实现。
> 目前的问题
>
> 1.
>
>
> 插件会通过PluginFactory中的getParams方法返回自己需要前端展示的参数，比如邮件告警需要前端填写收件人，抄送人。正常情况下前端是不知道有哪些告警插件可以用的，那前端就需要从一个地方读取系统当前已经配置了的可用的告警插件列表，然后将插件的名称（比如’邮件‘）、每个插件需要填写的参数展示在前端。那这些信息应该保存在哪？要设计一张mysql表用来保存这些信息吗？
> 2.
>
>
> 如果在A时刻，系统里有邮件告警插件，钉钉告警插件，用户在前端界面上启用了这两个插件，并且已经填写了邮件接收人，钉钉的接收人等等信息。时间来到B时刻，用户将邮件告警插件从系统中删除了，这个时候用户在邮件告警上填写的接收人信息等该保留吗？怎么保留？如果不保留，用户再次添加邮件告警插件的jar到系统中，那用户是需要重启填写邮件接收人信息吗？
>
> 以上的问题是目前告警插件化的一些遗留的问题，希望大家能一起出出主意。
>
> 当然，我一个人的力量是无法完成整个插件化的改造的，我希望得到大家的支持和帮助，给我提提建议，或者完成一些子任务，谢谢大家。
> 任务列表(暂不开放，需要等上面的问题讨论出结果之后再决定任务列表)
>
> 目前还有如下任务是需要完成的。
>
> 1.
>
>
> 设计数据库插件表，在插件加载时会需要把插件的ID，前端展示名（中文名/英文名），插件需要在前端展示的参数列表保存在数据库。这样前端页面才能动态的知道系统有哪些告警插件是可以使用的，每个插件又需要填写什么参数，每个参数是什么类型的表单元素。
> 2.
>
> 我只简单的完成了邮件告警的改造，还需要微信告警插件改造。
> 3.
>
> 钉钉告警插件改造。
> 4.
>
> 企业微信告警的改造。
> 5.
>
> 打包时需要把插件放到特殊的插件目录中。
> 6.
>
> ...
>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
回复： Re: [Poprosal] DolphinScheduler Plugin
Posted by 向偲彪 <xi...@qq.com>.
alpacajs can let json data be displayed in the form of form, but it lacks the method binding for each field and how to submit the entire form data documentation. Does anyone have used alpacajs or a friend who is familiar with alpacajs?
DolphinScheduler(Incubator) committer
Caibiao Xiang 向偲彪
------------------ 原始邮件 ------------------
发件人: "JUN GAO"<gaojun2048@gmail.com>;
发送时间: 2020年6月11日(星期四) 下午3:48
收件人: "dev"<dev@dolphinscheduler.apache.org>;"leonbao"<leonbao@apache.org>;"lgcareer2019"<lgcareer2019@outlook.com>;"GabryWu"<wu_shao_jie@qq.com>;"Xiaochun Liu"<liuxiaochun@apache.org>;"许昌群"<changqun.xu@17zuoye.com>;
主题: Re: Re: [Poprosal] DolphinScheduler Plugin
@leonbao@apache.org <leonbao@apache.org> @lgcareer2019@outlook.com
<lgcareer2019@outlook.com> @GabryWu <wu_shao_jie@qq.com> @Xiaochun Liu
<liuxiaochun@apache.org> @Han Gao @许昌群 <changqun.xu@17zuoye.com>
Thank you eveny one.
I see that most of the friends who participate in this proposal discussion
support it. Therefore, I will continue the work of DS plug-in design. Next,
I applied to add the incubator-dolphinscheduler-maven-plugin repository to
apache, and applied to create a branch on the incubator-dolphinscheduler
repository for the next plug-in development and collaboration. And then I
will open this todo list.
leon bao <leonbao@apache.org> 于2020年6月10日周三 下午6:19写道：
> As a developer, I think
> 1. A good architecture allows a modification that will not affect the
> operation of other modules, which is not possible in the current DS. For
> example, adding new task types may affect the execution of other tasks
> because they are all in one module. .
> If good isolation is achieved, any modification to a module will not affect
> the normal operation of other modules in the system.
> 2. We need an architecture so that developers and the community stay in
> sync, and each version of the community has the least impact on developers.
> This requires a more flexible architecture and will attract more
> developers.
>
> so i think the plug-in is suitable for DS.
>
>
> lgcareer2019@outlook.com <lgcareer2019@outlook.com> 于2020年6月10日周三
> 下午4:48写道：
>
> > @jun gao
> > So great to decouple the upper core logic of DS from the specific
> > implementation.
> > I think this design is more scalable and developers can cooperate in
> > develop more easily.So I agree with it.
> >
> > @wu shaoj @changqun
> > I saw you may be have other options that flexible plugin mechanism is
> > complex and in a low priority.
> >
> > I think we can redesign the alert service in this time first,and I saw
> jun
> > gao have already done a lot of works.
> >
> >
> >
> > DolphinScheduler(Incubator) PPMC
> > Gang Li 李岗
> >
> > lgcareer2019@outlook.com<mailto:lgcareer2019@outlook.com>
> >
> > 发件人： wu shaoj
> > 发送时间： 2020-06-04 17:44
> > 收件人： dev@dolphinscheduler.apache.org
> > 主题： Re: 答复: [Poprosal] DolphinScheduler Plugin
> > Yes, we should forcus on the job with high priority .
> > No need to satisfy all customer requirement which might be unreasonable
> >
> > On 2020/6/4, 17:41, "Hemin Wen" <wenhemin@apache.org> wrote:
> >
> > Very detailed design.
> > 1. About alert module, I agree with @许昌群.
> > Feedback from the community on the alert module, the alert module
> > function
> > is relatively stable,
> > and the current notification function plus webhook whether it can
> meet
> > the
> > needs.
> >
> > 2.It is not recommended that the UI be modularized, this will
> increase
> > the
> > complexity of module development and debugging,
> > If there is more complicated UI interaction, it will cause trouble to
> > the
> > development, Not only need to understand UI development,
> > but also understand modular configuration
> >
> > --------------------
> > DolphinScheduler(Incubator) Commtter
> > Hemin Wen 温合民
> > wenhemin@apache.org
> > --------------------
> >
> >
> > Han Gao <dhangao@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> >
> > > Hi folks,
> > >
> > > I prefer plugin way. @许昌群<mailto:changqun.xu@17zuoye.com> provide
> an
> > > alert way like webhook(pls correct me if I'm wrong), which we can
> > support
> > > as an official plugin.
> > >
> > > The difference between webhook and plugin is you should start a
> > server to
> > > accept alert request in webhook, but in plugin way you don't need
> to
> > do
> > > that.
> > >
> > > IMO, I prefer implement some official plugins in AlertServer
> package
> > > directly, as for others, we could put it in a plugin folder. I've
> > > implemented a plugin class loader in DS common package, which used
> > to load
> > > plugin and avoid package conflict. Put official plugins directly in
> > > AlertServer package can use the common packages, and will cost less
> > memory
> > > for class meta data.
> > >
> > > But put all plugin into a specific folder is fine too, because it
> > looks
> > > clean.
> > >
> > > BTW, I think there is no conflict to support both webhook and
> plugin.
> > >
> > > Thanks,
> > > Han Gao
> > >
> > > ________________________________
> > > From: wu shaoj <gabrywu@apache.org>
> > > Sent: Thursday, June 4, 2020 15:13
> > > To: dev@dolphinscheduler.apache.org <
> dev@dolphinscheduler.apache.org
> > >;
> > > 许昌群 <changqun.xu@17zuoye.com>
> > > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> > >
> > > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and
> in
> > a low
> > > priority I think.
> > >
> > > On 2020/6/4, 15:04, "许昌群" <changqun.xu@17zuoye.com> wrote:
> > >
> > > Hi all
> > >
> > > Regarding the alarm, I have some other ideas, please refer to
> it.
> > >
> > > At present, there are many types of message notification
> > components,
> > > such as email/WeChat/SMS, or some notification components developed
> > by the
> > > company.If it is not realistic to connect all the components, can
> we
> > change
> > > the way of thinking and flip this process? We provide an interface
> > for
> > > sending information, allowing users to interface with it.
> > >
> > > Described as follows:
> > > 1. Define the sending interface of the message in DS, such as
> > sendMsg
> > > 2. Load the user-configured interface implementation, for
> > example,
> > > http implementation is http://192.168.1.1:8080/sendMsg/email
> > > 3. Define the message sending format, {"tos":"softxcq@sina.com
> ",
> > > "content":"taskA failed."}
> > > 4. Send http request.
> > http://192.168.1.1:8080/sendMsg/email?data={
> > > "tos":"softxcq@sina.com", "content":"taskA failed."}
> > >
> > > If there are other alarm methods, add an http interface
> > >
> > > Finally, you only need to provide several default alarm
> > functions in
> > > DS to transfer other message notification functions to
> > users.Although you
> > > will lose some ease of use out of the box, it will increase
> > flexibility.
> > > You can implement your own system to ensure that the DS alarm logic
> > is
> > > simple and easy to use.
> > >
> > >
> > > ________________________________
> > > 发件人: JUN GAO <gaojun2048@gmail.com>
> > > 发送时间: 2020年6月3日 12:13:21
> > > 收件人: dev
> > > 主题: Re: [Poprosal] DolphinScheduler Plugin
> > >
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement
> > > it as plugin?
> > >
> > > I think Alert should be divided into two parts. One is that the
> > server
> > > collects various alarm information. The other is to pass these
> > warning
> > > contents through optional methods (such as email, WeChat). I
> > think the
> > > specific alarm method should not be implemented by the
> > AlertServer
> > > server.
> > > It should be abstracted into an interface and implemented by a
> > concrete
> > > plug-in implementation layer. If someone understands and is
> > familiar
> > > with
> > > other alarm methods, he can implement other alarm plug-ins
> based
> > on the
> > > plug-in design. If he wants, he can contribute this plug-in to
> > us.
> > > Plug-inization is just a way to decouple projects.
> > >
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > > thirdparty? If
> > > it is a thirdparty, it may generally be managed by its own git,
> > and it
> > > will
> > > not be placed in the dolphinscheduler related directory.
> > >
> > > Plug-inization is just a way to decouple projects. All Plugin
> is
> > part
> > > of dolphinscheduler.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> > powerful.
> > >
> > > @daolidong We can FYI this.
> > >
> > > Xiaochun Liu <liuxiaochun@apache.org> 于2020年6月3日周三 上午11:35写道：
> > >
> > > > Good proposal, but I have some question as following:
> > > > 1、Alert is very independent and common function, why should
> we
> > to
> > > > implement it as plugin?
> > > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > > thirdparty?
> > > > If it is a thirdparty, it may generally be managed by its own
> > git,
> > > and it
> > > > will not be placed in the dolphinscheduler related directory.
> > > >
> > > > And we can use jsonschema to solve the dynamic ui, for
> example:
> > > alpacajs(
> > > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> > powerful.
> > > >
> > > > Best Regards
> > > > ---------------
> > > > DolphinScheduler(Incubator) Committer
> > > > Xiaochun Liu 刘小春
> > > > liuxiaochun@apache.org
> > > > ---------------
> > > >
> > > >
> > > >
> > > > > 2020年6月2日 下午3:04，JUN GAO <gaojun2048@gmail.com> 写道：
> > > > >
> > > > >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > > >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > > >
> > >
> > >
> > > --
> > >
> > > DolphinScheduler(Incubator) PPMC
> > > Jun Gao 高俊
> > > gaojun2048@gmail.com
> > >
> >
>
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: Re: [Poprosal] DolphinScheduler Plugin
Posted by JUN GAO <ga...@gmail.com>.
@leonbao@apache.org <le...@apache.org> @lgcareer2019@outlook.com
<lg...@outlook.com> @GabryWu <wu...@qq.com> @Xiaochun Liu
<li...@apache.org> @Han Gao @许昌群 <ch...@17zuoye.com>
Thank you eveny one.
I see that most of the friends who participate in this proposal discussion
support it. Therefore, I will continue the work of DS plug-in design. Next,
I applied to add the incubator-dolphinscheduler-maven-plugin repository to
apache, and applied to create a branch on the incubator-dolphinscheduler
repository for the next plug-in development and collaboration. And then I
will open this todo list.
leon bao <le...@apache.org> 于2020年6月10日周三 下午6:19写道：
> As a developer, I think
> 1. A good architecture allows a modification that will not affect the
> operation of other modules, which is not possible in the current DS. For
> example, adding new task types may affect the execution of other tasks
> because they are all in one module. .
> If good isolation is achieved, any modification to a module will not affect
> the normal operation of other modules in the system.
> 2. We need an architecture so that developers and the community stay in
> sync, and each version of the community has the least impact on developers.
> This requires a more flexible architecture and will attract more
> developers.
>
> so i think the plug-in is suitable for DS.
>
>
> lgcareer2019@outlook.com <lg...@outlook.com> 于2020年6月10日周三
> 下午4:48写道：
>
> > @jun gao
> > So great to decouple the upper core logic of DS from the specific
> > implementation.
> > I think this design is more scalable and developers can cooperate in
> > develop more easily.So I agree with it.
> >
> > @wu shaoj @changqun
> > I saw you may be have other options that flexible plugin mechanism is
> > complex and in a low priority.
> >
> > I think we can redesign the alert service in this time first,and I saw
> jun
> > gao have already done a lot of works.
> >
> >
> >
> > DolphinScheduler(Incubator) PPMC
> > Gang Li 李岗
> >
> > lgcareer2019@outlook.com<ma...@outlook.com>
> >
> > 发件人： wu shaoj
> > 发送时间： 2020-06-04 17:44
> > 收件人： dev@dolphinscheduler.apache.org
> > 主题： Re: 答复: [Poprosal] DolphinScheduler Plugin
> > Yes, we should forcus on the job with high priority .
> > No need to satisfy all customer requirement which might be unreasonable
> >
> > On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
> >
> > Very detailed design.
> > 1. About alert module, I agree with @许昌群.
> > Feedback from the community on the alert module, the alert module
> > function
> > is relatively stable,
> > and the current notification function plus webhook whether it can
> meet
> > the
> > needs.
> >
> > 2.It is not recommended that the UI be modularized, this will
> increase
> > the
> > complexity of module development and debugging,
> > If there is more complicated UI interaction, it will cause trouble to
> > the
> > development, Not only need to understand UI development,
> > but also understand modular configuration
> >
> > --------------------
> > DolphinScheduler(Incubator) Commtter
> > Hemin Wen 温合民
> > wenhemin@apache.org
> > --------------------
> >
> >
> > Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> >
> > > Hi folks,
> > >
> > > I prefer plugin way. @许昌群<ma...@17zuoye.com> provide
> an
> > > alert way like webhook(pls correct me if I'm wrong), which we can
> > support
> > > as an official plugin.
> > >
> > > The difference between webhook and plugin is you should start a
> > server to
> > > accept alert request in webhook, but in plugin way you don't need
> to
> > do
> > > that.
> > >
> > > IMO, I prefer implement some official plugins in AlertServer
> package
> > > directly, as for others, we could put it in a plugin folder. I've
> > > implemented a plugin class loader in DS common package, which used
> > to load
> > > plugin and avoid package conflict. Put official plugins directly in
> > > AlertServer package can use the common packages, and will cost less
> > memory
> > > for class meta data.
> > >
> > > But put all plugin into a specific folder is fine too, because it
> > looks
> > > clean.
> > >
> > > BTW, I think there is no conflict to support both webhook and
> plugin.
> > >
> > > Thanks,
> > > Han Gao
> > >
> > > ________________________________
> > > From: wu shaoj <ga...@apache.org>
> > > Sent: Thursday, June 4, 2020 15:13
> > > To: dev@dolphinscheduler.apache.org <
> dev@dolphinscheduler.apache.org
> > >;
> > > 许昌群 <ch...@17zuoye.com>
> > > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> > >
> > > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and
> in
> > a low
> > > priority I think.
> > >
> > > On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
> > >
> > > Hi all
> > >
> > > Regarding the alarm, I have some other ideas, please refer to
> it.
> > >
> > > At present, there are many types of message notification
> > components,
> > > such as email/WeChat/SMS, or some notification components developed
> > by the
> > > company.If it is not realistic to connect all the components, can
> we
> > change
> > > the way of thinking and flip this process? We provide an interface
> > for
> > > sending information, allowing users to interface with it.
> > >
> > > Described as follows:
> > > 1. Define the sending interface of the message in DS, such as
> > sendMsg
> > > 2. Load the user-configured interface implementation, for
> > example,
> > > http implementation is http://192.168.1.1:8080/sendMsg/email
> > > 3. Define the message sending format, {"tos":"softxcq@sina.com
> ",
> > > "content":"taskA failed."}
> > > 4. Send http request.
> > http://192.168.1.1:8080/sendMsg/email?data={
> > > "tos":"softxcq@sina.com", "content":"taskA failed."}
> > >
> > > If there are other alarm methods, add an http interface
> > >
> > > Finally, you only need to provide several default alarm
> > functions in
> > > DS to transfer other message notification functions to
> > users.Although you
> > > will lose some ease of use out of the box, it will increase
> > flexibility.
> > > You can implement your own system to ensure that the DS alarm logic
> > is
> > > simple and easy to use.
> > >
> > >
> > > ________________________________
> > > 发件人: JUN GAO <ga...@gmail.com>
> > > 发送时间: 2020年6月3日 12:13:21
> > > 收件人: dev
> > > 主题: Re: [Poprosal] DolphinScheduler Plugin
> > >
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement
> > > it as plugin?
> > >
> > > I think Alert should be divided into two parts. One is that the
> > server
> > > collects various alarm information. The other is to pass these
> > warning
> > > contents through optional methods (such as email, WeChat). I
> > think the
> > > specific alarm method should not be implemented by the
> > AlertServer
> > > server.
> > > It should be abstracted into an interface and implemented by a
> > concrete
> > > plug-in implementation layer. If someone understands and is
> > familiar
> > > with
> > > other alarm methods, he can implement other alarm plug-ins
> based
> > on the
> > > plug-in design. If he wants, he can contribute this plug-in to
> > us.
> > > Plug-inization is just a way to decouple projects.
> > >
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > > thirdparty? If
> > > it is a thirdparty, it may generally be managed by its own git,
> > and it
> > > will
> > > not be placed in the dolphinscheduler related directory.
> > >
> > > Plug-inization is just a way to decouple projects. All Plugin
> is
> > part
> > > of dolphinscheduler.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> > powerful.
> > >
> > > @daolidong We can FYI this.
> > >
> > > Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> > >
> > > > Good proposal, but I have some question as following:
> > > > 1、Alert is very independent and common function, why should
> we
> > to
> > > > implement it as plugin?
> > > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > > thirdparty?
> > > > If it is a thirdparty, it may generally be managed by its own
> > git,
> > > and it
> > > > will not be placed in the dolphinscheduler related directory.
> > > >
> > > > And we can use jsonschema to solve the dynamic ui, for
> example:
> > > alpacajs(
> > > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> > powerful.
> > > >
> > > > Best Regards
> > > > ---------------
> > > > DolphinScheduler(Incubator) Committer
> > > > Xiaochun Liu 刘小春
> > > > liuxiaochun@apache.org
> > > > ---------------
> > > >
> > > >
> > > >
> > > > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > > > >
> > > > >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > > >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > > >
> > >
> > >
> > > --
> > >
> > > DolphinScheduler(Incubator) PPMC
> > > Jun Gao 高俊
> > > gaojun2048@gmail.com
> > >
> >
>
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: Re: [Poprosal] DolphinScheduler Plugin
Posted by leon bao <le...@apache.org>.
As a developer, I think
1. A good architecture allows a modification that will not affect the
operation of other modules, which is not possible in the current DS. For
example, adding new task types may affect the execution of other tasks
because they are all in one module. .
If good isolation is achieved, any modification to a module will not affect
the normal operation of other modules in the system.
2. We need an architecture so that developers and the community stay in
sync, and each version of the community has the least impact on developers.
This requires a more flexible architecture and will attract more developers.
so i think the plug-in is suitable for DS.
lgcareer2019@outlook.com <lg...@outlook.com> 于2020年6月10日周三 下午4:48写道：
> @jun gao
> So great to decouple the upper core logic of DS from the specific
> implementation.
> I think this design is more scalable and developers can cooperate in
> develop more easily.So I agree with it.
>
> @wu shaoj @changqun
> I saw you may be have other options that flexible plugin mechanism is
> complex and in a low priority.
>
> I think we can redesign the alert service in this time first,and I saw jun
> gao have already done a lot of works.
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> 发件人： wu shaoj
> 发送时间： 2020-06-04 17:44
> 收件人： dev@dolphinscheduler.apache.org
> 主题： Re: 答复: [Poprosal] DolphinScheduler Plugin
> Yes, we should forcus on the job with high priority .
> No need to satisfy all customer requirement which might be unreasonable
>
> On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
>
> Very detailed design.
> 1. About alert module, I agree with @许昌群.
> Feedback from the community on the alert module, the alert module
> function
> is relatively stable,
> and the current notification function plus webhook whether it can meet
> the
> needs.
>
> 2.It is not recommended that the UI be modularized, this will increase
> the
> complexity of module development and debugging,
> If there is more complicated UI interaction, it will cause trouble to
> the
> development, Not only need to understand UI development,
> but also understand modular configuration
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
>
> > Hi folks,
> >
> > I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> > alert way like webhook(pls correct me if I'm wrong), which we can
> support
> > as an official plugin.
> >
> > The difference between webhook and plugin is you should start a
> server to
> > accept alert request in webhook, but in plugin way you don't need to
> do
> > that.
> >
> > IMO, I prefer implement some official plugins in AlertServer package
> > directly, as for others, we could put it in a plugin folder. I've
> > implemented a plugin class loader in DS common package, which used
> to load
> > plugin and avoid package conflict. Put official plugins directly in
> > AlertServer package can use the common packages, and will cost less
> memory
> > for class meta data.
> >
> > But put all plugin into a specific folder is fine too, because it
> looks
> > clean.
> >
> > BTW, I think there is no conflict to support both webhook and plugin.
> >
> > Thanks,
> > Han Gao
> >
> > ________________________________
> > From: wu shaoj <ga...@apache.org>
> > Sent: Thursday, June 4, 2020 15:13
> > To: dev@dolphinscheduler.apache.org <dev@dolphinscheduler.apache.org
> >;
> > 许昌群 <ch...@17zuoye.com>
> > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> >
> > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in
> a low
> > priority I think.
> >
> > On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
> >
> > Hi all
> >
> > Regarding the alarm, I have some other ideas, please refer to it.
> >
> > At present, there are many types of message notification
> components,
> > such as email/WeChat/SMS, or some notification components developed
> by the
> > company.If it is not realistic to connect all the components, can we
> change
> > the way of thinking and flip this process? We provide an interface
> for
> > sending information, allowing users to interface with it.
> >
> > Described as follows:
> > 1. Define the sending interface of the message in DS, such as
> sendMsg
> > 2. Load the user-configured interface implementation, for
> example,
> > http implementation is http://192.168.1.1:8080/sendMsg/email
> > 3. Define the message sending format, {"tos":"softxcq@sina.com",
> > "content":"taskA failed."}
> > 4. Send http request.
> http://192.168.1.1:8080/sendMsg/email?data={
> > "tos":"softxcq@sina.com", "content":"taskA failed."}
> >
> > If there are other alarm methods, add an http interface
> >
> > Finally, you only need to provide several default alarm
> functions in
> > DS to transfer other message notification functions to
> users.Although you
> > will lose some ease of use out of the box, it will increase
> flexibility.
> > You can implement your own system to ensure that the DS alarm logic
> is
> > simple and easy to use.
> >
> >
> > ________________________________
> > 发件人: JUN GAO <ga...@gmail.com>
> > 发送时间: 2020年6月3日 12:13:21
> > 收件人: dev
> > 主题: Re: [Poprosal] DolphinScheduler Plugin
> >
> > 1、Alert is very independent and common function, why should we to
> > implement
> > it as plugin?
> >
> > I think Alert should be divided into two parts. One is that the
> server
> > collects various alarm information. The other is to pass these
> warning
> > contents through optional methods (such as email, WeChat). I
> think the
> > specific alarm method should not be implemented by the
> AlertServer
> > server.
> > It should be abstracted into an interface and implemented by a
> concrete
> > plug-in implementation layer. If someone understands and is
> familiar
> > with
> > other alarm methods, he can implement other alarm plug-ins based
> on the
> > plug-in design. If he wants, he can contribute this plug-in to
> us.
> > Plug-inization is just a way to decouple projects.
> >
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty? If
> > it is a thirdparty, it may generally be managed by its own git,
> and it
> > will
> > not be placed in the dolphinscheduler related directory.
> >
> > Plug-inization is just a way to decouple projects. All Plugin is
> part
> > of dolphinscheduler.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> >
> > @daolidong We can FYI this.
> >
> > Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> >
> > > Good proposal, but I have some question as following:
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement it as plugin?
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty?
> > > If it is a thirdparty, it may generally be managed by its own
> git,
> > and it
> > > will not be placed in the dolphinscheduler related directory.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> > >
> > > Best Regards
> > > ---------------
> > > DolphinScheduler(Incubator) Committer
> > > Xiaochun Liu 刘小春
> > > liuxiaochun@apache.org
> > > ---------------
> > >
> > >
> > >
> > > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > > >
> > > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > >
> >
> >
> > --
> >
> > DolphinScheduler(Incubator) PPMC
> > Jun Gao 高俊
> > gaojun2048@gmail.com
> >
>
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: Re: [Poprosal] DolphinScheduler Plugin
Posted by "lgcareer2019@outlook.com" <lg...@outlook.com>.
@jun gao
So great to decouple the upper core logic of DS from the specific implementation.
I think this design is more scalable and developers can cooperate in develop more easily.So I agree with it.
@wu shaoj @changqun
I saw you may be have other options that flexible plugin mechanism is complex and in a low priority.
I think we can redesign the alert service in this time first,and I saw jun gao have already done a lot of works.
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： wu shaoj
发送时间： 2020-06-04 17:44
收件人： dev@dolphinscheduler.apache.org
主题： Re: 答复: [Poprosal] DolphinScheduler Plugin
Yes, we should forcus on the job with high priority .
No need to satisfy all customer requirement which might be unreasonable
On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
Very detailed design.
1. About alert module, I agree with @许昌群.
Feedback from the community on the alert module, the alert module function
is relatively stable,
and the current notification function plus webhook whether it can meet the
needs.
2.It is not recommended that the UI be modularized, this will increase the
complexity of module development and debugging,
If there is more complicated UI interaction, it will cause trouble to the
development, Not only need to understand UI development,
but also understand modular configuration
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> Hi folks,
>
> I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> alert way like webhook(pls correct me if I'm wrong), which we can support
> as an official plugin.
>
> The difference between webhook and plugin is you should start a server to
> accept alert request in webhook, but in plugin way you don't need to do
> that.
>
> IMO, I prefer implement some official plugins in AlertServer package
> directly, as for others, we could put it in a plugin folder. I've
> implemented a plugin class loader in DS common package, which used to load
> plugin and avoid package conflict. Put official plugins directly in
> AlertServer package can use the common packages, and will cost less memory
> for class meta data.
>
> But put all plugin into a specific folder is fine too, because it looks
> clean.
>
> BTW, I think there is no conflict to support both webhook and plugin.
>
> Thanks,
> Han Gao
>
> ________________________________
> From: wu shaoj <ga...@apache.org>
> Sent: Thursday, June 4, 2020 15:13
> To: dev@dolphinscheduler.apache.org <de...@dolphinscheduler.apache.org>;
> 许昌群 <ch...@17zuoye.com>
> Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
>
> Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low
> priority I think.
>
> On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
>
> Hi all
>
> Regarding the alarm, I have some other ideas, please refer to it.
>
> At present, there are many types of message notification components,
> such as email/WeChat/SMS, or some notification components developed by the
> company.If it is not realistic to connect all the components, can we change
> the way of thinking and flip this process? We provide an interface for
> sending information, allowing users to interface with it.
>
> Described as follows:
> 1. Define the sending interface of the message in DS, such as sendMsg
> 2. Load the user-configured interface implementation, for example,
> http implementation is http://192.168.1.1:8080/sendMsg/email
> 3. Define the message sending format, {"tos":"softxcq@sina.com",
> "content":"taskA failed."}
> 4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={
> "tos":"softxcq@sina.com", "content":"taskA failed."}
>
> If there are other alarm methods, add an http interface
>
> Finally, you only need to provide several default alarm functions in
> DS to transfer other message notification functions to users.Although you
> will lose some ease of use out of the box, it will increase flexibility.
> You can implement your own system to ensure that the DS alarm logic is
> simple and easy to use.
>
>
> ________________________________
> 发件人: JUN GAO <ga...@gmail.com>
> 发送时间: 2020年6月3日 12:13:21
> 收件人: dev
> 主题: Re: [Poprosal] DolphinScheduler Plugin
>
> 1、Alert is very independent and common function, why should we to
> implement
> it as plugin?
>
> I think Alert should be divided into two parts. One is that the server
> collects various alarm information. The other is to pass these warning
> contents through optional methods (such as email, WeChat). I think the
> specific alarm method should not be implemented by the AlertServer
> server.
> It should be abstracted into an interface and implemented by a concrete
> plug-in implementation layer. If someone understands and is familiar
> with
> other alarm methods, he can implement other alarm plug-ins based on the
> plug-in design. If he wants, he can contribute this plug-in to us.
> Plug-inization is just a way to decouple projects.
>
> 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty? If
> it is a thirdparty, it may generally be managed by its own git, and it
> will
> not be placed in the dolphinscheduler related directory.
>
> Plug-inization is just a way to decouple projects. All Plugin is part
> of dolphinscheduler.
>
> And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> @daolidong We can FYI this.
>
> Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
>
> > Good proposal, but I have some question as following:
> > 1、Alert is very independent and common function, why should we to
> > implement it as plugin?
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty?
> > If it is a thirdparty, it may generally be managed by its own git,
> and it
> > will not be placed in the dolphinscheduler related directory.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) Committer
> > Xiaochun Liu 刘小春
> > liuxiaochun@apache.org
> > ---------------
> >
> >
> >
> > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > >
> > > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> >
>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by wu shaoj <ga...@apache.org>.
+1
On 2020/6/5, 11:51, "JUN GAO" <ga...@gmail.com> wrote:
Yes, Function development is very important, but I think the adjustment of
the architecture is a work that should be more valued. Because the more
work is done on an inappropriate architecture, the higher the cost of
future transformation.
The current code architecture is not friendly to the scalability of DS, I
think we need to do more work in this area. And plug-in is also a very
important feature of DS.
wu shaoj <ga...@apache.org> 于2020年6月4日周四 下午5:50写道：
> Yes, we should forcus on the job with high priority .
> No need to satisfy all customer requirement which might be unreasonable
>
> On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
>
> Very detailed design.
> 1. About alert module, I agree with @许昌群.
> Feedback from the community on the alert module, the alert module
> function
> is relatively stable,
> and the current notification function plus webhook whether it can meet
> the
> needs.
>
> 2.It is not recommended that the UI be modularized, this will increase
> the
> complexity of module development and debugging,
> If there is more complicated UI interaction, it will cause trouble to
> the
> development, Not only need to understand UI development,
> but also understand modular configuration
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
>
> > Hi folks,
> >
> > I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> > alert way like webhook(pls correct me if I'm wrong), which we can
> support
> > as an official plugin.
> >
> > The difference between webhook and plugin is you should start a
> server to
> > accept alert request in webhook, but in plugin way you don't need to
> do
> > that.
> >
> > IMO, I prefer implement some official plugins in AlertServer package
> > directly, as for others, we could put it in a plugin folder. I've
> > implemented a plugin class loader in DS common package, which used
> to load
> > plugin and avoid package conflict. Put official plugins directly in
> > AlertServer package can use the common packages, and will cost less
> memory
> > for class meta data.
> >
> > But put all plugin into a specific folder is fine too, because it
> looks
> > clean.
> >
> > BTW, I think there is no conflict to support both webhook and plugin.
> >
> > Thanks,
> > Han Gao
> >
> > ________________________________
> > From: wu shaoj <ga...@apache.org>
> > Sent: Thursday, June 4, 2020 15:13
> > To: dev@dolphinscheduler.apache.org <dev@dolphinscheduler.apache.org
> >;
> > 许昌群 <ch...@17zuoye.com>
> > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> >
> > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in
> a low
> > priority I think.
> >
> > On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
> >
> > Hi all
> >
> > Regarding the alarm, I have some other ideas, please refer to it.
> >
> > At present, there are many types of message notification
> components,
> > such as email/WeChat/SMS, or some notification components developed
> by the
> > company.If it is not realistic to connect all the components, can we
> change
> > the way of thinking and flip this process? We provide an interface
> for
> > sending information, allowing users to interface with it.
> >
> > Described as follows:
> > 1. Define the sending interface of the message in DS, such as
> sendMsg
> > 2. Load the user-configured interface implementation, for
> example,
> > http implementation is http://192.168.1.1:8080/sendMsg/email
> > 3. Define the message sending format, {"tos":"softxcq@sina.com",
> > "content":"taskA failed."}
> > 4. Send http request.
> http://192.168.1.1:8080/sendMsg/email?data={
> > "tos":"softxcq@sina.com", "content":"taskA failed."}
> >
> > If there are other alarm methods, add an http interface
> >
> > Finally, you only need to provide several default alarm
> functions in
> > DS to transfer other message notification functions to
> users.Although you
> > will lose some ease of use out of the box, it will increase
> flexibility.
> > You can implement your own system to ensure that the DS alarm logic
> is
> > simple and easy to use.
> >
> >
> > ________________________________
> > 发件人: JUN GAO <ga...@gmail.com>
> > 发送时间: 2020年6月3日 12:13:21
> > 收件人: dev
> > 主题: Re: [Poprosal] DolphinScheduler Plugin
> >
> > 1、Alert is very independent and common function, why should we to
> > implement
> > it as plugin?
> >
> > I think Alert should be divided into two parts. One is that the
> server
> > collects various alarm information. The other is to pass these
> warning
> > contents through optional methods (such as email, WeChat). I
> think the
> > specific alarm method should not be implemented by the
> AlertServer
> > server.
> > It should be abstracted into an interface and implemented by a
> concrete
> > plug-in implementation layer. If someone understands and is
> familiar
> > with
> > other alarm methods, he can implement other alarm plug-ins based
> on the
> > plug-in design. If he wants, he can contribute this plug-in to
> us.
> > Plug-inization is just a way to decouple projects.
> >
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty? If
> > it is a thirdparty, it may generally be managed by its own git,
> and it
> > will
> > not be placed in the dolphinscheduler related directory.
> >
> > Plug-inization is just a way to decouple projects. All Plugin is
> part
> > of dolphinscheduler.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> >
> > @daolidong We can FYI this.
> >
> > Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> >
> > > Good proposal, but I have some question as following:
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement it as plugin?
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty?
> > > If it is a thirdparty, it may generally be managed by its own
> git,
> > and it
> > > will not be placed in the dolphinscheduler related directory.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> > >
> > > Best Regards
> > > ---------------
> > > DolphinScheduler(Incubator) Committer
> > > Xiaochun Liu 刘小春
> > > liuxiaochun@apache.org
> > > ---------------
> > >
> > >
> > >
> > > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > > >
> > > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > >
> >
> >
> > --
> >
> > DolphinScheduler(Incubator) PPMC
> > Jun Gao 高俊
> > gaojun2048@gmail.com
> >
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by JUN GAO <ga...@gmail.com>.
Yes, Function development is very important, but I think the adjustment of
the architecture is a work that should be more valued. Because the more
work is done on an inappropriate architecture, the higher the cost of
future transformation.
The current code architecture is not friendly to the scalability of DS, I
think we need to do more work in this area. And plug-in is also a very
important feature of DS.
wu shaoj <ga...@apache.org> 于2020年6月4日周四 下午5:50写道：
> Yes, we should forcus on the job with high priority .
> No need to satisfy all customer requirement which might be unreasonable
>
> On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
>
> Very detailed design.
> 1. About alert module, I agree with @许昌群.
> Feedback from the community on the alert module, the alert module
> function
> is relatively stable,
> and the current notification function plus webhook whether it can meet
> the
> needs.
>
> 2.It is not recommended that the UI be modularized, this will increase
> the
> complexity of module development and debugging,
> If there is more complicated UI interaction, it will cause trouble to
> the
> development, Not only need to understand UI development,
> but also understand modular configuration
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
>
> > Hi folks,
> >
> > I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> > alert way like webhook(pls correct me if I'm wrong), which we can
> support
> > as an official plugin.
> >
> > The difference between webhook and plugin is you should start a
> server to
> > accept alert request in webhook, but in plugin way you don't need to
> do
> > that.
> >
> > IMO, I prefer implement some official plugins in AlertServer package
> > directly, as for others, we could put it in a plugin folder. I've
> > implemented a plugin class loader in DS common package, which used
> to load
> > plugin and avoid package conflict. Put official plugins directly in
> > AlertServer package can use the common packages, and will cost less
> memory
> > for class meta data.
> >
> > But put all plugin into a specific folder is fine too, because it
> looks
> > clean.
> >
> > BTW, I think there is no conflict to support both webhook and plugin.
> >
> > Thanks,
> > Han Gao
> >
> > ________________________________
> > From: wu shaoj <ga...@apache.org>
> > Sent: Thursday, June 4, 2020 15:13
> > To: dev@dolphinscheduler.apache.org <dev@dolphinscheduler.apache.org
> >;
> > 许昌群 <ch...@17zuoye.com>
> > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> >
> > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in
> a low
> > priority I think.
> >
> > On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
> >
> > Hi all
> >
> > Regarding the alarm, I have some other ideas, please refer to it.
> >
> > At present, there are many types of message notification
> components,
> > such as email/WeChat/SMS, or some notification components developed
> by the
> > company.If it is not realistic to connect all the components, can we
> change
> > the way of thinking and flip this process? We provide an interface
> for
> > sending information, allowing users to interface with it.
> >
> > Described as follows:
> > 1. Define the sending interface of the message in DS, such as
> sendMsg
> > 2. Load the user-configured interface implementation, for
> example,
> > http implementation is http://192.168.1.1:8080/sendMsg/email
> > 3. Define the message sending format, {"tos":"softxcq@sina.com",
> > "content":"taskA failed."}
> > 4. Send http request.
> http://192.168.1.1:8080/sendMsg/email?data={
> > "tos":"softxcq@sina.com", "content":"taskA failed."}
> >
> > If there are other alarm methods, add an http interface
> >
> > Finally, you only need to provide several default alarm
> functions in
> > DS to transfer other message notification functions to
> users.Although you
> > will lose some ease of use out of the box, it will increase
> flexibility.
> > You can implement your own system to ensure that the DS alarm logic
> is
> > simple and easy to use.
> >
> >
> > ________________________________
> > 发件人: JUN GAO <ga...@gmail.com>
> > 发送时间: 2020年6月3日 12:13:21
> > 收件人: dev
> > 主题: Re: [Poprosal] DolphinScheduler Plugin
> >
> > 1、Alert is very independent and common function, why should we to
> > implement
> > it as plugin?
> >
> > I think Alert should be divided into two parts. One is that the
> server
> > collects various alarm information. The other is to pass these
> warning
> > contents through optional methods (such as email, WeChat). I
> think the
> > specific alarm method should not be implemented by the
> AlertServer
> > server.
> > It should be abstracted into an interface and implemented by a
> concrete
> > plug-in implementation layer. If someone understands and is
> familiar
> > with
> > other alarm methods, he can implement other alarm plug-ins based
> on the
> > plug-in design. If he wants, he can contribute this plug-in to
> us.
> > Plug-inization is just a way to decouple projects.
> >
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty? If
> > it is a thirdparty, it may generally be managed by its own git,
> and it
> > will
> > not be placed in the dolphinscheduler related directory.
> >
> > Plug-inization is just a way to decouple projects. All Plugin is
> part
> > of dolphinscheduler.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> >
> > @daolidong We can FYI this.
> >
> > Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> >
> > > Good proposal, but I have some question as following:
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement it as plugin?
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty?
> > > If it is a thirdparty, it may generally be managed by its own
> git,
> > and it
> > > will not be placed in the dolphinscheduler related directory.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> > >
> > > Best Regards
> > > ---------------
> > > DolphinScheduler(Incubator) Committer
> > > Xiaochun Liu 刘小春
> > > liuxiaochun@apache.org
> > > ---------------
> > >
> > >
> > >
> > > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > > >
> > > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > >
> >
> >
> > --
> >
> > DolphinScheduler(Incubator) PPMC
> > Jun Gao 高俊
> > gaojun2048@gmail.com
> >
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by wu shaoj <ga...@apache.org>.
Yes, we should forcus on the job with high priority .
No need to satisfy all customer requirement which might be unreasonable
On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
Very detailed design.
1. About alert module, I agree with @许昌群.
Feedback from the community on the alert module, the alert module function
is relatively stable,
and the current notification function plus webhook whether it can meet the
needs.
2.It is not recommended that the UI be modularized, this will increase the
complexity of module development and debugging,
If there is more complicated UI interaction, it will cause trouble to the
development, Not only need to understand UI development,
but also understand modular configuration
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> Hi folks,
>
> I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> alert way like webhook(pls correct me if I'm wrong), which we can support
> as an official plugin.
>
> The difference between webhook and plugin is you should start a server to
> accept alert request in webhook, but in plugin way you don't need to do
> that.
>
> IMO, I prefer implement some official plugins in AlertServer package
> directly, as for others, we could put it in a plugin folder. I've
> implemented a plugin class loader in DS common package, which used to load
> plugin and avoid package conflict. Put official plugins directly in
> AlertServer package can use the common packages, and will cost less memory
> for class meta data.
>
> But put all plugin into a specific folder is fine too, because it looks
> clean.
>
> BTW, I think there is no conflict to support both webhook and plugin.
>
> Thanks,
> Han Gao
>
> ________________________________
> From: wu shaoj <ga...@apache.org>
> Sent: Thursday, June 4, 2020 15:13
> To: dev@dolphinscheduler.apache.org <de...@dolphinscheduler.apache.org>;
> 许昌群 <ch...@17zuoye.com>
> Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
>
> Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low
> priority I think.
>
> On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
>
> Hi all
>
> Regarding the alarm, I have some other ideas, please refer to it.
>
> At present, there are many types of message notification components,
> such as email/WeChat/SMS, or some notification components developed by the
> company.If it is not realistic to connect all the components, can we change
> the way of thinking and flip this process? We provide an interface for
> sending information, allowing users to interface with it.
>
> Described as follows:
> 1. Define the sending interface of the message in DS, such as sendMsg
> 2. Load the user-configured interface implementation, for example,
> http implementation is http://192.168.1.1:8080/sendMsg/email
> 3. Define the message sending format, {"tos":"softxcq@sina.com",
> "content":"taskA failed."}
> 4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={
> "tos":"softxcq@sina.com", "content":"taskA failed."}
>
> If there are other alarm methods, add an http interface
>
> Finally, you only need to provide several default alarm functions in
> DS to transfer other message notification functions to users.Although you
> will lose some ease of use out of the box, it will increase flexibility.
> You can implement your own system to ensure that the DS alarm logic is
> simple and easy to use.
>
>
> ________________________________
> 发件人: JUN GAO <ga...@gmail.com>
> 发送时间: 2020年6月3日 12:13:21
> 收件人: dev
> 主题: Re: [Poprosal] DolphinScheduler Plugin
>
> 1、Alert is very independent and common function, why should we to
> implement
> it as plugin?
>
> I think Alert should be divided into two parts. One is that the server
> collects various alarm information. The other is to pass these warning
> contents through optional methods (such as email, WeChat). I think the
> specific alarm method should not be implemented by the AlertServer
> server.
> It should be abstracted into an interface and implemented by a concrete
> plug-in implementation layer. If someone understands and is familiar
> with
> other alarm methods, he can implement other alarm plug-ins based on the
> plug-in design. If he wants, he can contribute this plug-in to us.
> Plug-inization is just a way to decouple projects.
>
> 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty? If
> it is a thirdparty, it may generally be managed by its own git, and it
> will
> not be placed in the dolphinscheduler related directory.
>
> Plug-inization is just a way to decouple projects. All Plugin is part
> of dolphinscheduler.
>
> And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> @daolidong We can FYI this.
>
> Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
>
> > Good proposal, but I have some question as following:
> > 1、Alert is very independent and common function, why should we to
> > implement it as plugin?
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty?
> > If it is a thirdparty, it may generally be managed by its own git,
> and it
> > will not be placed in the dolphinscheduler related directory.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) Committer
> > Xiaochun Liu 刘小春
> > liuxiaochun@apache.org
> > ---------------
> >
> >
> >
> > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > >
> > > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> >
>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by Hemin Wen <we...@apache.org>.
Very detailed design.
1. About alert module, I agree with @许昌群.
Feedback from the community on the alert module, the alert module function
is relatively stable,
and the current notification function plus webhook whether it can meet the
needs.
2.It is not recommended that the UI be modularized, this will increase the
complexity of module development and debugging,
If there is more complicated UI interaction, it will cause trouble to the
development, Not only need to understand UI development,
but also understand modular configuration
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> Hi folks,
>
> I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> alert way like webhook(pls correct me if I'm wrong), which we can support
> as an official plugin.
>
> The difference between webhook and plugin is you should start a server to
> accept alert request in webhook, but in plugin way you don't need to do
> that.
>
> IMO, I prefer implement some official plugins in AlertServer package
> directly, as for others, we could put it in a plugin folder. I've
> implemented a plugin class loader in DS common package, which used to load
> plugin and avoid package conflict. Put official plugins directly in
> AlertServer package can use the common packages, and will cost less memory
> for class meta data.
>
> But put all plugin into a specific folder is fine too, because it looks
> clean.
>
> BTW, I think there is no conflict to support both webhook and plugin.
>
> Thanks,
> Han Gao
>
> ________________________________
> From: wu shaoj <ga...@apache.org>
> Sent: Thursday, June 4, 2020 15:13
> To: dev@dolphinscheduler.apache.org <de...@dolphinscheduler.apache.org>;
> 许昌群 <ch...@17zuoye.com>
> Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
>
> Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low
> priority I think.
>
> On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
>
> Hi all
>
> Regarding the alarm, I have some other ideas, please refer to it.
>
> At present, there are many types of message notification components,
> such as email/WeChat/SMS, or some notification components developed by the
> company.If it is not realistic to connect all the components, can we change
> the way of thinking and flip this process? We provide an interface for
> sending information, allowing users to interface with it.
>
> Described as follows:
> 1. Define the sending interface of the message in DS, such as sendMsg
> 2. Load the user-configured interface implementation, for example,
> http implementation is http://192.168.1.1:8080/sendMsg/email
> 3. Define the message sending format, {"tos":"softxcq@sina.com",
> "content":"taskA failed."}
> 4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={
> "tos":"softxcq@sina.com", "content":"taskA failed."}
>
> If there are other alarm methods, add an http interface
>
> Finally, you only need to provide several default alarm functions in
> DS to transfer other message notification functions to users.Although you
> will lose some ease of use out of the box, it will increase flexibility.
> You can implement your own system to ensure that the DS alarm logic is
> simple and easy to use.
>
>
> ________________________________
> 发件人: JUN GAO <ga...@gmail.com>
> 发送时间: 2020年6月3日 12:13:21
> 收件人: dev
> 主题: Re: [Poprosal] DolphinScheduler Plugin
>
> 1、Alert is very independent and common function, why should we to
> implement
> it as plugin?
>
> I think Alert should be divided into two parts. One is that the server
> collects various alarm information. The other is to pass these warning
> contents through optional methods (such as email, WeChat). I think the
> specific alarm method should not be implemented by the AlertServer
> server.
> It should be abstracted into an interface and implemented by a concrete
> plug-in implementation layer. If someone understands and is familiar
> with
> other alarm methods, he can implement other alarm plug-ins based on the
> plug-in design. If he wants, he can contribute this plug-in to us.
> Plug-inization is just a way to decouple projects.
>
> 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty? If
> it is a thirdparty, it may generally be managed by its own git, and it
> will
> not be placed in the dolphinscheduler related directory.
>
> Plug-inization is just a way to decouple projects. All Plugin is part
> of dolphinscheduler.
>
> And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> @daolidong We can FYI this.
>
> Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
>
> > Good proposal, but I have some question as following:
> > 1、Alert is very independent and common function, why should we to
> > implement it as plugin?
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty?
> > If it is a thirdparty, it may generally be managed by its own git,
> and it
> > will not be placed in the dolphinscheduler related directory.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) Committer
> > Xiaochun Liu 刘小春
> > liuxiaochun@apache.org
> > ---------------
> >
> >
> >
> > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > >
> > > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> >
>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by Han Gao <dh...@hotmail.com>.
Hi folks,
I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an alert way like webhook(pls correct me if I'm wrong), which we can support as an official plugin.
The difference between webhook and plugin is you should start a server to accept alert request in webhook, but in plugin way you don't need to do that.
IMO, I prefer implement some official plugins in AlertServer package directly, as for others, we could put it in a plugin folder. I've implemented a plugin class loader in DS common package, which used to load plugin and avoid package conflict. Put official plugins directly in AlertServer package can use the common packages, and will cost less memory for class meta data.
But put all plugin into a specific folder is fine too, because it looks clean.
BTW, I think there is no conflict to support both webhook and plugin.
Thanks,
Han Gao
________________________________
From: wu shaoj <ga...@apache.org>
Sent: Thursday, June 4, 2020 15:13
To: dev@dolphinscheduler.apache.org <de...@dolphinscheduler.apache.org>; 许昌群 <ch...@17zuoye.com>
Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low priority I think.
On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
Hi all
Regarding the alarm, I have some other ideas, please refer to it.
At present, there are many types of message notification components, such as email/WeChat/SMS, or some notification components developed by the company.If it is not realistic to connect all the components, can we change the way of thinking and flip this process? We provide an interface for sending information, allowing users to interface with it.
Described as follows:
1. Define the sending interface of the message in DS, such as sendMsg
2. Load the user-configured interface implementation, for example, http implementation is http://192.168.1.1:8080/sendMsg/email
3. Define the message sending format, {"tos":"softxcq@sina.com", "content":"taskA failed."}
4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={"tos":"softxcq@sina.com", "content":"taskA failed."}
If there are other alarm methods, add an http interface
Finally, you only need to provide several default alarm functions in DS to transfer other message notification functions to users.Although you will lose some ease of use out of the box, it will increase flexibility. You can implement your own system to ensure that the DS alarm logic is simple and easy to use.
________________________________
发件人: JUN GAO <ga...@gmail.com>
发送时间: 2020年6月3日 12:13:21
收件人: dev
主题: Re: [Poprosal] DolphinScheduler Plugin
1、Alert is very independent and common function, why should we to implement
it as plugin?
I think Alert should be divided into two parts. One is that the server
collects various alarm information. The other is to pass these warning
contents through optional methods (such as email, WeChat). I think the
specific alarm method should not be implemented by the AlertServer server.
It should be abstracted into an interface and implemented by a concrete
plug-in implementation layer. If someone understands and is familiar with
other alarm methods, he can implement other alarm plug-ins based on the
plug-in design. If he wants, he can contribute this plug-in to us.
Plug-inization is just a way to decouple projects.
2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty? If
it is a thirdparty, it may generally be managed by its own git, and it will
not be placed in the dolphinscheduler related directory.
Plug-inization is just a way to decouple projects. All Plugin is part
of dolphinscheduler.
And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
@daolidong We can FYI this.
Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> Good proposal, but I have some question as following:
> 1、Alert is very independent and common function, why should we to
> implement it as plugin?
> 2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty?
> If it is a thirdparty, it may generally be managed by its own git, and it
> will not be placed in the dolphinscheduler related directory.
>
> And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org
> ---------------
>
>
>
> > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by wu shaoj <ga...@apache.org>.
Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low priority I think.
On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
Hi all
Regarding the alarm, I have some other ideas, please refer to it.
At present, there are many types of message notification components, such as email/WeChat/SMS, or some notification components developed by the company.If it is not realistic to connect all the components, can we change the way of thinking and flip this process? We provide an interface for sending information, allowing users to interface with it.
Described as follows:
1. Define the sending interface of the message in DS, such as sendMsg
2. Load the user-configured interface implementation, for example, http implementation is http://192.168.1.1:8080/sendMsg/email
3. Define the message sending format, {"tos":"softxcq@sina.com", "content":"taskA failed."}
4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={"tos":"softxcq@sina.com", "content":"taskA failed."}
If there are other alarm methods, add an http interface
Finally, you only need to provide several default alarm functions in DS to transfer other message notification functions to users.Although you will lose some ease of use out of the box, it will increase flexibility. You can implement your own system to ensure that the DS alarm logic is simple and easy to use.
________________________________
发件人: JUN GAO <ga...@gmail.com>
发送时间: 2020年6月3日 12:13:21
收件人: dev
主题: Re: [Poprosal] DolphinScheduler Plugin
1、Alert is very independent and common function, why should we to implement
it as plugin?
I think Alert should be divided into two parts. One is that the server
collects various alarm information. The other is to pass these warning
contents through optional methods (such as email, WeChat). I think the
specific alarm method should not be implemented by the AlertServer server.
It should be abstracted into an interface and implemented by a concrete
plug-in implementation layer. If someone understands and is familiar with
other alarm methods, he can implement other alarm plug-ins based on the
plug-in design. If he wants, he can contribute this plug-in to us.
Plug-inization is just a way to decouple projects.
2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty? If
it is a thirdparty, it may generally be managed by its own git, and it will
not be placed in the dolphinscheduler related directory.
Plug-inization is just a way to decouple projects. All Plugin is part
of dolphinscheduler.
And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
@daolidong We can FYI this.
Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> Good proposal, but I have some question as following:
> 1、Alert is very independent and common function, why should we to
> implement it as plugin?
> 2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty?
> If it is a thirdparty, it may generally be managed by its own git, and it
> will not be placed in the dolphinscheduler related directory.
>
> And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org
> ---------------
>
>
>
> > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
答复: [Poprosal] DolphinScheduler Plugin
Posted by 许昌群 <ch...@17zuoye.com>.
Hi all
Regarding the alarm, I have some other ideas, please refer to it.
At present, there are many types of message notification components, such as email/WeChat/SMS, or some notification components developed by the company.If it is not realistic to connect all the components, can we change the way of thinking and flip this process? We provide an interface for sending information, allowing users to interface with it.
Described as follows:
1. Define the sending interface of the message in DS, such as sendMsg
2. Load the user-configured interface implementation, for example, http implementation is http://192.168.1.1:8080/sendMsg/email
3. Define the message sending format, {"tos":"softxcq@sina.com", "content":"taskA failed."}
4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={"tos":"softxcq@sina.com", "content":"taskA failed."}
If there are other alarm methods, add an http interface
Finally, you only need to provide several default alarm functions in DS to transfer other message notification functions to users.Although you will lose some ease of use out of the box, it will increase flexibility. You can implement your own system to ensure that the DS alarm logic is simple and easy to use.
________________________________
发件人: JUN GAO <ga...@gmail.com>
发送时间: 2020年6月3日 12:13:21
收件人: dev
主题: Re: [Poprosal] DolphinScheduler Plugin
1、Alert is very independent and common function, why should we to implement
it as plugin?
I think Alert should be divided into two parts. One is that the server
collects various alarm information. The other is to pass these warning
contents through optional methods (such as email, WeChat). I think the
specific alarm method should not be implemented by the AlertServer server.
It should be abstracted into an interface and implemented by a concrete
plug-in implementation layer. If someone understands and is familiar with
other alarm methods, he can implement other alarm plug-ins based on the
plug-in design. If he wants, he can contribute this plug-in to us.
Plug-inization is just a way to decouple projects.
2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty? If
it is a thirdparty, it may generally be managed by its own git, and it will
not be placed in the dolphinscheduler related directory.
Plug-inization is just a way to decouple projects. All Plugin is part
of dolphinscheduler.
And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
@daolidong We can FYI this.
Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> Good proposal, but I have some question as following:
> 1、Alert is very independent and common function, why should we to
> implement it as plugin?
> 2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty?
> If it is a thirdparty, it may generally be managed by its own git, and it
> will not be placed in the dolphinscheduler related directory.
>
> And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org
> ---------------
>
>
>
> > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: [Poprosal] DolphinScheduler Plugin
Posted by JUN GAO <ga...@gmail.com>.
1、Alert is very independent and common function, why should we to implement
it as plugin?
I think Alert should be divided into two parts. One is that the server
collects various alarm information. The other is to pass these warning
contents through optional methods (such as email, WeChat). I think the
specific alarm method should not be implemented by the AlertServer server.
It should be abstracted into an interface and implemented by a concrete
plug-in implementation layer. If someone understands and is familiar with
other alarm methods, he can implement other alarm plug-ins based on the
plug-in design. If he wants, he can contribute this plug-in to us.
Plug-inization is just a way to decouple projects.
2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty? If
it is a thirdparty, it may generally be managed by its own git, and it will
not be placed in the dolphinscheduler related directory.
Plug-inization is just a way to decouple projects. All Plugin is part
of dolphinscheduler.
And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
@daolidong We can FYI this.
Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> Good proposal, but I have some question as following:
> 1、Alert is very independent and common function, why should we to
> implement it as plugin?
> 2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty?
> If it is a thirdparty, it may generally be managed by its own git, and it
> will not be placed in the dolphinscheduler related directory.
>
> And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org
> ---------------
>
>
>
> > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: [Poprosal] DolphinScheduler Plugin
Posted by Xiaochun Liu <li...@apache.org>.
Good proposal, but I have some question as following:
1、Alert is very independent and common function, why should we to implement it as plugin?
2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty? If it is a thirdparty, it may generally be managed by its own git, and it will not be placed in the dolphinscheduler related directory.
And we can use jsonschema to solve the dynamic ui, for example: alpacajs(http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
> 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
>
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>