You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by "xingchun.chen" <xi...@qq.com> on 2020/04/28 07:25:12 UTC
回复：[PROPOSAL][Feature] Support plugin in Alert service
It sounds very good, looking forward to your contribution
best wish!
------------------ 原始邮件 ------------------
发件人: "hgaol"<dhangao@hotmail.com>;
发送时间: 2020年4月28日(星期二) 下午3:11
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: [PROPOSAL][Feature] Support plugin in Alert service
Hi folks,
I think it will be useful for Alert service supporting the plugin management. I’ve create an issue in Github and described about it, here <https://github.com/apache/incubator-dolphinscheduler/issues/2553> is the link. Copy here for convenience.
> I think the plugin feature is needed by the customers, because different user may have different environment. For example, I want to use SMS to send alert information, but the SMS API is provided by some system we don't know before, and our DS has already deployed. In this case, I just want to implement the plugin code, and don't change the source code.
>
> Another benefits are developers can implement different plugin, like slack, wechat, telegram, github issue, and etc...
>
> Describe the solution you'd like
> A clear and concise description of what you want to happen.
>
> A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
>
> Currently, I've done some investigations, and if this feature is accepted, I think maybe I can implement it.
>
> Firstly, I'd better to adapt the current function, which is send mail and something like that.
>
> Then, we can add some more features, such as Alert handler instance, which is created to describe the alert information(email/slack/wechat/..., to whom, the title, content, and etc…).
>
> BTW, email and webhook are used frequently. I think they can be the default plugin, and written in the source code, instead of a jar file.
Thanks,
hgaol
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by leon bao <le...@apache.org>.
*> A better way is that user do not need to change the source code to
support new alert method. Instead, they just need to implement some
interface, and compliance with the convention, then package all(include
dependencies) in a single jar file, and put the jar file in some specific
place, such as $DS_HOME/plugins. After that, when start Alert service, the
plugins can be recognized and loaded. So user can choose the alert method
in the portal.*
yes, that is what we need, but i think we don't need a separate module for
alert plugin,
why don't we put the alert plug-in as an alert module? Is all plugins(alert
plugin/task plugin...) in one module is confusing?
hgaol <dh...@hotmail.com> 于2020年5月11日周一 下午7:00写道：
> Hi Gang,
>
> I agree with that making examples as a pure folder, thanks for your
> suggestion.
>
> For dolphinscheduler-plugin-api, do you mean it’s better to rename this
> module? I’m not clear about ‘all plugin function need to be implemented
> here’, do you mean we should put implemented plugin here?
>
> BTW, I’ve write a simple doc about how to implement an alert plugin.
> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
> <
> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
> >
>
> Also, I think for some frequently used plugins(email/webhook/...), we
> could just implement in alert service. Here is the former PR about it.
> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
> https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>
> Thanks for your careful review!
>
> Thanks,
> hgaol
>
> > On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
> >
> > Hi Gao,
> >
> > I agree with the following you said,If this is a plugin module,all the
> plugin function need to be implemented here.
> > Now the name is dolphinscheduler-plugin-api,I think we can change the
> module name, Such as `dolphinscheduler-plugin`.
> > > I think dolphinscheduler-plugin-api could not only support alert
> plugin, but also support java job task for future. That’s another topic.
> >
> > Yes,I think it just a folder,not a sub-module.
> > We can also look at other developers' comments on this.
> > >Do you mean the examples module should not be a sub-module in
> dolphinscheduler project pom file, just a pure folder?
> >
> > DolphinScheduler(Incubator) PPMC
> > Gang Li 李岗
> >
> > lgcareer2019@outlook.com <ma...@outlook.com><mailto:
> <ma...@outlook.com>lgcareer2019 <mailto:
> lgcareer2019@outlook.com>@outlook.com <ma...@outlook.com>>
> >
> > From: hgaol <ma...@hotmail.com>
> > Date: 2020-05-11 17:45
> > To: dev <ma...@dolphinscheduler.apache.org>; lgcareer2019 <mailto:
> lgcareer2019@outlook.com>
> > Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> > Hi Gang,
> >
> > Thanks for your suggestion.
> > 1.Whether the function of module dolphinscheduler-plugin-api can be
> puted in the alert module.
> > [hgaol] The customer should depend on this module to implement their own
> plugin. If put it into alert service, customer should put
> `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
> >
> > Also, I think dolphinscheduler-plugin-api could not only support alert
> plugin, but also support java job task for future. That’s another topic.
> >
> > 2. I think the examples module can be a folder.
> > [hgaol] Do you mean the examples module should not be a sub-module in
> dolphinscheduler project pom file, just a pure folder?
> >
> > Thanks,
> > hgaol
> >
> >> On May 11, 2020, at 11:51, lgcareer2019@outlook.com <mailto:
> lgcareer2019@outlook.com> wrote:
> >>
> >> Hi hgaol,
> >> Sorry for not replying in time.
> >> I saw you added the module dolphinscheduler-plugin-api and examples
> module,here are my two points want to discuss.
> >>
> >> 1、Whether the function of module dolphinscheduler-plugin-api can be
> puted in the alert module.
> >> 2、I think the examples module can be a folder.
> >>
> >>
> >>
> >>
> >>
> >>
> >> DolphinScheduler(Incubator) PPMC
> >> Gang Li 李岗
> >>
> >> lgcareer2019@outlook.com <ma...@outlook.com><mailto:
> lgcareer2019@outlook.com <ma...@outlook.com>>
> >>
> >> From: hgaol
> >> Date: 2020-04-29 23:37
> >> To: dev
> >> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> >> Hi folks,
> >>
> >> I’ve created a PR for this feature, here is the link
> >> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
> https://github.com/apache/incubator-dolphinscheduler/pull/2572> <
> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
> https://github.com/apache/incubator-dolphinscheduler/pull/2572>>
> >>
> >> Feel free to let me know if there is any suggestion.
> >>
> >> Thanks,
> >> Han Gao
> >>
> >>> On Apr 28, 2020, at 15:25, xingchun.chen <xingchun.chen@qq.com
> <ma...@qq.com>> wrote:
> >>>
> >>> It sounds very good, looking forward to your contribution
> >>>
> >>>
> >>> best wish!
> >>>
> >>>
> >>> ------------------ 原始邮件 ------------------
> >>> 发件人: "hgaol"<dhangao@hotmail.com <mailto:dhangao@hotmail.com
> >>;
> >>> 发送时间: 2020年4月28日(星期二) 下午3:11
> >>> 收件人: "dev"<dev@dolphinscheduler.apache.org <mailto:
> dev@dolphinscheduler.apache.org>>;
> >>>
> >>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
> >>>
> >>>
> >>>
> >>> Hi folks,
> >>>
> >>> I think it will be useful for Alert service supporting the plugin
> management. I’ve create an issue in Github and described about it, here <
> https://github.com/apache/incubator-dolphinscheduler/issues/2553> <
> https://github.com/apache/incubator-dolphinscheduler/issues/2553>>; is
> the link. Copy here for convenience.
> >>>
> >>> > I think the plugin feature is needed by the customers, because
> different user may have different environment. For example, I want to use
> SMS to send alert information, but the SMS API is provided by some system
> we don't know before, and our DS has already deployed. In this case, I just
> want to implement the plugin code, and don't change the source code.
> >>> >
> >>> > Another benefits are developers can implement different plugin,
> like slack, wechat, telegram, github issue, and etc...
> >>> >
> >>> > Describe the solution you'd like
> >>> > A clear and concise description of what you want to happen.
> >>> >
> >>> > A better way is that user do not need to change the source code
> to support new alert method. Instead, they just need to implement some
> interface, and compliance with the convention, then package all(include
> dependencies) in a single jar file, and put the jar file in some specific
> place, such as $DS_HOME/plugins. After that, when start Alert service, the
> plugins can be recognized and loaded. So user can choose the alert method
> in the portal.
> >>> >
> >>> > Currently, I've done some investigations, and if this feature is
> accepted, I think maybe I can implement it.
> >>> >
> >>> > Firstly, I'd better to adapt the current function, which is send
> mail and something like that.
> >>> >
> >>> > Then, we can add some more features, such as Alert handler
> instance, which is created to describe the alert
> information(email/slack/wechat/..., to whom, the title, content, and etc…).
> >>> >
> >>> > BTW, email and webhook are used frequently. I think they can be
> the default plugin, and written in the source code, instead of a jar file.
> >>>
> >>> Thanks,
> >>> hgaol
>
>
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by 高涵 <dh...@hotmail.com>.
Cool. Agree+1
Sent from my iPhone
> On May 14, 2020, at 22:27, zhangchunyang <18...@163.com> wrote:
>
> 赞
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2020-05-14 19:51:26, "lidong dai" <da...@gmail.com> wrote:
>> I agree with you, I think a dolphinscheduler-plugin module and sub module
>> under dolphinscheduler-plugin will be better
>>
>>
>> Best Regards
>> ---------------
>> DolphinScheduler(Incubator) PPMC
>> Lidong Dai 代立冬
>> dailidong66@gmail.com
>> ---------------
>>
>>
>> leon bao <le...@apache.org> 于2020年5月14日周四 下午7:18写道：
>>
>>> I think a common plugin module is necessary, but I don't recommend putting
>>> all the plugins in the root directory, which looks complicated.
>>> We can have ds-plugin module in root with many sub modules, such as
>>> alert-plugin / task-plugin /...
>>>
>>> hgaol <dh...@hotmail.com> 于2020年5月13日周三 下午8:24写道：
>>>
>>>> Hi @Jun Gao,
>>>>
>>>> I went through the link you sent, and I agree that if the job was plugged
>>>> in will be better. But I’m not quite familiar with DS job currently, so I
>>>> may not able to answer your question about how to add a ClickHouse
>>> plugin.
>>>>
>>>> I’ve written interfaces about Alert plugin to support different alert
>>>> type. If you want to implement alert plugin, you can refer to the PR
>>>> mentioned before.
>>>>
>>>> For job plugin, I think you can leverage the plugin design and enhance it
>>>> to support job plugin. You may need to define a JobProvider, a JobPlugin
>>>> interface, option with a AbstractJobPlugin.
>>>>
>>>> Currently, this PR is about Alert plugin. For job plugin or other plugin
>>>> design, I think it may be many topics and details to discuss.
>>>>
>>>>
>>>> Thanks,
>>>> Han Gao
>>>>
>>>>
>>>>> On May 13, 2020, at 10:19, JUN GAO <ga...@gmail.com> wrote:
>>>>>
>>>>> In addition to the job can be plugged in, is there any other content that
>>>>> can be plugged in? If so, we need to consider the structural design in
>>>>> advance, because this is a very important design transformation, we can
>>>>> spend more time to think more
>>>>>
>>>>> JUN GAO <ga...@gmail.com> 于2020年5月13日周三 上午10:11写道：
>>>>>
>>>>>> @hgaol <dh...@hotmail.com> There are some discuss about DS Plugin
>>>>>> . But this is an early design and can only be used as a reference . I
>>> hope
>>>>> it can help your design.
>>>>> https://github.com/apache/incubator-dolphinscheduler/issues/201
>>>>>
>>>>> I see the model `dolphinscheduler-plugin-api` . When I want to add a
>>>>> ClickHouse Job Plugin, should I create a new module called
>>>>> plugin-clickhouse-job directly in the root directory of the project
>>>>> (because I don’t know if only the job can do plugin)? Or we create a
>>> moudle
>>>>> called ds-job-plugin in the root directory, and then create a moudle
>>> called
>>>>> clickhouse-job under this moudle? Can you make a standardized design in
>>>>> this regard? Everyone can follow this design afterwards.
>>>>>
>>>>> @leon bao @Gang li Do you guys have any ideas ？
>>>>>
>>>>>
>>>>>
>>>>> hgaol <dh...@hotmail.com> 于2020年5月12日周二 下午9:33写道：
>>>>>
>>>>>> Hi @Gang Li, I agree with you that the plug-in module should be a
>>> common
>>>>>> module for all plug-in design. Currently I just create the SPI and
>>> plugin
>>>>>> interface for alert plug-in. If need more type for future, it’s very
>>> easy
>>>>>> to define other plugins.
>>>>>>
>>>>>> Currently the alert service can detect the plug-in jars and load it.
>>> But
>>>>>> I didn’t implement how to collaborate with front-end about the alert
>>> types
>>>>>> API and how many properties a plugin should have yet. For example, if
>>> we
>>>>>> add a webhook plugin, we should have the http method(form select),
>>> url(form
>>>>>> text), etc… So the plugin interface may change to achieve this goal.
>>>>>>
>>>>>> Hi @leon bao, as mentioned above, I agree to make the plugin module as
>>> a
>>>>>> common module for all plugin design, is that what you mean?
>>>>>>
>>>>>>> why don't we put the alert plug-in as an alert module?
>>>>>> I’m not clear about it, do you mean we could just put alert plugin in
>>>>>> alert module? I decide to put frequently used plugins in alert module,
>>> as
>>>>>> for others like SMS(which may has different api), make it as a plugin
>>> jar
>>>>>> file.
>>>>>>
>>>>>> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient.
>>>>>> Developers just need to implement the interfaces in SPI, which are in
>>>>>> `dolphinscheduler-plugin-api` module.
>>>>>>
>>>>>> Hi all, thanks for your careful review! Here is some link.
>>>>>>
>>>>>> PR about plugin management.
>>>>>> <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>>>>>> PR about how to develop a simple plugin.
>>>>>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593>
>>>>>> README about how to develop a simple plugin.
>>>>>> <
>>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
>>>>
>>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>> Thanks,
>>>>>> hgaol
>>>>>>
>>>>>> On May 12, 2020, at 18:57, JUN GAO <ga...@gmail.com> wrote:
>>>>>>
>>>>>> we can refer to presto plug-in, etc.
>>>>>> 1. Design the SPI and define the interface that the plugin needs to
>>>>>> implement
>>>>>> 2. All plugins create new models to implement the interface in SPI.
>>>>>> The benefits are as follows:
>>>>>> 1. For developers, they only need to care about the interface in SPI,
>>>>>> and do not need to care about other people's code, so it is easier to
>>> get
>>>>>> started developing plug-ins.
>>>>>> 2. For code reviewers, to improve the efficiency of review, independent
>>>>>> models will be more easily reviewed. If there is a problem with the
>>> code,
>>>>>> it is easy to roll back.
>>>>>> 3. For users, there are more flexible options. For example, some people
>>>>>> only want to use the A plug-in but not the B plug-in.
>>>>>>
>>>>>> Thank you !
>>>>>>
>>>>>> lgcareer2019@outlook.com <lg...@outlook.com> 于2020年5月12日周二
>>>>>> 下午6:31写道：
>>>>>>
>>>>>>> Hi Gao,
>>>>>>>
>>>>>>> May be I didn't express clearly,I think the dolphinscheduler-plugin
>>>>>>> module is a common module for all plugin design,
>>>>>>> It's not neccessarily to put the implement of a plugin in the
>>>>>>> dolphinscheduler-plugin module.
>>>>>>>
>>>>>>> I also agree with the following way your said,I see the pr you
>>> submited
>>>>>>> didn't match it now,Will you give a deep design about the following
>>> way?
>>>>>>>
>>>>>>>> A better way is that user do not need to change the source code to
>>>>>>> support new alert method. Instead, they just need to implement some
>>>>>>> interface, and compliance with the convention, then package
>>> all(include
>>>>>>> dependencies) in a single jar file, and put the jar file in some
>>> specific
>>>>>>> place, such as $DS_HOME/plugins. After that, when start Alert
>>> service, the
>>>>>>> plugins can be recognized and loaded. So user can choose the alert
>>> method
>>>>>>> in the portal.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> DolphinScheduler(Incubator) PPMC
>>>>>>> Gang Li 李岗
>>>>>>>
>>>>>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>>>>>
>>>>>>> From: hgaol
>>>>>>> Date: 2020-05-11 19:00
>>>>>>> To: lgcareer2019@outlook.com
>>>>>>> CC: dev
>>>>>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>>>>>>> Hi Gang,
>>>>>>>
>>>>>>> I agree with that making examples as a pure folder, thanks for your
>>>>>>> suggestion.
>>>>>>>
>>>>>>> For dolphinscheduler-plugin-api, do you mean it’s better to rename
>>> this
>>>>>>> module? I’m not clear about ‘all plugin function need to be
>>> implemented
>>>>>>> here’, do you mean we should put implemented plugin here?
>>>>>>>
>>>>>>> BTW, I’ve write a simple doc about how to implement an alert plugin.
>>>>>>>
>>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
>>>>>>>
>>>>>>> Also, I think for some frequently used plugins(email/webhook/...), we
>>>>>>> could just implement in alert service. Here is the former PR about it.
>>>>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572
>>>>>>>
>>>>>>> Thanks for your careful review!
>>>>>>>
>>>>>>> Thanks,
>>>>>>> hgaol
>>>>>>>
>>>>>>> On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
>>>>>>>
>>>>>>> Hi Gao,
>>>>>>>
>>>>>>> I agree with the following you said,If this is a plugin module,all the
>>>>>>> plugin function need to be implemented here.
>>>>>>> Now the name is dolphinscheduler-plugin-api,I think we can change the
>>>>>>> module name, Such as `dolphinscheduler-plugin`.
>>>>>>>> I think dolphinscheduler-plugin-api could not only support alert
>>>>>>> plugin, but also support java job task for future. That’s another
>>> topic.
>>>>>>>
>>>>>>> Yes,I think it just a folder,not a sub-module.
>>>>>>> We can also look at other developers' comments on this.
>>>>>>>> Do you mean the examples module should not be a sub-module in
>>>>>>> dolphinscheduler project pom file, just a pure folder?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> DolphinScheduler(Incubator) PPMC
>>>>>>> Gang Li 李岗
>>>>>>>
>>>>>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>>>>>
>>>>>>> From: hgaol
>>>>>>> Date: 2020-05-11 17:45
>>>>>>> To: dev; lgcareer2019
>>>>>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>>>>>>> Hi Gang,
>>>>>>>
>>>>>>> Thanks for your suggestion.
>>>>>>> 1.Whether the function of module dolphinscheduler-plugin-api can be
>>>>>>> puted in the alert module.
>>>>>>> [hgaol] The customer should depend on this module to implement their
>>>>>>> own plugin. If put it into alert service, customer should put
>>>>>>> `dolphinscheduler-alert` as dependence, which is too heavy for a
>>> plugin.
>>>>>>>
>>>>>>> Also, I think dolphinscheduler-plugin-api could not only support alert
>>>>>>> plugin, but also support java job task for future. That’s another
>>> topic.
>>>>>>>
>>>>>>> 2. I think the examples module can be a folder.
>>>>>>> [hgaol] Do you mean the examples module should not be a sub-module in
>>>>>>> dolphinscheduler project pom file, just a pure folder?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> hgaol
>>>>>>>
>>>>>>> On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
>>>>>>>
>>>>>>> Hi hgaol,
>>>>>>> Sorry for not replying in time.
>>>>>>> I saw you added the module dolphinscheduler-plugin-api and examples
>>>>>>> module,here are my two points want to discuss.
>>>>>>>
>>>>>>> 1、Whether the function of module dolphinscheduler-plugin-api can be
>>>>>>> puted in the alert module.
>>>>>>> 2、I think the examples module can be a folder.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> DolphinScheduler(Incubator) PPMC
>>>>>>> Gang Li 李岗
>>>>>>>
>>>>>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>>>>>
>>>>>>> From: hgaol
>>>>>>> Date: 2020-04-29 23:37
>>>>>>> To: dev
>>>>>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> I’ve created a PR for this feature, here is the link
>>>>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
>>>>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>>>>>>>
>>>>>>> Feel free to let me know if there is any suggestion.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Han Gao
>>>>>>>
>>>>>>> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com>
>>> wrote:
>>>>>>>
>>>>>>> It sounds very good, looking forward to your contribution
>>>>>>>
>>>>>>>
>>>>>>> best wish!
>>>>>>>
>>>>>>>
>>>>>>> ------------------ 原始邮件 ------------------
>>>>>>> 发件人: "hgaol"<dhangao@hotmail.com>;
>>>>>>> 发送时间: 2020年4月28日(星期二) 下午3:11
>>>>>>> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>>>>>>>
>>>>>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> I think it will be useful for Alert service supporting the plugin
>>>>>>> management. I’ve create an issue in Github and described about it,
>>> here <
>>>>>>> https://github.com/apache/incubator-dolphinscheduler/issues/2553>
>>>>>>> is the link. Copy here for convenience.
>>>>>>>
>>>>>>> > I think the plugin feature is needed by the customers, because
>>>>>>> different user may have different environment. For example, I want to
>>> use
>>>>>>> SMS to send alert information, but the SMS API is provided by some
>>> system
>>>>>>> we don't know before, and our DS has already deployed. In this case,
>>> I just
>>>>>>> want to implement the plugin code, and don't change the source code.
>>>>>>> >
>>>>>>> > Another benefits are developers can implement different plugin,
>>>>>>> like slack, wechat, telegram, github issue, and etc...
>>>>>>> >
>>>>>>> > Describe the solution you'd like
>>>>>>> > A clear and concise description of what you want to happen.
>>>>>>> >
>>>>>>> > A better way is that user do not need to change the source code
>>> to
>>>>>>> support new alert method. Instead, they just need to implement some
>>>>>>> interface, and compliance with the convention, then package
>>> all(include
>>>>>>> dependencies) in a single jar file, and put the jar file in some
>>> specific
>>>>>>> place, such as $DS_HOME/plugins. After that, when start Alert
>>> service, the
>>>>>>> plugins can be recognized and loaded. So user can choose the alert
>>> method
>>>>>>> in the portal.
>>>>>>> >
>>>>>>> > Currently, I've done some investigations, and if this feature is
>>>>>>> accepted, I think maybe I can implement it.
>>>>>>> >
>>>>>>> > Firstly, I'd better to adapt the current function, which is send
>>>>>>> mail and something like that.
>>>>>>> >
>>>>>>> > Then, we can add some more features, such as Alert handler
>>>>>>> instance, which is created to describe the alert
>>>>>>> information(email/slack/wechat/..., to whom, the title, content, and
>>> etc…).
>>>>>>> >
>>>>>>> > BTW, email and webhook are used frequently. I think they can be
>>>>>>> the default plugin, and written in the source code, instead of a jar
>>> file.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> hgaol
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> DolphinScheduler(Incubator) PPMC
>>>>>> Jun Gao 高俊
>>>>>> gaojun2048@gmail.com
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> DolphinScheduler(Incubator) PPMC
>>>>> Jun Gao 高俊
>>>>> gaojun2048@gmail.com
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> DolphinScheduler(Incubator) PPMC
>>>> Jun Gao 高俊
>>>> gaojun2048@gmail.com
>>>>
>>>>
>>>>
>>>
>>> --
>>> DolphinScheduler(Incubator) PPMC
>>> BaoLiang 鲍亮
>>> leonbao@apache.org
>>>
Re:Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by zhangchunyang <18...@163.com>.
赞
At 2020-05-14 19:51:26, "lidong dai" <da...@gmail.com> wrote:
>I agree with you, I think a dolphinscheduler-plugin module and sub module
>under dolphinscheduler-plugin will be better
>
>
>Best Regards
>---------------
>DolphinScheduler(Incubator) PPMC
>Lidong Dai 代立冬
>dailidong66@gmail.com
>---------------
>
>
>leon bao <le...@apache.org> 于2020年5月14日周四 下午7:18写道：
>
>> I think a common plugin module is necessary, but I don't recommend putting
>> all the plugins in the root directory, which looks complicated.
>> We can have ds-plugin module in root with many sub modules, such as
>> alert-plugin / task-plugin /...
>>
>> hgaol <dh...@hotmail.com> 于2020年5月13日周三 下午8:24写道：
>>
>> > Hi @Jun Gao,
>> >
>> > I went through the link you sent, and I agree that if the job was plugged
>> > in will be better. But I’m not quite familiar with DS job currently, so I
>> > may not able to answer your question about how to add a ClickHouse
>> plugin.
>> >
>> > I’ve written interfaces about Alert plugin to support different alert
>> > type. If you want to implement alert plugin, you can refer to the PR
>> > mentioned before.
>> >
>> > For job plugin, I think you can leverage the plugin design and enhance it
>> > to support job plugin. You may need to define a JobProvider, a JobPlugin
>> > interface, option with a AbstractJobPlugin.
>> >
>> > Currently, this PR is about Alert plugin. For job plugin or other plugin
>> > design, I think it may be many topics and details to discuss.
>> >
>> >
>> > Thanks,
>> > Han Gao
>> >
>> >
>> > On May 13, 2020, at 10:19, JUN GAO <ga...@gmail.com> wrote:
>> >
>> > In addition to the job can be plugged in, is there any other content that
>> > can be plugged in? If so, we need to consider the structural design in
>> > advance, because this is a very important design transformation, we can
>> > spend more time to think more
>> >
>> > JUN GAO <ga...@gmail.com> 于2020年5月13日周三 上午10:11写道：
>> >
>> >> @hgaol <dh...@hotmail.com> There are some discuss about DS Plugin
>> >> . But this is an early design and can only be used as a reference . I
>> hope
>> >> it can help your design.
>> >> https://github.com/apache/incubator-dolphinscheduler/issues/201
>> >>
>> >> I see the model `dolphinscheduler-plugin-api` . When I want to add a
>> >> ClickHouse Job Plugin, should I create a new module called
>> >> plugin-clickhouse-job directly in the root directory of the project
>> >> (because I don’t know if only the job can do plugin)? Or we create a
>> moudle
>> >> called ds-job-plugin in the root directory, and then create a moudle
>> called
>> >> clickhouse-job under this moudle? Can you make a standardized design in
>> >> this regard? Everyone can follow this design afterwards.
>> >>
>> >> @leon bao @Gang li Do you guys have any ideas ？
>> >>
>> >>
>> >>
>> >> hgaol <dh...@hotmail.com> 于2020年5月12日周二 下午9:33写道：
>> >>
>> >>> Hi @Gang Li, I agree with you that the plug-in module should be a
>> common
>> >>> module for all plug-in design. Currently I just create the SPI and
>> plugin
>> >>> interface for alert plug-in. If need more type for future, it’s very
>> easy
>> >>> to define other plugins.
>> >>>
>> >>> Currently the alert service can detect the plug-in jars and load it.
>> But
>> >>> I didn’t implement how to collaborate with front-end about the alert
>> types
>> >>> API and how many properties a plugin should have yet. For example, if
>> we
>> >>> add a webhook plugin, we should have the http method(form select),
>> url(form
>> >>> text), etc… So the plugin interface may change to achieve this goal.
>> >>>
>> >>> Hi @leon bao, as mentioned above, I agree to make the plugin module as
>> a
>> >>> common module for all plugin design, is that what you mean?
>> >>>
>> >>> > why don't we put the alert plug-in as an alert module?
>> >>> I’m not clear about it, do you mean we could just put alert plugin in
>> >>> alert module? I decide to put frequently used plugins in alert module,
>> as
>> >>> for others like SMS(which may has different api), make it as a plugin
>> jar
>> >>> file.
>> >>>
>> >>> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient.
>> >>> Developers just need to implement the interfaces in SPI, which are in
>> >>> `dolphinscheduler-plugin-api` module.
>> >>>
>> >>> Hi all, thanks for your careful review! Here is some link.
>> >>>
>> >>> PR about plugin management.
>> >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>> >>> PR about how to develop a simple plugin.
>> >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593>
>> >>> README about how to develop a simple plugin.
>> >>> <
>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
>> >
>> >>>
>> >>> Thanks in advance!
>> >>>
>> >>> Thanks,
>> >>> hgaol
>> >>>
>> >>> On May 12, 2020, at 18:57, JUN GAO <ga...@gmail.com> wrote:
>> >>>
>> >>> we can refer to presto plug-in, etc.
>> >>> 1. Design the SPI and define the interface that the plugin needs to
>> >>> implement
>> >>> 2. All plugins create new models to implement the interface in SPI.
>> >>> The benefits are as follows:
>> >>> 1. For developers, they only need to care about the interface in SPI,
>> >>> and do not need to care about other people's code, so it is easier to
>> get
>> >>> started developing plug-ins.
>> >>> 2. For code reviewers, to improve the efficiency of review, independent
>> >>> models will be more easily reviewed. If there is a problem with the
>> code,
>> >>> it is easy to roll back.
>> >>> 3. For users, there are more flexible options. For example, some people
>> >>> only want to use the A plug-in but not the B plug-in.
>> >>>
>> >>> Thank you !
>> >>>
>> >>> lgcareer2019@outlook.com <lg...@outlook.com> 于2020年5月12日周二
>> >>> 下午6:31写道：
>> >>>
>> >>>> Hi Gao,
>> >>>>
>> >>>> May be I didn't express clearly,I think the dolphinscheduler-plugin
>> >>>> module is a common module for all plugin design,
>> >>>> It's not neccessarily to put the implement of a plugin in the
>> >>>> dolphinscheduler-plugin module.
>> >>>>
>> >>>> I also agree with the following way your said,I see the pr you
>> submited
>> >>>> didn't match it now,Will you give a deep design about the following
>> way?
>> >>>>
>> >>>> > A better way is that user do not need to change the source code to
>> >>>> support new alert method. Instead, they just need to implement some
>> >>>> interface, and compliance with the convention, then package
>> all(include
>> >>>> dependencies) in a single jar file, and put the jar file in some
>> specific
>> >>>> place, such as $DS_HOME/plugins. After that, when start Alert
>> service, the
>> >>>> plugins can be recognized and loaded. So user can choose the alert
>> method
>> >>>> in the portal.
>> >>>>
>> >>>>
>> >>>>
>> >>>> DolphinScheduler(Incubator) PPMC
>> >>>> Gang Li 李岗
>> >>>>
>> >>>> lgcareer2019@outlook.com<ma...@outlook.com>
>> >>>>
>> >>>> From: hgaol
>> >>>> Date: 2020-05-11 19:00
>> >>>> To: lgcareer2019@outlook.com
>> >>>> CC: dev
>> >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> >>>> Hi Gang,
>> >>>>
>> >>>> I agree with that making examples as a pure folder, thanks for your
>> >>>> suggestion.
>> >>>>
>> >>>> For dolphinscheduler-plugin-api, do you mean it’s better to rename
>> this
>> >>>> module? I’m not clear about ‘all plugin function need to be
>> implemented
>> >>>> here’, do you mean we should put implemented plugin here?
>> >>>>
>> >>>> BTW, I’ve write a simple doc about how to implement an alert plugin.
>> >>>>
>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
>> >>>>
>> >>>> Also, I think for some frequently used plugins(email/webhook/...), we
>> >>>> could just implement in alert service. Here is the former PR about it.
>> >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572
>> >>>>
>> >>>> Thanks for your careful review!
>> >>>>
>> >>>> Thanks,
>> >>>> hgaol
>> >>>>
>> >>>> On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
>> >>>>
>> >>>> Hi Gao,
>> >>>>
>> >>>> I agree with the following you said,If this is a plugin module,all the
>> >>>> plugin function need to be implemented here.
>> >>>> Now the name is dolphinscheduler-plugin-api,I think we can change the
>> >>>> module name, Such as `dolphinscheduler-plugin`.
>> >>>> > I think dolphinscheduler-plugin-api could not only support alert
>> >>>> plugin, but also support java job task for future. That’s another
>> topic.
>> >>>>
>> >>>> Yes,I think it just a folder,not a sub-module.
>> >>>> We can also look at other developers' comments on this.
>> >>>> >Do you mean the examples module should not be a sub-module in
>> >>>> dolphinscheduler project pom file, just a pure folder?
>> >>>>
>> >>>>
>> >>>>
>> >>>> DolphinScheduler(Incubator) PPMC
>> >>>> Gang Li 李岗
>> >>>>
>> >>>> lgcareer2019@outlook.com<ma...@outlook.com>
>> >>>>
>> >>>> From: hgaol
>> >>>> Date: 2020-05-11 17:45
>> >>>> To: dev; lgcareer2019
>> >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> >>>> Hi Gang,
>> >>>>
>> >>>> Thanks for your suggestion.
>> >>>> 1.Whether the function of module dolphinscheduler-plugin-api can be
>> >>>> puted in the alert module.
>> >>>> [hgaol] The customer should depend on this module to implement their
>> >>>> own plugin. If put it into alert service, customer should put
>> >>>> `dolphinscheduler-alert` as dependence, which is too heavy for a
>> plugin.
>> >>>>
>> >>>> Also, I think dolphinscheduler-plugin-api could not only support alert
>> >>>> plugin, but also support java job task for future. That’s another
>> topic.
>> >>>>
>> >>>> 2. I think the examples module can be a folder.
>> >>>> [hgaol] Do you mean the examples module should not be a sub-module in
>> >>>> dolphinscheduler project pom file, just a pure folder?
>> >>>>
>> >>>> Thanks,
>> >>>> hgaol
>> >>>>
>> >>>> On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
>> >>>>
>> >>>> Hi hgaol,
>> >>>> Sorry for not replying in time.
>> >>>> I saw you added the module dolphinscheduler-plugin-api and examples
>> >>>> module,here are my two points want to discuss.
>> >>>>
>> >>>> 1、Whether the function of module dolphinscheduler-plugin-api can be
>> >>>> puted in the alert module.
>> >>>> 2、I think the examples module can be a folder.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> DolphinScheduler(Incubator) PPMC
>> >>>> Gang Li 李岗
>> >>>>
>> >>>> lgcareer2019@outlook.com<ma...@outlook.com>
>> >>>>
>> >>>> From: hgaol
>> >>>> Date: 2020-04-29 23:37
>> >>>> To: dev
>> >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> >>>> Hi folks,
>> >>>>
>> >>>> I’ve created a PR for this feature, here is the link
>> >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
>> >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>> >>>>
>> >>>> Feel free to let me know if there is any suggestion.
>> >>>>
>> >>>> Thanks,
>> >>>> Han Gao
>> >>>>
>> >>>> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com>
>> wrote:
>> >>>>
>> >>>> It sounds very good, looking forward to your contribution
>> >>>>
>> >>>>
>> >>>> best wish!
>> >>>>
>> >>>>
>> >>>> ------------------ 原始邮件 ------------------
>> >>>> 发件人: "hgaol"<dhangao@hotmail.com>;
>> >>>> 发送时间: 2020年4月28日(星期二) 下午3:11
>> >>>> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>> >>>>
>> >>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>> >>>>
>> >>>>
>> >>>>
>> >>>> Hi folks,
>> >>>>
>> >>>> I think it will be useful for Alert service supporting the plugin
>> >>>> management. I’ve create an issue in Github and described about it,
>> here <
>> >>>> https://github.com/apache/incubator-dolphinscheduler/issues/2553>
>> >>>> is the link. Copy here for convenience.
>> >>>>
>> >>>> > I think the plugin feature is needed by the customers, because
>> >>>> different user may have different environment. For example, I want to
>> use
>> >>>> SMS to send alert information, but the SMS API is provided by some
>> system
>> >>>> we don't know before, and our DS has already deployed. In this case,
>> I just
>> >>>> want to implement the plugin code, and don't change the source code.
>> >>>> >
>> >>>> > Another benefits are developers can implement different plugin,
>> >>>> like slack, wechat, telegram, github issue, and etc...
>> >>>> >
>> >>>> > Describe the solution you'd like
>> >>>> > A clear and concise description of what you want to happen.
>> >>>> >
>> >>>> > A better way is that user do not need to change the source code
>> to
>> >>>> support new alert method. Instead, they just need to implement some
>> >>>> interface, and compliance with the convention, then package
>> all(include
>> >>>> dependencies) in a single jar file, and put the jar file in some
>> specific
>> >>>> place, such as $DS_HOME/plugins. After that, when start Alert
>> service, the
>> >>>> plugins can be recognized and loaded. So user can choose the alert
>> method
>> >>>> in the portal.
>> >>>> >
>> >>>> > Currently, I've done some investigations, and if this feature is
>> >>>> accepted, I think maybe I can implement it.
>> >>>> >
>> >>>> > Firstly, I'd better to adapt the current function, which is send
>> >>>> mail and something like that.
>> >>>> >
>> >>>> > Then, we can add some more features, such as Alert handler
>> >>>> instance, which is created to describe the alert
>> >>>> information(email/slack/wechat/..., to whom, the title, content, and
>> etc…).
>> >>>> >
>> >>>> > BTW, email and webhook are used frequently. I think they can be
>> >>>> the default plugin, and written in the source code, instead of a jar
>> file.
>> >>>>
>> >>>> Thanks,
>> >>>> hgaol
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>>
>> >>> DolphinScheduler(Incubator) PPMC
>> >>> Jun Gao 高俊
>> >>> gaojun2048@gmail.com
>> >>>
>> >>>
>> >>>
>> >>
>> >> --
>> >>
>> >> DolphinScheduler(Incubator) PPMC
>> >> Jun Gao 高俊
>> >> gaojun2048@gmail.com
>> >>
>> >>
>> >
>> > --
>> >
>> > DolphinScheduler(Incubator) PPMC
>> > Jun Gao 高俊
>> > gaojun2048@gmail.com
>> >
>> >
>> >
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by lidong dai <da...@gmail.com>.
I agree with you, I think a dolphinscheduler-plugin module and sub module
under dolphinscheduler-plugin will be better
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
leon bao <le...@apache.org> 于2020年5月14日周四 下午7:18写道：
> I think a common plugin module is necessary, but I don't recommend putting
> all the plugins in the root directory, which looks complicated.
> We can have ds-plugin module in root with many sub modules, such as
> alert-plugin / task-plugin /...
>
> hgaol <dh...@hotmail.com> 于2020年5月13日周三 下午8:24写道：
>
> > Hi @Jun Gao,
> >
> > I went through the link you sent, and I agree that if the job was plugged
> > in will be better. But I’m not quite familiar with DS job currently, so I
> > may not able to answer your question about how to add a ClickHouse
> plugin.
> >
> > I’ve written interfaces about Alert plugin to support different alert
> > type. If you want to implement alert plugin, you can refer to the PR
> > mentioned before.
> >
> > For job plugin, I think you can leverage the plugin design and enhance it
> > to support job plugin. You may need to define a JobProvider, a JobPlugin
> > interface, option with a AbstractJobPlugin.
> >
> > Currently, this PR is about Alert plugin. For job plugin or other plugin
> > design, I think it may be many topics and details to discuss.
> >
> >
> > Thanks,
> > Han Gao
> >
> >
> > On May 13, 2020, at 10:19, JUN GAO <ga...@gmail.com> wrote:
> >
> > In addition to the job can be plugged in, is there any other content that
> > can be plugged in? If so, we need to consider the structural design in
> > advance, because this is a very important design transformation, we can
> > spend more time to think more
> >
> > JUN GAO <ga...@gmail.com> 于2020年5月13日周三 上午10:11写道：
> >
> >> @hgaol <dh...@hotmail.com> There are some discuss about DS Plugin
> >> . But this is an early design and can only be used as a reference . I
> hope
> >> it can help your design.
> >> https://github.com/apache/incubator-dolphinscheduler/issues/201
> >>
> >> I see the model `dolphinscheduler-plugin-api` . When I want to add a
> >> ClickHouse Job Plugin, should I create a new module called
> >> plugin-clickhouse-job directly in the root directory of the project
> >> (because I don’t know if only the job can do plugin)? Or we create a
> moudle
> >> called ds-job-plugin in the root directory, and then create a moudle
> called
> >> clickhouse-job under this moudle? Can you make a standardized design in
> >> this regard? Everyone can follow this design afterwards.
> >>
> >> @leon bao @Gang li Do you guys have any ideas ？
> >>
> >>
> >>
> >> hgaol <dh...@hotmail.com> 于2020年5月12日周二 下午9:33写道：
> >>
> >>> Hi @Gang Li, I agree with you that the plug-in module should be a
> common
> >>> module for all plug-in design. Currently I just create the SPI and
> plugin
> >>> interface for alert plug-in. If need more type for future, it’s very
> easy
> >>> to define other plugins.
> >>>
> >>> Currently the alert service can detect the plug-in jars and load it.
> But
> >>> I didn’t implement how to collaborate with front-end about the alert
> types
> >>> API and how many properties a plugin should have yet. For example, if
> we
> >>> add a webhook plugin, we should have the http method(form select),
> url(form
> >>> text), etc… So the plugin interface may change to achieve this goal.
> >>>
> >>> Hi @leon bao, as mentioned above, I agree to make the plugin module as
> a
> >>> common module for all plugin design, is that what you mean?
> >>>
> >>> > why don't we put the alert plug-in as an alert module?
> >>> I’m not clear about it, do you mean we could just put alert plugin in
> >>> alert module? I decide to put frequently used plugins in alert module,
> as
> >>> for others like SMS(which may has different api), make it as a plugin
> jar
> >>> file.
> >>>
> >>> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient.
> >>> Developers just need to implement the interfaces in SPI, which are in
> >>> `dolphinscheduler-plugin-api` module.
> >>>
> >>> Hi all, thanks for your careful review! Here is some link.
> >>>
> >>> PR about plugin management.
> >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
> >>> PR about how to develop a simple plugin.
> >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593>
> >>> README about how to develop a simple plugin.
> >>> <
> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
> >
> >>>
> >>> Thanks in advance!
> >>>
> >>> Thanks,
> >>> hgaol
> >>>
> >>> On May 12, 2020, at 18:57, JUN GAO <ga...@gmail.com> wrote:
> >>>
> >>> we can refer to presto plug-in, etc.
> >>> 1. Design the SPI and define the interface that the plugin needs to
> >>> implement
> >>> 2. All plugins create new models to implement the interface in SPI.
> >>> The benefits are as follows:
> >>> 1. For developers, they only need to care about the interface in SPI,
> >>> and do not need to care about other people's code, so it is easier to
> get
> >>> started developing plug-ins.
> >>> 2. For code reviewers, to improve the efficiency of review, independent
> >>> models will be more easily reviewed. If there is a problem with the
> code,
> >>> it is easy to roll back.
> >>> 3. For users, there are more flexible options. For example, some people
> >>> only want to use the A plug-in but not the B plug-in.
> >>>
> >>> Thank you !
> >>>
> >>> lgcareer2019@outlook.com <lg...@outlook.com> 于2020年5月12日周二
> >>> 下午6:31写道：
> >>>
> >>>> Hi Gao,
> >>>>
> >>>> May be I didn't express clearly,I think the dolphinscheduler-plugin
> >>>> module is a common module for all plugin design,
> >>>> It's not neccessarily to put the implement of a plugin in the
> >>>> dolphinscheduler-plugin module.
> >>>>
> >>>> I also agree with the following way your said,I see the pr you
> submited
> >>>> didn't match it now,Will you give a deep design about the following
> way?
> >>>>
> >>>> > A better way is that user do not need to change the source code to
> >>>> support new alert method. Instead, they just need to implement some
> >>>> interface, and compliance with the convention, then package
> all(include
> >>>> dependencies) in a single jar file, and put the jar file in some
> specific
> >>>> place, such as $DS_HOME/plugins. After that, when start Alert
> service, the
> >>>> plugins can be recognized and loaded. So user can choose the alert
> method
> >>>> in the portal.
> >>>>
> >>>>
> >>>>
> >>>> DolphinScheduler(Incubator) PPMC
> >>>> Gang Li 李岗
> >>>>
> >>>> lgcareer2019@outlook.com<ma...@outlook.com>
> >>>>
> >>>> From: hgaol
> >>>> Date: 2020-05-11 19:00
> >>>> To: lgcareer2019@outlook.com
> >>>> CC: dev
> >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> >>>> Hi Gang,
> >>>>
> >>>> I agree with that making examples as a pure folder, thanks for your
> >>>> suggestion.
> >>>>
> >>>> For dolphinscheduler-plugin-api, do you mean it’s better to rename
> this
> >>>> module? I’m not clear about ‘all plugin function need to be
> implemented
> >>>> here’, do you mean we should put implemented plugin here?
> >>>>
> >>>> BTW, I’ve write a simple doc about how to implement an alert plugin.
> >>>>
> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
> >>>>
> >>>> Also, I think for some frequently used plugins(email/webhook/...), we
> >>>> could just implement in alert service. Here is the former PR about it.
> >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572
> >>>>
> >>>> Thanks for your careful review!
> >>>>
> >>>> Thanks,
> >>>> hgaol
> >>>>
> >>>> On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
> >>>>
> >>>> Hi Gao,
> >>>>
> >>>> I agree with the following you said,If this is a plugin module,all the
> >>>> plugin function need to be implemented here.
> >>>> Now the name is dolphinscheduler-plugin-api,I think we can change the
> >>>> module name, Such as `dolphinscheduler-plugin`.
> >>>> > I think dolphinscheduler-plugin-api could not only support alert
> >>>> plugin, but also support java job task for future. That’s another
> topic.
> >>>>
> >>>> Yes,I think it just a folder,not a sub-module.
> >>>> We can also look at other developers' comments on this.
> >>>> >Do you mean the examples module should not be a sub-module in
> >>>> dolphinscheduler project pom file, just a pure folder?
> >>>>
> >>>>
> >>>>
> >>>> DolphinScheduler(Incubator) PPMC
> >>>> Gang Li 李岗
> >>>>
> >>>> lgcareer2019@outlook.com<ma...@outlook.com>
> >>>>
> >>>> From: hgaol
> >>>> Date: 2020-05-11 17:45
> >>>> To: dev; lgcareer2019
> >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> >>>> Hi Gang,
> >>>>
> >>>> Thanks for your suggestion.
> >>>> 1.Whether the function of module dolphinscheduler-plugin-api can be
> >>>> puted in the alert module.
> >>>> [hgaol] The customer should depend on this module to implement their
> >>>> own plugin. If put it into alert service, customer should put
> >>>> `dolphinscheduler-alert` as dependence, which is too heavy for a
> plugin.
> >>>>
> >>>> Also, I think dolphinscheduler-plugin-api could not only support alert
> >>>> plugin, but also support java job task for future. That’s another
> topic.
> >>>>
> >>>> 2. I think the examples module can be a folder.
> >>>> [hgaol] Do you mean the examples module should not be a sub-module in
> >>>> dolphinscheduler project pom file, just a pure folder?
> >>>>
> >>>> Thanks,
> >>>> hgaol
> >>>>
> >>>> On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
> >>>>
> >>>> Hi hgaol,
> >>>> Sorry for not replying in time.
> >>>> I saw you added the module dolphinscheduler-plugin-api and examples
> >>>> module,here are my two points want to discuss.
> >>>>
> >>>> 1、Whether the function of module dolphinscheduler-plugin-api can be
> >>>> puted in the alert module.
> >>>> 2、I think the examples module can be a folder.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> DolphinScheduler(Incubator) PPMC
> >>>> Gang Li 李岗
> >>>>
> >>>> lgcareer2019@outlook.com<ma...@outlook.com>
> >>>>
> >>>> From: hgaol
> >>>> Date: 2020-04-29 23:37
> >>>> To: dev
> >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> >>>> Hi folks,
> >>>>
> >>>> I’ve created a PR for this feature, here is the link
> >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
> >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572>
> >>>>
> >>>> Feel free to let me know if there is any suggestion.
> >>>>
> >>>> Thanks,
> >>>> Han Gao
> >>>>
> >>>> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com>
> wrote:
> >>>>
> >>>> It sounds very good, looking forward to your contribution
> >>>>
> >>>>
> >>>> best wish!
> >>>>
> >>>>
> >>>> ------------------ 原始邮件 ------------------
> >>>> 发件人: "hgaol"<dhangao@hotmail.com>;
> >>>> 发送时间: 2020年4月28日(星期二) 下午3:11
> >>>> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
> >>>>
> >>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
> >>>>
> >>>>
> >>>>
> >>>> Hi folks,
> >>>>
> >>>> I think it will be useful for Alert service supporting the plugin
> >>>> management. I’ve create an issue in Github and described about it,
> here <
> >>>> https://github.com/apache/incubator-dolphinscheduler/issues/2553>
> >>>> is the link. Copy here for convenience.
> >>>>
> >>>> > I think the plugin feature is needed by the customers, because
> >>>> different user may have different environment. For example, I want to
> use
> >>>> SMS to send alert information, but the SMS API is provided by some
> system
> >>>> we don't know before, and our DS has already deployed. In this case,
> I just
> >>>> want to implement the plugin code, and don't change the source code.
> >>>> >
> >>>> > Another benefits are developers can implement different plugin,
> >>>> like slack, wechat, telegram, github issue, and etc...
> >>>> >
> >>>> > Describe the solution you'd like
> >>>> > A clear and concise description of what you want to happen.
> >>>> >
> >>>> > A better way is that user do not need to change the source code
> to
> >>>> support new alert method. Instead, they just need to implement some
> >>>> interface, and compliance with the convention, then package
> all(include
> >>>> dependencies) in a single jar file, and put the jar file in some
> specific
> >>>> place, such as $DS_HOME/plugins. After that, when start Alert
> service, the
> >>>> plugins can be recognized and loaded. So user can choose the alert
> method
> >>>> in the portal.
> >>>> >
> >>>> > Currently, I've done some investigations, and if this feature is
> >>>> accepted, I think maybe I can implement it.
> >>>> >
> >>>> > Firstly, I'd better to adapt the current function, which is send
> >>>> mail and something like that.
> >>>> >
> >>>> > Then, we can add some more features, such as Alert handler
> >>>> instance, which is created to describe the alert
> >>>> information(email/slack/wechat/..., to whom, the title, content, and
> etc…).
> >>>> >
> >>>> > BTW, email and webhook are used frequently. I think they can be
> >>>> the default plugin, and written in the source code, instead of a jar
> file.
> >>>>
> >>>> Thanks,
> >>>> hgaol
> >>>>
> >>>>
> >>>
> >>> --
> >>>
> >>> DolphinScheduler(Incubator) PPMC
> >>> Jun Gao 高俊
> >>> gaojun2048@gmail.com
> >>>
> >>>
> >>>
> >>
> >> --
> >>
> >> DolphinScheduler(Incubator) PPMC
> >> Jun Gao 高俊
> >> gaojun2048@gmail.com
> >>
> >>
> >
> > --
> >
> > DolphinScheduler(Incubator) PPMC
> > Jun Gao 高俊
> > gaojun2048@gmail.com
> >
> >
> >
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by leon bao <le...@apache.org>.
I think a common plugin module is necessary, but I don't recommend putting
all the plugins in the root directory, which looks complicated.
We can have ds-plugin module in root with many sub modules, such as
alert-plugin / task-plugin /...
hgaol <dh...@hotmail.com> 于2020年5月13日周三 下午8:24写道：
> Hi @Jun Gao,
>
> I went through the link you sent, and I agree that if the job was plugged
> in will be better. But I’m not quite familiar with DS job currently, so I
> may not able to answer your question about how to add a ClickHouse plugin.
>
> I’ve written interfaces about Alert plugin to support different alert
> type. If you want to implement alert plugin, you can refer to the PR
> mentioned before.
>
> For job plugin, I think you can leverage the plugin design and enhance it
> to support job plugin. You may need to define a JobProvider, a JobPlugin
> interface, option with a AbstractJobPlugin.
>
> Currently, this PR is about Alert plugin. For job plugin or other plugin
> design, I think it may be many topics and details to discuss.
>
>
> Thanks,
> Han Gao
>
>
> On May 13, 2020, at 10:19, JUN GAO <ga...@gmail.com> wrote:
>
> In addition to the job can be plugged in, is there any other content that
> can be plugged in? If so, we need to consider the structural design in
> advance, because this is a very important design transformation, we can
> spend more time to think more
>
> JUN GAO <ga...@gmail.com> 于2020年5月13日周三 上午10:11写道：
>
>> @hgaol <dh...@hotmail.com> There are some discuss about DS Plugin
>> . But this is an early design and can only be used as a reference . I hope
>> it can help your design.
>> https://github.com/apache/incubator-dolphinscheduler/issues/201
>>
>> I see the model `dolphinscheduler-plugin-api` . When I want to add a
>> ClickHouse Job Plugin, should I create a new module called
>> plugin-clickhouse-job directly in the root directory of the project
>> (because I don’t know if only the job can do plugin)? Or we create a moudle
>> called ds-job-plugin in the root directory, and then create a moudle called
>> clickhouse-job under this moudle? Can you make a standardized design in
>> this regard? Everyone can follow this design afterwards.
>>
>> @leon bao @Gang li Do you guys have any ideas ？
>>
>>
>>
>> hgaol <dh...@hotmail.com> 于2020年5月12日周二 下午9:33写道：
>>
>>> Hi @Gang Li, I agree with you that the plug-in module should be a common
>>> module for all plug-in design. Currently I just create the SPI and plugin
>>> interface for alert plug-in. If need more type for future, it’s very easy
>>> to define other plugins.
>>>
>>> Currently the alert service can detect the plug-in jars and load it. But
>>> I didn’t implement how to collaborate with front-end about the alert types
>>> API and how many properties a plugin should have yet. For example, if we
>>> add a webhook plugin, we should have the http method(form select), url(form
>>> text), etc… So the plugin interface may change to achieve this goal.
>>>
>>> Hi @leon bao, as mentioned above, I agree to make the plugin module as a
>>> common module for all plugin design, is that what you mean?
>>>
>>> > why don't we put the alert plug-in as an alert module?
>>> I’m not clear about it, do you mean we could just put alert plugin in
>>> alert module? I decide to put frequently used plugins in alert module, as
>>> for others like SMS(which may has different api), make it as a plugin jar
>>> file.
>>>
>>> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient.
>>> Developers just need to implement the interfaces in SPI, which are in
>>> `dolphinscheduler-plugin-api` module.
>>>
>>> Hi all, thanks for your careful review! Here is some link.
>>>
>>> PR about plugin management.
>>> <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>>> PR about how to develop a simple plugin.
>>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593>
>>> README about how to develop a simple plugin.
>>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07>
>>>
>>> Thanks in advance!
>>>
>>> Thanks,
>>> hgaol
>>>
>>> On May 12, 2020, at 18:57, JUN GAO <ga...@gmail.com> wrote:
>>>
>>> we can refer to presto plug-in, etc.
>>> 1. Design the SPI and define the interface that the plugin needs to
>>> implement
>>> 2. All plugins create new models to implement the interface in SPI.
>>> The benefits are as follows:
>>> 1. For developers, they only need to care about the interface in SPI,
>>> and do not need to care about other people's code, so it is easier to get
>>> started developing plug-ins.
>>> 2. For code reviewers, to improve the efficiency of review, independent
>>> models will be more easily reviewed. If there is a problem with the code,
>>> it is easy to roll back.
>>> 3. For users, there are more flexible options. For example, some people
>>> only want to use the A plug-in but not the B plug-in.
>>>
>>> Thank you !
>>>
>>> lgcareer2019@outlook.com <lg...@outlook.com> 于2020年5月12日周二
>>> 下午6:31写道：
>>>
>>>> Hi Gao,
>>>>
>>>> May be I didn't express clearly,I think the dolphinscheduler-plugin
>>>> module is a common module for all plugin design,
>>>> It's not neccessarily to put the implement of a plugin in the
>>>> dolphinscheduler-plugin module.
>>>>
>>>> I also agree with the following way your said,I see the pr you submited
>>>> didn't match it now,Will you give a deep design about the following way?
>>>>
>>>> > A better way is that user do not need to change the source code to
>>>> support new alert method. Instead, they just need to implement some
>>>> interface, and compliance with the convention, then package all(include
>>>> dependencies) in a single jar file, and put the jar file in some specific
>>>> place, such as $DS_HOME/plugins. After that, when start Alert service, the
>>>> plugins can be recognized and loaded. So user can choose the alert method
>>>> in the portal.
>>>>
>>>>
>>>>
>>>> DolphinScheduler(Incubator) PPMC
>>>> Gang Li 李岗
>>>>
>>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>>
>>>> From: hgaol
>>>> Date: 2020-05-11 19:00
>>>> To: lgcareer2019@outlook.com
>>>> CC: dev
>>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>>>> Hi Gang,
>>>>
>>>> I agree with that making examples as a pure folder, thanks for your
>>>> suggestion.
>>>>
>>>> For dolphinscheduler-plugin-api, do you mean it’s better to rename this
>>>> module? I’m not clear about ‘all plugin function need to be implemented
>>>> here’, do you mean we should put implemented plugin here?
>>>>
>>>> BTW, I’ve write a simple doc about how to implement an alert plugin.
>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
>>>>
>>>> Also, I think for some frequently used plugins(email/webhook/...), we
>>>> could just implement in alert service. Here is the former PR about it.
>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572
>>>>
>>>> Thanks for your careful review!
>>>>
>>>> Thanks,
>>>> hgaol
>>>>
>>>> On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
>>>>
>>>> Hi Gao,
>>>>
>>>> I agree with the following you said,If this is a plugin module,all the
>>>> plugin function need to be implemented here.
>>>> Now the name is dolphinscheduler-plugin-api,I think we can change the
>>>> module name, Such as `dolphinscheduler-plugin`.
>>>> > I think dolphinscheduler-plugin-api could not only support alert
>>>> plugin, but also support java job task for future. That’s another topic.
>>>>
>>>> Yes,I think it just a folder,not a sub-module.
>>>> We can also look at other developers' comments on this.
>>>> >Do you mean the examples module should not be a sub-module in
>>>> dolphinscheduler project pom file, just a pure folder?
>>>>
>>>>
>>>>
>>>> DolphinScheduler(Incubator) PPMC
>>>> Gang Li 李岗
>>>>
>>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>>
>>>> From: hgaol
>>>> Date: 2020-05-11 17:45
>>>> To: dev; lgcareer2019
>>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>>>> Hi Gang,
>>>>
>>>> Thanks for your suggestion.
>>>> 1.Whether the function of module dolphinscheduler-plugin-api can be
>>>> puted in the alert module.
>>>> [hgaol] The customer should depend on this module to implement their
>>>> own plugin. If put it into alert service, customer should put
>>>> `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
>>>>
>>>> Also, I think dolphinscheduler-plugin-api could not only support alert
>>>> plugin, but also support java job task for future. That’s another topic.
>>>>
>>>> 2. I think the examples module can be a folder.
>>>> [hgaol] Do you mean the examples module should not be a sub-module in
>>>> dolphinscheduler project pom file, just a pure folder?
>>>>
>>>> Thanks,
>>>> hgaol
>>>>
>>>> On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
>>>>
>>>> Hi hgaol,
>>>> Sorry for not replying in time.
>>>> I saw you added the module dolphinscheduler-plugin-api and examples
>>>> module,here are my two points want to discuss.
>>>>
>>>> 1、Whether the function of module dolphinscheduler-plugin-api can be
>>>> puted in the alert module.
>>>> 2、I think the examples module can be a folder.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> DolphinScheduler(Incubator) PPMC
>>>> Gang Li 李岗
>>>>
>>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>>
>>>> From: hgaol
>>>> Date: 2020-04-29 23:37
>>>> To: dev
>>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>>>> Hi folks,
>>>>
>>>> I’ve created a PR for this feature, here is the link
>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
>>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>>>>
>>>> Feel free to let me know if there is any suggestion.
>>>>
>>>> Thanks,
>>>> Han Gao
>>>>
>>>> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com> wrote:
>>>>
>>>> It sounds very good, looking forward to your contribution
>>>>
>>>>
>>>> best wish!
>>>>
>>>>
>>>> ------------------ 原始邮件 ------------------
>>>> 发件人: "hgaol"<dhangao@hotmail.com>;
>>>> 发送时间: 2020年4月28日(星期二) 下午3:11
>>>> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>>>>
>>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>>>>
>>>>
>>>>
>>>> Hi folks,
>>>>
>>>> I think it will be useful for Alert service supporting the plugin
>>>> management. I’ve create an issue in Github and described about it, here <
>>>> https://github.com/apache/incubator-dolphinscheduler/issues/2553>
>>>> is the link. Copy here for convenience.
>>>>
>>>> > I think the plugin feature is needed by the customers, because
>>>> different user may have different environment. For example, I want to use
>>>> SMS to send alert information, but the SMS API is provided by some system
>>>> we don't know before, and our DS has already deployed. In this case, I just
>>>> want to implement the plugin code, and don't change the source code.
>>>> >
>>>> > Another benefits are developers can implement different plugin,
>>>> like slack, wechat, telegram, github issue, and etc...
>>>> >
>>>> > Describe the solution you'd like
>>>> > A clear and concise description of what you want to happen.
>>>> >
>>>> > A better way is that user do not need to change the source code to
>>>> support new alert method. Instead, they just need to implement some
>>>> interface, and compliance with the convention, then package all(include
>>>> dependencies) in a single jar file, and put the jar file in some specific
>>>> place, such as $DS_HOME/plugins. After that, when start Alert service, the
>>>> plugins can be recognized and loaded. So user can choose the alert method
>>>> in the portal.
>>>> >
>>>> > Currently, I've done some investigations, and if this feature is
>>>> accepted, I think maybe I can implement it.
>>>> >
>>>> > Firstly, I'd better to adapt the current function, which is send
>>>> mail and something like that.
>>>> >
>>>> > Then, we can add some more features, such as Alert handler
>>>> instance, which is created to describe the alert
>>>> information(email/slack/wechat/..., to whom, the title, content, and etc…).
>>>> >
>>>> > BTW, email and webhook are used frequently. I think they can be
>>>> the default plugin, and written in the source code, instead of a jar file.
>>>>
>>>> Thanks,
>>>> hgaol
>>>>
>>>>
>>>
>>> --
>>>
>>> DolphinScheduler(Incubator) PPMC
>>> Jun Gao 高俊
>>> gaojun2048@gmail.com
>>>
>>>
>>>
>>
>> --
>>
>> DolphinScheduler(Incubator) PPMC
>> Jun Gao 高俊
>> gaojun2048@gmail.com
>>
>>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
>
>
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by hgaol <dh...@hotmail.com>.
Hi @Jun Gao,
I went through the link you sent, and I agree that if the job was plugged in will be better. But I’m not quite familiar with DS job currently, so I may not able to answer your question about how to add a ClickHouse plugin.
I’ve written interfaces about Alert plugin to support different alert type. If you want to implement alert plugin, you can refer to the PR mentioned before.
For job plugin, I think you can leverage the plugin design and enhance it to support job plugin. You may need to define a JobProvider, a JobPlugin interface, option with a AbstractJobPlugin.
Currently, this PR is about Alert plugin. For job plugin or other plugin design, I think it may be many topics and details to discuss.
Thanks,
Han Gao
> On May 13, 2020, at 10:19, JUN GAO <ga...@gmail.com> wrote:
>
> In addition to the job can be plugged in, is there any other content that can be plugged in? If so, we need to consider the structural design in advance, because this is a very important design transformation, we can spend more time to think more
>
> JUN GAO <gaojun2048@gmail.com <ma...@gmail.com>> 于2020年5月13日周三 上午10:11写道：
> @hgaol <ma...@hotmail.com> There are some discuss about DS Plugin . But this is an early design and can only be used as a reference . I hope it can help your design. https://github.com/apache/incubator-dolphinscheduler/issues/201 <https://github.com/apache/incubator-dolphinscheduler/issues/201>
>
> I see the model `dolphinscheduler-plugin-api` . When I want to add a ClickHouse Job Plugin, should I create a new module called plugin-clickhouse-job directly in the root directory of the project (because I don’t know if only the job can do plugin)? Or we create a moudle called ds-job-plugin in the root directory, and then create a moudle called clickhouse-job under this moudle? Can you make a standardized design in this regard? Everyone can follow this design afterwards.
>
> @leon bao @Gang li Do you guys have any ideas ？
>
>
>
> hgaol <dhangao@hotmail.com <ma...@hotmail.com>> 于2020年5月12日周二 下午9:33写道：
> Hi @Gang Li, I agree with you that the plug-in module should be a common module for all plug-in design. Currently I just create the SPI and plugin interface for alert plug-in. If need more type for future, it’s very easy to define other plugins.
>
> Currently the alert service can detect the plug-in jars and load it. But I didn’t implement how to collaborate with front-end about the alert types API and how many properties a plugin should have yet. For example, if we add a webhook plugin, we should have the http method(form select), url(form text), etc… So the plugin interface may change to achieve this goal.
>
> Hi @leon bao, as mentioned above, I agree to make the plugin module as a common module for all plugin design, is that what you mean?
>
> > why don't we put the alert plug-in as an alert module?
> I’m not clear about it, do you mean we could just put alert plugin in alert module? I decide to put frequently used plugins in alert module, as for others like SMS(which may has different api), make it as a plugin jar file.
>
> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient. Developers just need to implement the interfaces in SPI, which are in `dolphinscheduler-plugin-api` module.
>
> Hi all, thanks for your careful review! Here is some link.
>
> PR about plugin management. <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
> PR about how to develop a simple plugin. <https://github.com/apache/incubator-dolphinscheduler/pull/2593>
> README about how to develop a simple plugin. <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07>
>
> Thanks in advance!
>
> Thanks,
> hgaol
>
>> On May 12, 2020, at 18:57, JUN GAO <gaojun2048@gmail.com <ma...@gmail.com>> wrote:
>>
>> we can refer to presto plug-in, etc.
>> 1. Design the SPI and define the interface that the plugin needs to implement
>> 2. All plugins create new models to implement the interface in SPI.
>> The benefits are as follows:
>> 1. For developers, they only need to care about the interface in SPI, and do not need to care about other people's code, so it is easier to get started developing plug-ins.
>> 2. For code reviewers, to improve the efficiency of review, independent models will be more easily reviewed. If there is a problem with the code, it is easy to roll back.
>> 3. For users, there are more flexible options. For example, some people only want to use the A plug-in but not the B plug-in.
>>
>> Thank you !
>>
>> lgcareer2019@outlook.com <ma...@outlook.com> <lgcareer2019@outlook.com <ma...@outlook.com>> 于2020年5月12日周二 下午6:31写道：
>> Hi Gao,
>>
>> May be I didn't express clearly,I think the dolphinscheduler-plugin module is a common module for all plugin design,
>> It's not neccessarily to put the implement of a plugin in the dolphinscheduler-plugin module.
>>
>> I also agree with the following way your said,I see the pr you submited didn't match it now,Will you give a deep design about the following way?
>>
>> > A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
>>
>>
>>
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com <ma...@outlook.com><mailto:lgcareer2019@outlook.com <ma...@outlook.com>>
>>
>> From: hgaol
>> Date: 2020-05-11 19:00
>> To: lgcareer2019@outlook.com <ma...@outlook.com>
>> CC: dev
>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> Hi Gang,
>>
>> I agree with that making examples as a pure folder, thanks for your suggestion.
>>
>> For dolphinscheduler-plugin-api, do you mean it’s better to rename this module? I’m not clear about ‘all plugin function need to be implemented here’, do you mean we should put implemented plugin here?
>>
>> BTW, I’ve write a simple doc about how to implement an alert plugin.https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07>
>>
>> Also, I think for some frequently used plugins(email/webhook/...), we could just implement in alert service. Here is the former PR about it. https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>>
>> Thanks for your careful review!
>>
>> Thanks,
>> hgaol
>>
>> On May 11, 2020, at 18:09, lgcareer2019@outlook.com <ma...@outlook.com> wrote:
>>
>> Hi Gao,
>>
>> I agree with the following you said,If this is a plugin module,all the plugin function need to be implemented here.
>> Now the name is dolphinscheduler-plugin-api,I think we can change the module name, Such as `dolphinscheduler-plugin`.
>> > I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
>>
>> Yes,I think it just a folder,not a sub-module.
>> We can also look at other developers' comments on this.
>> >Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
>>
>>
>>
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com <ma...@outlook.com><mailto:lgcareer2019@outlook.com <ma...@outlook.com>>
>>
>> From: hgaol
>> Date: 2020-05-11 17:45
>> To: dev; lgcareer2019
>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> Hi Gang,
>>
>> Thanks for your suggestion.
>> 1.Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
>> [hgaol] The customer should depend on this module to implement their own plugin. If put it into alert service, customer should put `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
>>
>> Also, I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
>>
>> 2. I think the examples module can be a folder.
>> [hgaol] Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
>>
>> Thanks,
>> hgaol
>>
>> On May 11, 2020, at 11:51, lgcareer2019@outlook.com <ma...@outlook.com> wrote:
>>
>> Hi hgaol,
>> Sorry for not replying in time.
>> I saw you added the module dolphinscheduler-plugin-api and examples module,here are my two points want to discuss.
>>
>> 1、Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
>> 2、I think the examples module can be a folder.
>>
>>
>>
>>
>>
>>
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com <ma...@outlook.com><mailto:lgcareer2019@outlook.com <ma...@outlook.com>>
>>
>> From: hgaol
>> Date: 2020-04-29 23:37
>> To: dev
>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> Hi folks,
>>
>> I’ve created a PR for this feature, here is the link
>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572> <https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>>
>>
>> Feel free to let me know if there is any suggestion.
>>
>> Thanks,
>> Han Gao
>>
>> On Apr 28, 2020, at 15:25, xingchun.chen <xingchun.chen@qq.com <ma...@qq.com>> wrote:
>>
>> It sounds very good, looking forward to your contribution
>>
>>
>> best wish!
>>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人: "hgaol"<dhangao@hotmail.com <ma...@hotmail.com>>;
>> 发送时间: 2020年4月28日(星期二) 下午3:11
>> 收件人: "dev"<dev@dolphinscheduler.apache.org <ma...@dolphinscheduler.apache.org>>;
>>
>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>>
>>
>>
>> Hi folks,
>>
>> I think it will be useful for Alert service supporting the plugin management. I’ve create an issue in Github and described about it, here <https://github.com/apache/incubator-dolphinscheduler/issues/2553> <https://github.com/apache/incubator-dolphinscheduler/issues/2553>>; is the link. Copy here for convenience.
>>
>> > I think the plugin feature is needed by the customers, because different user may have different environment. For example, I want to use SMS to send alert information, but the SMS API is provided by some system we don't know before, and our DS has already deployed. In this case, I just want to implement the plugin code, and don't change the source code.
>> >
>> > Another benefits are developers can implement different plugin, like slack, wechat, telegram, github issue, and etc...
>> >
>> > Describe the solution you'd like
>> > A clear and concise description of what you want to happen.
>> >
>> > A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
>> >
>> > Currently, I've done some investigations, and if this feature is accepted, I think maybe I can implement it.
>> >
>> > Firstly, I'd better to adapt the current function, which is send mail and something like that.
>> >
>> > Then, we can add some more features, such as Alert handler instance, which is created to describe the alert information(email/slack/wechat/..., to whom, the title, content, and etc…).
>> >
>> > BTW, email and webhook are used frequently. I think they can be the default plugin, and written in the source code, instead of a jar file.
>>
>> Thanks,
>> hgaol
>>
>>
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> Jun Gao 高俊
>>
>> gaojun2048@gmail.com <ma...@gmail.com>
>>
>
>
>
> --
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
>
> gaojun2048@gmail.com <ma...@gmail.com>
>
>
>
> --
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
>
> gaojun2048@gmail.com <ma...@gmail.com>
>
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by JUN GAO <ga...@gmail.com>.
In addition to the job can be plugged in, is there any other content that
can be plugged in? If so, we need to consider the structural design in
advance, because this is a very important design transformation, we can
spend more time to think more
JUN GAO <ga...@gmail.com> 于2020年5月13日周三 上午10:11写道：
> @hgaol <dh...@hotmail.com> There are some discuss about DS Plugin
> . But this is an early design and can only be used as a reference . I hope
> it can help your design.
> https://github.com/apache/incubator-dolphinscheduler/issues/201
>
> I see the model `dolphinscheduler-plugin-api` . When I want to add a
> ClickHouse Job Plugin, should I create a new module called
> plugin-clickhouse-job directly in the root directory of the project
> (because I don’t know if only the job can do plugin)? Or we create a moudle
> called ds-job-plugin in the root directory, and then create a moudle called
> clickhouse-job under this moudle? Can you make a standardized design in
> this regard? Everyone can follow this design afterwards.
>
> @leon bao @Gang li Do you guys have any ideas ？
>
>
>
> hgaol <dh...@hotmail.com> 于2020年5月12日周二 下午9:33写道：
>
>> Hi @Gang Li, I agree with you that the plug-in module should be a common
>> module for all plug-in design. Currently I just create the SPI and plugin
>> interface for alert plug-in. If need more type for future, it’s very easy
>> to define other plugins.
>>
>> Currently the alert service can detect the plug-in jars and load it. But
>> I didn’t implement how to collaborate with front-end about the alert types
>> API and how many properties a plugin should have yet. For example, if we
>> add a webhook plugin, we should have the http method(form select), url(form
>> text), etc… So the plugin interface may change to achieve this goal.
>>
>> Hi @leon bao, as mentioned above, I agree to make the plugin module as a
>> common module for all plugin design, is that what you mean?
>>
>> > why don't we put the alert plug-in as an alert module?
>> I’m not clear about it, do you mean we could just put alert plugin in
>> alert module? I decide to put frequently used plugins in alert module, as
>> for others like SMS(which may has different api), make it as a plugin jar
>> file.
>>
>> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient.
>> Developers just need to implement the interfaces in SPI, which are in
>> `dolphinscheduler-plugin-api` module.
>>
>> Hi all, thanks for your careful review! Here is some link.
>>
>> PR about plugin management.
>> <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>> PR about how to develop a simple plugin.
>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593>
>> README about how to develop a simple plugin.
>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07>
>>
>> Thanks in advance!
>>
>> Thanks,
>> hgaol
>>
>> On May 12, 2020, at 18:57, JUN GAO <ga...@gmail.com> wrote:
>>
>> we can refer to presto plug-in, etc.
>> 1. Design the SPI and define the interface that the plugin needs to
>> implement
>> 2. All plugins create new models to implement the interface in SPI.
>> The benefits are as follows:
>> 1. For developers, they only need to care about the interface in SPI, and
>> do not need to care about other people's code, so it is easier to get
>> started developing plug-ins.
>> 2. For code reviewers, to improve the efficiency of review, independent
>> models will be more easily reviewed. If there is a problem with the code,
>> it is easy to roll back.
>> 3. For users, there are more flexible options. For example, some people
>> only want to use the A plug-in but not the B plug-in.
>>
>> Thank you !
>>
>> lgcareer2019@outlook.com <lg...@outlook.com> 于2020年5月12日周二
>> 下午6:31写道：
>>
>>> Hi Gao,
>>>
>>> May be I didn't express clearly,I think the dolphinscheduler-plugin
>>> module is a common module for all plugin design,
>>> It's not neccessarily to put the implement of a plugin in the
>>> dolphinscheduler-plugin module.
>>>
>>> I also agree with the following way your said,I see the pr you submited
>>> didn't match it now,Will you give a deep design about the following way?
>>>
>>> > A better way is that user do not need to change the source code to
>>> support new alert method. Instead, they just need to implement some
>>> interface, and compliance with the convention, then package all(include
>>> dependencies) in a single jar file, and put the jar file in some specific
>>> place, such as $DS_HOME/plugins. After that, when start Alert service, the
>>> plugins can be recognized and loaded. So user can choose the alert method
>>> in the portal.
>>>
>>>
>>>
>>> DolphinScheduler(Incubator) PPMC
>>> Gang Li 李岗
>>>
>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>
>>> From: hgaol
>>> Date: 2020-05-11 19:00
>>> To: lgcareer2019@outlook.com
>>> CC: dev
>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>>> Hi Gang,
>>>
>>> I agree with that making examples as a pure folder, thanks for your
>>> suggestion.
>>>
>>> For dolphinscheduler-plugin-api, do you mean it’s better to rename this
>>> module? I’m not clear about ‘all plugin function need to be implemented
>>> here’, do you mean we should put implemented plugin here?
>>>
>>> BTW, I’ve write a simple doc about how to implement an alert plugin.
>>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
>>>
>>> Also, I think for some frequently used plugins(email/webhook/...), we
>>> could just implement in alert service. Here is the former PR about it.
>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572
>>>
>>> Thanks for your careful review!
>>>
>>> Thanks,
>>> hgaol
>>>
>>> On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
>>>
>>> Hi Gao,
>>>
>>> I agree with the following you said,If this is a plugin module,all the
>>> plugin function need to be implemented here.
>>> Now the name is dolphinscheduler-plugin-api,I think we can change the
>>> module name, Such as `dolphinscheduler-plugin`.
>>> > I think dolphinscheduler-plugin-api could not only support alert
>>> plugin, but also support java job task for future. That’s another topic.
>>>
>>> Yes,I think it just a folder,not a sub-module.
>>> We can also look at other developers' comments on this.
>>> >Do you mean the examples module should not be a sub-module in
>>> dolphinscheduler project pom file, just a pure folder?
>>>
>>>
>>>
>>> DolphinScheduler(Incubator) PPMC
>>> Gang Li 李岗
>>>
>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>
>>> From: hgaol
>>> Date: 2020-05-11 17:45
>>> To: dev; lgcareer2019
>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>>> Hi Gang,
>>>
>>> Thanks for your suggestion.
>>> 1.Whether the function of module dolphinscheduler-plugin-api can be
>>> puted in the alert module.
>>> [hgaol] The customer should depend on this module to implement their own
>>> plugin. If put it into alert service, customer should put
>>> `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
>>>
>>> Also, I think dolphinscheduler-plugin-api could not only support alert
>>> plugin, but also support java job task for future. That’s another topic.
>>>
>>> 2. I think the examples module can be a folder.
>>> [hgaol] Do you mean the examples module should not be a sub-module in
>>> dolphinscheduler project pom file, just a pure folder?
>>>
>>> Thanks,
>>> hgaol
>>>
>>> On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
>>>
>>> Hi hgaol,
>>> Sorry for not replying in time.
>>> I saw you added the module dolphinscheduler-plugin-api and examples
>>> module,here are my two points want to discuss.
>>>
>>> 1、Whether the function of module dolphinscheduler-plugin-api can be
>>> puted in the alert module.
>>> 2、I think the examples module can be a folder.
>>>
>>>
>>>
>>>
>>>
>>>
>>> DolphinScheduler(Incubator) PPMC
>>> Gang Li 李岗
>>>
>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>
>>> From: hgaol
>>> Date: 2020-04-29 23:37
>>> To: dev
>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>>> Hi folks,
>>>
>>> I’ve created a PR for this feature, here is the link
>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>>>
>>> Feel free to let me know if there is any suggestion.
>>>
>>> Thanks,
>>> Han Gao
>>>
>>> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com> wrote:
>>>
>>> It sounds very good, looking forward to your contribution
>>>
>>>
>>> best wish!
>>>
>>>
>>> ------------------ 原始邮件 ------------------
>>> 发件人: "hgaol"<dhangao@hotmail.com>;
>>> 发送时间: 2020年4月28日(星期二) 下午3:11
>>> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>>>
>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>>>
>>>
>>>
>>> Hi folks,
>>>
>>> I think it will be useful for Alert service supporting the plugin
>>> management. I’ve create an issue in Github and described about it, here <
>>> https://github.com/apache/incubator-dolphinscheduler/issues/2553> is
>>> the link. Copy here for convenience.
>>>
>>> > I think the plugin feature is needed by the customers, because
>>> different user may have different environment. For example, I want to use
>>> SMS to send alert information, but the SMS API is provided by some system
>>> we don't know before, and our DS has already deployed. In this case, I just
>>> want to implement the plugin code, and don't change the source code.
>>> >
>>> > Another benefits are developers can implement different plugin,
>>> like slack, wechat, telegram, github issue, and etc...
>>> >
>>> > Describe the solution you'd like
>>> > A clear and concise description of what you want to happen.
>>> >
>>> > A better way is that user do not need to change the source code to
>>> support new alert method. Instead, they just need to implement some
>>> interface, and compliance with the convention, then package all(include
>>> dependencies) in a single jar file, and put the jar file in some specific
>>> place, such as $DS_HOME/plugins. After that, when start Alert service, the
>>> plugins can be recognized and loaded. So user can choose the alert method
>>> in the portal.
>>> >
>>> > Currently, I've done some investigations, and if this feature is
>>> accepted, I think maybe I can implement it.
>>> >
>>> > Firstly, I'd better to adapt the current function, which is send
>>> mail and something like that.
>>> >
>>> > Then, we can add some more features, such as Alert handler
>>> instance, which is created to describe the alert
>>> information(email/slack/wechat/..., to whom, the title, content, and etc…).
>>> >
>>> > BTW, email and webhook are used frequently. I think they can be the
>>> default plugin, and written in the source code, instead of a jar file.
>>>
>>> Thanks,
>>> hgaol
>>>
>>>
>>
>> --
>>
>> DolphinScheduler(Incubator) PPMC
>> Jun Gao 高俊
>> gaojun2048@gmail.com
>>
>>
>>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by JUN GAO <ga...@gmail.com>.
@hgaol <dh...@hotmail.com> There are some discuss about DS Plugin . But
this is an early design and can only be used as a reference . I hope it
can help your design.
https://github.com/apache/incubator-dolphinscheduler/issues/201
I see the model `dolphinscheduler-plugin-api` . When I want to add a
ClickHouse Job Plugin, should I create a new module called
plugin-clickhouse-job directly in the root directory of the project
(because I don’t know if only the job can do plugin)? Or we create a moudle
called ds-job-plugin in the root directory, and then create a moudle called
clickhouse-job under this moudle? Can you make a standardized design in
this regard? Everyone can follow this design afterwards.
@leon bao @Gang li Do you guys have any ideas ？
hgaol <dh...@hotmail.com> 于2020年5月12日周二 下午9:33写道：
> Hi @Gang Li, I agree with you that the plug-in module should be a common
> module for all plug-in design. Currently I just create the SPI and plugin
> interface for alert plug-in. If need more type for future, it’s very easy
> to define other plugins.
>
> Currently the alert service can detect the plug-in jars and load it. But I
> didn’t implement how to collaborate with front-end about the alert types
> API and how many properties a plugin should have yet. For example, if we
> add a webhook plugin, we should have the http method(form select), url(form
> text), etc… So the plugin interface may change to achieve this goal.
>
> Hi @leon bao, as mentioned above, I agree to make the plugin module as a
> common module for all plugin design, is that what you mean?
>
> > why don't we put the alert plug-in as an alert module?
> I’m not clear about it, do you mean we could just put alert plugin in
> alert module? I decide to put frequently used plugins in alert module, as
> for others like SMS(which may has different api), make it as a plugin jar
> file.
>
> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient.
> Developers just need to implement the interfaces in SPI, which are in
> `dolphinscheduler-plugin-api` module.
>
> Hi all, thanks for your careful review! Here is some link.
>
> PR about plugin management.
> <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
> PR about how to develop a simple plugin.
> <https://github.com/apache/incubator-dolphinscheduler/pull/2593>
> README about how to develop a simple plugin.
> <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07>
>
> Thanks in advance!
>
> Thanks,
> hgaol
>
> On May 12, 2020, at 18:57, JUN GAO <ga...@gmail.com> wrote:
>
> we can refer to presto plug-in, etc.
> 1. Design the SPI and define the interface that the plugin needs to
> implement
> 2. All plugins create new models to implement the interface in SPI.
> The benefits are as follows:
> 1. For developers, they only need to care about the interface in SPI, and
> do not need to care about other people's code, so it is easier to get
> started developing plug-ins.
> 2. For code reviewers, to improve the efficiency of review, independent
> models will be more easily reviewed. If there is a problem with the code,
> it is easy to roll back.
> 3. For users, there are more flexible options. For example, some people
> only want to use the A plug-in but not the B plug-in.
>
> Thank you !
>
> lgcareer2019@outlook.com <lg...@outlook.com> 于2020年5月12日周二
> 下午6:31写道：
>
>> Hi Gao,
>>
>> May be I didn't express clearly,I think the dolphinscheduler-plugin
>> module is a common module for all plugin design,
>> It's not neccessarily to put the implement of a plugin in the
>> dolphinscheduler-plugin module.
>>
>> I also agree with the following way your said,I see the pr you submited
>> didn't match it now,Will you give a deep design about the following way?
>>
>> > A better way is that user do not need to change the source code to
>> support new alert method. Instead, they just need to implement some
>> interface, and compliance with the convention, then package all(include
>> dependencies) in a single jar file, and put the jar file in some specific
>> place, such as $DS_HOME/plugins. After that, when start Alert service, the
>> plugins can be recognized and loaded. So user can choose the alert method
>> in the portal.
>>
>>
>>
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com<ma...@outlook.com>
>>
>> From: hgaol
>> Date: 2020-05-11 19:00
>> To: lgcareer2019@outlook.com
>> CC: dev
>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> Hi Gang,
>>
>> I agree with that making examples as a pure folder, thanks for your
>> suggestion.
>>
>> For dolphinscheduler-plugin-api, do you mean it’s better to rename this
>> module? I’m not clear about ‘all plugin function need to be implemented
>> here’, do you mean we should put implemented plugin here?
>>
>> BTW, I’ve write a simple doc about how to implement an alert plugin.
>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
>>
>> Also, I think for some frequently used plugins(email/webhook/...), we
>> could just implement in alert service. Here is the former PR about it.
>> https://github.com/apache/incubator-dolphinscheduler/pull/2572
>>
>> Thanks for your careful review!
>>
>> Thanks,
>> hgaol
>>
>> On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
>>
>> Hi Gao,
>>
>> I agree with the following you said,If this is a plugin module,all the
>> plugin function need to be implemented here.
>> Now the name is dolphinscheduler-plugin-api,I think we can change the
>> module name, Such as `dolphinscheduler-plugin`.
>> > I think dolphinscheduler-plugin-api could not only support alert
>> plugin, but also support java job task for future. That’s another topic.
>>
>> Yes,I think it just a folder,not a sub-module.
>> We can also look at other developers' comments on this.
>> >Do you mean the examples module should not be a sub-module in
>> dolphinscheduler project pom file, just a pure folder?
>>
>>
>>
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com<ma...@outlook.com>
>>
>> From: hgaol
>> Date: 2020-05-11 17:45
>> To: dev; lgcareer2019
>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> Hi Gang,
>>
>> Thanks for your suggestion.
>> 1.Whether the function of module dolphinscheduler-plugin-api can be puted
>> in the alert module.
>> [hgaol] The customer should depend on this module to implement their own
>> plugin. If put it into alert service, customer should put
>> `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
>>
>> Also, I think dolphinscheduler-plugin-api could not only support alert
>> plugin, but also support java job task for future. That’s another topic.
>>
>> 2. I think the examples module can be a folder.
>> [hgaol] Do you mean the examples module should not be a sub-module in
>> dolphinscheduler project pom file, just a pure folder?
>>
>> Thanks,
>> hgaol
>>
>> On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
>>
>> Hi hgaol,
>> Sorry for not replying in time.
>> I saw you added the module dolphinscheduler-plugin-api and examples
>> module,here are my two points want to discuss.
>>
>> 1、Whether the function of module dolphinscheduler-plugin-api can be puted
>> in the alert module.
>> 2、I think the examples module can be a folder.
>>
>>
>>
>>
>>
>>
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com<ma...@outlook.com>
>>
>> From: hgaol
>> Date: 2020-04-29 23:37
>> To: dev
>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> Hi folks,
>>
>> I’ve created a PR for this feature, here is the link
>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
>> https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>>
>> Feel free to let me know if there is any suggestion.
>>
>> Thanks,
>> Han Gao
>>
>> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com> wrote:
>>
>> It sounds very good, looking forward to your contribution
>>
>>
>> best wish!
>>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人: "hgaol"<dhangao@hotmail.com>;
>> 发送时间: 2020年4月28日(星期二) 下午3:11
>> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>>
>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>>
>>
>>
>> Hi folks,
>>
>> I think it will be useful for Alert service supporting the plugin
>> management. I’ve create an issue in Github and described about it, here <
>> https://github.com/apache/incubator-dolphinscheduler/issues/2553> is
>> the link. Copy here for convenience.
>>
>> > I think the plugin feature is needed by the customers, because
>> different user may have different environment. For example, I want to use
>> SMS to send alert information, but the SMS API is provided by some system
>> we don't know before, and our DS has already deployed. In this case, I just
>> want to implement the plugin code, and don't change the source code.
>> >
>> > Another benefits are developers can implement different plugin, like
>> slack, wechat, telegram, github issue, and etc...
>> >
>> > Describe the solution you'd like
>> > A clear and concise description of what you want to happen.
>> >
>> > A better way is that user do not need to change the source code to
>> support new alert method. Instead, they just need to implement some
>> interface, and compliance with the convention, then package all(include
>> dependencies) in a single jar file, and put the jar file in some specific
>> place, such as $DS_HOME/plugins. After that, when start Alert service, the
>> plugins can be recognized and loaded. So user can choose the alert method
>> in the portal.
>> >
>> > Currently, I've done some investigations, and if this feature is
>> accepted, I think maybe I can implement it.
>> >
>> > Firstly, I'd better to adapt the current function, which is send
>> mail and something like that.
>> >
>> > Then, we can add some more features, such as Alert handler instance,
>> which is created to describe the alert information(email/slack/wechat/...,
>> to whom, the title, content, and etc…).
>> >
>> > BTW, email and webhook are used frequently. I think they can be the
>> default plugin, and written in the source code, instead of a jar file.
>>
>> Thanks,
>> hgaol
>>
>>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by hgaol <dh...@hotmail.com>.
Hi @Gang Li, I agree with you that the plug-in module should be a common module for all plug-in design. Currently I just create the SPI and plugin interface for alert plug-in. If need more type for future, it’s very easy to define other plugins.
Currently the alert service can detect the plug-in jars and load it. But I didn’t implement how to collaborate with front-end about the alert types API and how many properties a plugin should have yet. For example, if we add a webhook plugin, we should have the http method(form select), url(form text), etc… So the plugin interface may change to achieve this goal.
Hi @leon bao, as mentioned above, I agree to make the plugin module as a common module for all plugin design, is that what you mean?
> why don't we put the alert plug-in as an alert module?
I’m not clear about it, do you mean we could just put alert plugin in alert module? I decide to put frequently used plugins in alert module, as for others like SMS(which may has different api), make it as a plugin jar file.
Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient. Developers just need to implement the interfaces in SPI, which are in `dolphinscheduler-plugin-api` module.
Hi all, thanks for your careful review! Here is some link.
PR about plugin management. <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
PR about how to develop a simple plugin. <https://github.com/apache/incubator-dolphinscheduler/pull/2593>
README about how to develop a simple plugin. <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07>
Thanks in advance!
Thanks,
hgaol
> On May 12, 2020, at 18:57, JUN GAO <ga...@gmail.com> wrote:
>
> we can refer to presto plug-in, etc.
> 1. Design the SPI and define the interface that the plugin needs to implement
> 2. All plugins create new models to implement the interface in SPI.
> The benefits are as follows:
> 1. For developers, they only need to care about the interface in SPI, and do not need to care about other people's code, so it is easier to get started developing plug-ins.
> 2. For code reviewers, to improve the efficiency of review, independent models will be more easily reviewed. If there is a problem with the code, it is easy to roll back.
> 3. For users, there are more flexible options. For example, some people only want to use the A plug-in but not the B plug-in.
>
> Thank you !
>
> lgcareer2019@outlook.com <ma...@outlook.com> <lgcareer2019@outlook.com <ma...@outlook.com>> 于2020年5月12日周二 下午6:31写道：
> Hi Gao,
>
> May be I didn't express clearly,I think the dolphinscheduler-plugin module is a common module for all plugin design,
> It's not neccessarily to put the implement of a plugin in the dolphinscheduler-plugin module.
>
> I also agree with the following way your said,I see the pr you submited didn't match it now,Will you give a deep design about the following way?
>
> > A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com <ma...@outlook.com><mailto:lgcareer2019@outlook.com <ma...@outlook.com>>
>
> From: hgaol
> Date: 2020-05-11 19:00
> To: lgcareer2019@outlook.com <ma...@outlook.com>
> CC: dev
> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> Hi Gang,
>
> I agree with that making examples as a pure folder, thanks for your suggestion.
>
> For dolphinscheduler-plugin-api, do you mean it’s better to rename this module? I’m not clear about ‘all plugin function need to be implemented here’, do you mean we should put implemented plugin here?
>
> BTW, I’ve write a simple doc about how to implement an alert plugin.https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07>
>
> Also, I think for some frequently used plugins(email/webhook/...), we could just implement in alert service. Here is the former PR about it. https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>
> Thanks for your careful review!
>
> Thanks,
> hgaol
>
> On May 11, 2020, at 18:09, lgcareer2019@outlook.com <ma...@outlook.com> wrote:
>
> Hi Gao,
>
> I agree with the following you said,If this is a plugin module,all the plugin function need to be implemented here.
> Now the name is dolphinscheduler-plugin-api,I think we can change the module name, Such as `dolphinscheduler-plugin`.
> > I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
>
> Yes,I think it just a folder,not a sub-module.
> We can also look at other developers' comments on this.
> >Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com <ma...@outlook.com><mailto:lgcareer2019@outlook.com <ma...@outlook.com>>
>
> From: hgaol
> Date: 2020-05-11 17:45
> To: dev; lgcareer2019
> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> Hi Gang,
>
> Thanks for your suggestion.
> 1.Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
> [hgaol] The customer should depend on this module to implement their own plugin. If put it into alert service, customer should put `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
>
> Also, I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
>
> 2. I think the examples module can be a folder.
> [hgaol] Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
>
> Thanks,
> hgaol
>
> On May 11, 2020, at 11:51, lgcareer2019@outlook.com <ma...@outlook.com> wrote:
>
> Hi hgaol,
> Sorry for not replying in time.
> I saw you added the module dolphinscheduler-plugin-api and examples module,here are my two points want to discuss.
>
> 1、Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
> 2、I think the examples module can be a folder.
>
>
>
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com <ma...@outlook.com><mailto:lgcareer2019@outlook.com <ma...@outlook.com>>
>
> From: hgaol
> Date: 2020-04-29 23:37
> To: dev
> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> Hi folks,
>
> I’ve created a PR for this feature, here is the link
> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572> <https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>>
>
> Feel free to let me know if there is any suggestion.
>
> Thanks,
> Han Gao
>
> On Apr 28, 2020, at 15:25, xingchun.chen <xingchun.chen@qq.com <ma...@qq.com>> wrote:
>
> It sounds very good, looking forward to your contribution
>
>
> best wish!
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "hgaol"<dhangao@hotmail.com <ma...@hotmail.com>>;
> 发送时间: 2020年4月28日(星期二) 下午3:11
> 收件人: "dev"<dev@dolphinscheduler.apache.org <ma...@dolphinscheduler.apache.org>>;
>
> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>
>
>
> Hi folks,
>
> I think it will be useful for Alert service supporting the plugin management. I’ve create an issue in Github and described about it, here <https://github.com/apache/incubator-dolphinscheduler/issues/2553> <https://github.com/apache/incubator-dolphinscheduler/issues/2553>>; is the link. Copy here for convenience.
>
> > I think the plugin feature is needed by the customers, because different user may have different environment. For example, I want to use SMS to send alert information, but the SMS API is provided by some system we don't know before, and our DS has already deployed. In this case, I just want to implement the plugin code, and don't change the source code.
> >
> > Another benefits are developers can implement different plugin, like slack, wechat, telegram, github issue, and etc...
> >
> > Describe the solution you'd like
> > A clear and concise description of what you want to happen.
> >
> > A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
> >
> > Currently, I've done some investigations, and if this feature is accepted, I think maybe I can implement it.
> >
> > Firstly, I'd better to adapt the current function, which is send mail and something like that.
> >
> > Then, we can add some more features, such as Alert handler instance, which is created to describe the alert information(email/slack/wechat/..., to whom, the title, content, and etc…).
> >
> > BTW, email and webhook are used frequently. I think they can be the default plugin, and written in the source code, instead of a jar file.
>
> Thanks,
> hgaol
>
>
>
> --
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
>
> gaojun2048@gmail.com <ma...@gmail.com>
>
Re: Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by JUN GAO <ga...@gmail.com>.
we can refer to presto plug-in, etc.
1. Design the SPI and define the interface that the plugin needs to
implement
2. All plugins create new models to implement the interface in SPI.
The benefits are as follows:
1. For developers, they only need to care about the interface in SPI, and
do not need to care about other people's code, so it is easier to get
started developing plug-ins.
2. For code reviewers, to improve the efficiency of review, independent
models will be more easily reviewed. If there is a problem with the code,
it is easy to roll back.
3. For users, there are more flexible options. For example, some people
only want to use the A plug-in but not the B plug-in.
Thank you !
lgcareer2019@outlook.com <lg...@outlook.com> 于2020年5月12日周二 下午6:31写道：
> Hi Gao,
>
> May be I didn't express clearly,I think the dolphinscheduler-plugin module
> is a common module for all plugin design,
> It's not neccessarily to put the implement of a plugin in the
> dolphinscheduler-plugin module.
>
> I also agree with the following way your said,I see the pr you submited
> didn't match it now,Will you give a deep design about the following way?
>
> > A better way is that user do not need to change the source code to
> support new alert method. Instead, they just need to implement some
> interface, and compliance with the convention, then package all(include
> dependencies) in a single jar file, and put the jar file in some specific
> place, such as $DS_HOME/plugins. After that, when start Alert service, the
> plugins can be recognized and loaded. So user can choose the alert method
> in the portal.
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> From: hgaol
> Date: 2020-05-11 19:00
> To: lgcareer2019@outlook.com
> CC: dev
> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> Hi Gang,
>
> I agree with that making examples as a pure folder, thanks for your
> suggestion.
>
> For dolphinscheduler-plugin-api, do you mean it’s better to rename this
> module? I’m not clear about ‘all plugin function need to be implemented
> here’, do you mean we should put implemented plugin here?
>
> BTW, I’ve write a simple doc about how to implement an alert plugin.
> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
>
> Also, I think for some frequently used plugins(email/webhook/...), we
> could just implement in alert service. Here is the former PR about it.
> https://github.com/apache/incubator-dolphinscheduler/pull/2572
>
> Thanks for your careful review!
>
> Thanks,
> hgaol
>
> On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
>
> Hi Gao,
>
> I agree with the following you said,If this is a plugin module,all the
> plugin function need to be implemented here.
> Now the name is dolphinscheduler-plugin-api,I think we can change the
> module name, Such as `dolphinscheduler-plugin`.
> > I think dolphinscheduler-plugin-api could not only support alert plugin,
> but also support java job task for future. That’s another topic.
>
> Yes,I think it just a folder,not a sub-module.
> We can also look at other developers' comments on this.
> >Do you mean the examples module should not be a sub-module in
> dolphinscheduler project pom file, just a pure folder?
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> From: hgaol
> Date: 2020-05-11 17:45
> To: dev; lgcareer2019
> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> Hi Gang,
>
> Thanks for your suggestion.
> 1.Whether the function of module dolphinscheduler-plugin-api can be puted
> in the alert module.
> [hgaol] The customer should depend on this module to implement their own
> plugin. If put it into alert service, customer should put
> `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
>
> Also, I think dolphinscheduler-plugin-api could not only support alert
> plugin, but also support java job task for future. That’s another topic.
>
> 2. I think the examples module can be a folder.
> [hgaol] Do you mean the examples module should not be a sub-module in
> dolphinscheduler project pom file, just a pure folder?
>
> Thanks,
> hgaol
>
> On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
>
> Hi hgaol,
> Sorry for not replying in time.
> I saw you added the module dolphinscheduler-plugin-api and examples
> module,here are my two points want to discuss.
>
> 1、Whether the function of module dolphinscheduler-plugin-api can be puted
> in the alert module.
> 2、I think the examples module can be a folder.
>
>
>
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> From: hgaol
> Date: 2020-04-29 23:37
> To: dev
> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> Hi folks,
>
> I’ve created a PR for this feature, here is the link
> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <
> https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>
> Feel free to let me know if there is any suggestion.
>
> Thanks,
> Han Gao
>
> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com> wrote:
>
> It sounds very good, looking forward to your contribution
>
>
> best wish!
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "hgaol"<dhangao@hotmail.com>;
> 发送时间: 2020年4月28日(星期二) 下午3:11
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>
>
>
> Hi folks,
>
> I think it will be useful for Alert service supporting the plugin
> management. I’ve create an issue in Github and described about it, here <
> https://github.com/apache/incubator-dolphinscheduler/issues/2553> is
> the link. Copy here for convenience.
>
> > I think the plugin feature is needed by the customers, because
> different user may have different environment. For example, I want to use
> SMS to send alert information, but the SMS API is provided by some system
> we don't know before, and our DS has already deployed. In this case, I just
> want to implement the plugin code, and don't change the source code.
> >
> > Another benefits are developers can implement different plugin, like
> slack, wechat, telegram, github issue, and etc...
> >
> > Describe the solution you'd like
> > A clear and concise description of what you want to happen.
> >
> > A better way is that user do not need to change the source code to
> support new alert method. Instead, they just need to implement some
> interface, and compliance with the convention, then package all(include
> dependencies) in a single jar file, and put the jar file in some specific
> place, such as $DS_HOME/plugins. After that, when start Alert service, the
> plugins can be recognized and loaded. So user can choose the alert method
> in the portal.
> >
> > Currently, I've done some investigations, and if this feature is
> accepted, I think maybe I can implement it.
> >
> > Firstly, I'd better to adapt the current function, which is send mail
> and something like that.
> >
> > Then, we can add some more features, such as Alert handler instance,
> which is created to describe the alert information(email/slack/wechat/...,
> to whom, the title, content, and etc…).
> >
> > BTW, email and webhook are used frequently. I think they can be the
> default plugin, and written in the source code, instead of a jar file.
>
> Thanks,
> hgaol
>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by "lgcareer2019@outlook.com" <lg...@outlook.com>.
Hi Gao,
May be I didn't express clearly,I think the dolphinscheduler-plugin module is a common module for all plugin design,
It's not neccessarily to put the implement of a plugin in the dolphinscheduler-plugin module.
I also agree with the following way your said,I see the pr you submited didn't match it now,Will you give a deep design about the following way?
> A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: hgaol
Date: 2020-05-11 19:00
To: lgcareer2019@outlook.com
CC: dev
Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
Hi Gang,
I agree with that making examples as a pure folder, thanks for your suggestion.
For dolphinscheduler-plugin-api, do you mean it’s better to rename this module? I’m not clear about ‘all plugin function need to be implemented here’, do you mean we should put implemented plugin here?
BTW, I’ve write a simple doc about how to implement an alert plugin.https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07
Also, I think for some frequently used plugins(email/webhook/...), we could just implement in alert service. Here is the former PR about it. https://github.com/apache/incubator-dolphinscheduler/pull/2572
Thanks for your careful review!
Thanks,
hgaol
On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
Hi Gao,
I agree with the following you said,If this is a plugin module,all the plugin function need to be implemented here.
Now the name is dolphinscheduler-plugin-api,I think we can change the module name, Such as `dolphinscheduler-plugin`.
> I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
Yes,I think it just a folder,not a sub-module.
We can also look at other developers' comments on this.
>Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: hgaol
Date: 2020-05-11 17:45
To: dev; lgcareer2019
Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
Hi Gang,
Thanks for your suggestion.
1.Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
[hgaol] The customer should depend on this module to implement their own plugin. If put it into alert service, customer should put `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
Also, I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
2. I think the examples module can be a folder.
[hgaol] Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
Thanks,
hgaol
On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
Hi hgaol,
Sorry for not replying in time.
I saw you added the module dolphinscheduler-plugin-api and examples module,here are my two points want to discuss.
1、Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
2、I think the examples module can be a folder.
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: hgaol
Date: 2020-04-29 23:37
To: dev
Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
Hi folks,
I’ve created a PR for this feature, here is the link
https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
Feel free to let me know if there is any suggestion.
Thanks,
Han Gao
On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com> wrote:
It sounds very good, looking forward to your contribution
best wish!
------------------ 原始邮件 ------------------
发件人: "hgaol"<dhangao@hotmail.com>;
发送时间: 2020年4月28日(星期二) 下午3:11
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: [PROPOSAL][Feature] Support plugin in Alert service
Hi folks,
I think it will be useful for Alert service supporting the plugin management. I’ve create an issue in Github and described about it, here <https://github.com/apache/incubator-dolphinscheduler/issues/2553> is the link. Copy here for convenience.
> I think the plugin feature is needed by the customers, because different user may have different environment. For example, I want to use SMS to send alert information, but the SMS API is provided by some system we don't know before, and our DS has already deployed. In this case, I just want to implement the plugin code, and don't change the source code.
>
> Another benefits are developers can implement different plugin, like slack, wechat, telegram, github issue, and etc...
>
> Describe the solution you'd like
> A clear and concise description of what you want to happen.
>
> A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
>
> Currently, I've done some investigations, and if this feature is accepted, I think maybe I can implement it.
>
> Firstly, I'd better to adapt the current function, which is send mail and something like that.
>
> Then, we can add some more features, such as Alert handler instance, which is created to describe the alert information(email/slack/wechat/..., to whom, the title, content, and etc…).
>
> BTW, email and webhook are used frequently. I think they can be the default plugin, and written in the source code, instead of a jar file.
Thanks,
hgaol
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by hgaol <dh...@hotmail.com>.
Hi Gang,
I agree with that making examples as a pure folder, thanks for your suggestion.
For dolphinscheduler-plugin-api, do you mean it’s better to rename this module? I’m not clear about ‘all plugin function need to be implemented here’, do you mean we should put implemented plugin here?
BTW, I’ve write a simple doc about how to implement an alert plugin.https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07>
Also, I think for some frequently used plugins(email/webhook/...), we could just implement in alert service. Here is the former PR about it. https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
Thanks for your careful review!
Thanks,
hgaol
> On May 11, 2020, at 18:09, lgcareer2019@outlook.com wrote:
>
> Hi Gao,
>
> I agree with the following you said,If this is a plugin module,all the plugin function need to be implemented here.
> Now the name is dolphinscheduler-plugin-api,I think we can change the module name, Such as `dolphinscheduler-plugin`.
> > I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
>
> Yes,I think it just a folder,not a sub-module.
> We can also look at other developers' comments on this.
> >Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com <ma...@outlook.com><mailto: <ma...@outlook.com>lgcareer2019 <ma...@outlook.com>@outlook.com <ma...@outlook.com>>
>
> From: hgaol <ma...@hotmail.com>
> Date: 2020-05-11 17:45
> To: dev <ma...@dolphinscheduler.apache.org>; lgcareer2019 <ma...@outlook.com>
> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> Hi Gang,
>
> Thanks for your suggestion.
> 1.Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
> [hgaol] The customer should depend on this module to implement their own plugin. If put it into alert service, customer should put `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
>
> Also, I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
>
> 2. I think the examples module can be a folder.
> [hgaol] Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
>
> Thanks,
> hgaol
>
>> On May 11, 2020, at 11:51, lgcareer2019@outlook.com <ma...@outlook.com> wrote:
>>
>> Hi hgaol,
>> Sorry for not replying in time.
>> I saw you added the module dolphinscheduler-plugin-api and examples module,here are my two points want to discuss.
>>
>> 1、Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
>> 2、I think the examples module can be a folder.
>>
>>
>>
>>
>>
>>
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com <ma...@outlook.com><mailto:lgcareer2019@outlook.com <ma...@outlook.com>>
>>
>> From: hgaol
>> Date: 2020-04-29 23:37
>> To: dev
>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
>> Hi folks,
>>
>> I’ve created a PR for this feature, here is the link
>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572> <https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>>
>>
>> Feel free to let me know if there is any suggestion.
>>
>> Thanks,
>> Han Gao
>>
>>> On Apr 28, 2020, at 15:25, xingchun.chen <xingchun.chen@qq.com <ma...@qq.com>> wrote:
>>>
>>> It sounds very good, looking forward to your contribution
>>>
>>>
>>> best wish!
>>>
>>>
>>> ------------------ 原始邮件 ------------------
>>> 发件人: "hgaol"<dhangao@hotmail.com <ma...@hotmail.com>>;
>>> 发送时间: 2020年4月28日(星期二) 下午3:11
>>> 收件人: "dev"<dev@dolphinscheduler.apache.org <ma...@dolphinscheduler.apache.org>>;
>>>
>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>>>
>>>
>>>
>>> Hi folks,
>>>
>>> I think it will be useful for Alert service supporting the plugin management. I’ve create an issue in Github and described about it, here <https://github.com/apache/incubator-dolphinscheduler/issues/2553> <https://github.com/apache/incubator-dolphinscheduler/issues/2553>>; is the link. Copy here for convenience.
>>>
>>> > I think the plugin feature is needed by the customers, because different user may have different environment. For example, I want to use SMS to send alert information, but the SMS API is provided by some system we don't know before, and our DS has already deployed. In this case, I just want to implement the plugin code, and don't change the source code.
>>> >
>>> > Another benefits are developers can implement different plugin, like slack, wechat, telegram, github issue, and etc...
>>> >
>>> > Describe the solution you'd like
>>> > A clear and concise description of what you want to happen.
>>> >
>>> > A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
>>> >
>>> > Currently, I've done some investigations, and if this feature is accepted, I think maybe I can implement it.
>>> >
>>> > Firstly, I'd better to adapt the current function, which is send mail and something like that.
>>> >
>>> > Then, we can add some more features, such as Alert handler instance, which is created to describe the alert information(email/slack/wechat/..., to whom, the title, content, and etc…).
>>> >
>>> > BTW, email and webhook are used frequently. I think they can be the default plugin, and written in the source code, instead of a jar file.
>>>
>>> Thanks,
>>> hgaol
Re: Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by "lgcareer2019@outlook.com" <lg...@outlook.com>.
Hi Gao,
I agree with the following you said,If this is a plugin module,all the plugin function need to be implemented here.
Now the name is dolphinscheduler-plugin-api,I think we can change the module name, Such as `dolphinscheduler-plugin`.
> I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
Yes,I think it just a folder,not a sub-module.
We can also look at other developers' comments on this.
>Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: hgaol
Date: 2020-05-11 17:45
To: dev; lgcareer2019
Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
Hi Gang,
Thanks for your suggestion.
1.Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
[hgaol] The customer should depend on this module to implement their own plugin. If put it into alert service, customer should put `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
Also, I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
2. I think the examples module can be a folder.
[hgaol] Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
Thanks,
hgaol
On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
Hi hgaol,
Sorry for not replying in time.
I saw you added the module dolphinscheduler-plugin-api and examples module,here are my two points want to discuss.
1、Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
2、I think the examples module can be a folder.
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: hgaol
Date: 2020-04-29 23:37
To: dev
Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
Hi folks,
I’ve created a PR for this feature, here is the link
https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
Feel free to let me know if there is any suggestion.
Thanks,
Han Gao
On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com> wrote:
It sounds very good, looking forward to your contribution
best wish!
------------------ 原始邮件 ------------------
发件人: "hgaol"<dhangao@hotmail.com>;
发送时间: 2020年4月28日(星期二) 下午3:11
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: [PROPOSAL][Feature] Support plugin in Alert service
Hi folks,
I think it will be useful for Alert service supporting the plugin management. I’ve create an issue in Github and described about it, here <https://github.com/apache/incubator-dolphinscheduler/issues/2553> is the link. Copy here for convenience.
> I think the plugin feature is needed by the customers, because different user may have different environment. For example, I want to use SMS to send alert information, but the SMS API is provided by some system we don't know before, and our DS has already deployed. In this case, I just want to implement the plugin code, and don't change the source code.
>
> Another benefits are developers can implement different plugin, like slack, wechat, telegram, github issue, and etc...
>
> Describe the solution you'd like
> A clear and concise description of what you want to happen.
>
> A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
>
> Currently, I've done some investigations, and if this feature is accepted, I think maybe I can implement it.
>
> Firstly, I'd better to adapt the current function, which is send mail and something like that.
>
> Then, we can add some more features, such as Alert handler instance, which is created to describe the alert information(email/slack/wechat/..., to whom, the title, content, and etc…).
>
> BTW, email and webhook are used frequently. I think they can be the default plugin, and written in the source code, instead of a jar file.
Thanks,
hgaol
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by hgaol <dh...@hotmail.com>.
Hi Gang,
Thanks for your suggestion.
1.Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
[hgaol] The customer should depend on this module to implement their own plugin. If put it into alert service, customer should put `dolphinscheduler-alert` as dependence, which is too heavy for a plugin.
Also, I think dolphinscheduler-plugin-api could not only support alert plugin, but also support java job task for future. That’s another topic.
2. I think the examples module can be a folder.
[hgaol] Do you mean the examples module should not be a sub-module in dolphinscheduler project pom file, just a pure folder?
Thanks,
hgaol
> On May 11, 2020, at 11:51, lgcareer2019@outlook.com wrote:
>
> Hi hgaol,
> Sorry for not replying in time.
> I saw you added the module dolphinscheduler-plugin-api and examples module,here are my two points want to discuss.
>
> 1、Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
> 2、I think the examples module can be a folder.
>
>
>
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> From: hgaol
> Date: 2020-04-29 23:37
> To: dev
> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
> Hi folks,
>
> I’ve created a PR for this feature, here is the link
> https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
>
> Feel free to let me know if there is any suggestion.
>
> Thanks,
> Han Gao
>
>> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com> wrote:
>>
>> It sounds very good, looking forward to your contribution
>>
>>
>> best wish!
>>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人: "hgaol"<dhangao@hotmail.com>;
>> 发送时间: 2020年4月28日(星期二) 下午3:11
>> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>>
>> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>>
>>
>>
>> Hi folks,
>>
>> I think it will be useful for Alert service supporting the plugin management. I’ve create an issue in Github and described about it, here <https://github.com/apache/incubator-dolphinscheduler/issues/2553> is the link. Copy here for convenience.
>>
>> > I think the plugin feature is needed by the customers, because different user may have different environment. For example, I want to use SMS to send alert information, but the SMS API is provided by some system we don't know before, and our DS has already deployed. In this case, I just want to implement the plugin code, and don't change the source code.
>> >
>> > Another benefits are developers can implement different plugin, like slack, wechat, telegram, github issue, and etc...
>> >
>> > Describe the solution you'd like
>> > A clear and concise description of what you want to happen.
>> >
>> > A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
>> >
>> > Currently, I've done some investigations, and if this feature is accepted, I think maybe I can implement it.
>> >
>> > Firstly, I'd better to adapt the current function, which is send mail and something like that.
>> >
>> > Then, we can add some more features, such as Alert handler instance, which is created to describe the alert information(email/slack/wechat/..., to whom, the title, content, and etc…).
>> >
>> > BTW, email and webhook are used frequently. I think they can be the default plugin, and written in the source code, instead of a jar file.
>>
>> Thanks,
>> hgaol
>
Re: Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by "lgcareer2019@outlook.com" <lg...@outlook.com>.
Hi hgaol,
Sorry for not replying in time.
I saw you added the module dolphinscheduler-plugin-api and examples module,here are my two points want to discuss.
1、Whether the function of module dolphinscheduler-plugin-api can be puted in the alert module.
2、I think the examples module can be a folder.
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: hgaol
Date: 2020-04-29 23:37
To: dev
Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service
Hi folks,
I’ve created a PR for this feature, here is the link
https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
Feel free to let me know if there is any suggestion.
Thanks,
Han Gao
> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com> wrote:
>
> It sounds very good, looking forward to your contribution
>
>
> best wish!
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "hgaol"<dhangao@hotmail.com>;
> 发送时间: 2020年4月28日(星期二) 下午3:11
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>
>
>
> Hi folks,
>
> I think it will be useful for Alert service supporting the plugin management. I’ve create an issue in Github and described about it, here <https://github.com/apache/incubator-dolphinscheduler/issues/2553> is the link. Copy here for convenience.
>
> > I think the plugin feature is needed by the customers, because different user may have different environment. For example, I want to use SMS to send alert information, but the SMS API is provided by some system we don't know before, and our DS has already deployed. In this case, I just want to implement the plugin code, and don't change the source code.
> >
> > Another benefits are developers can implement different plugin, like slack, wechat, telegram, github issue, and etc...
> >
> > Describe the solution you'd like
> > A clear and concise description of what you want to happen.
> >
> > A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
> >
> > Currently, I've done some investigations, and if this feature is accepted, I think maybe I can implement it.
> >
> > Firstly, I'd better to adapt the current function, which is send mail and something like that.
> >
> > Then, we can add some more features, such as Alert handler instance, which is created to describe the alert information(email/slack/wechat/..., to whom, the title, content, and etc…).
> >
> > BTW, email and webhook are used frequently. I think they can be the default plugin, and written in the source code, instead of a jar file.
>
> Thanks,
> hgaol
Re: [PROPOSAL][Feature] Support plugin in Alert service
Posted by hgaol <dh...@hotmail.com>.
Hi folks,
I’ve created a PR for this feature, here is the link
https://github.com/apache/incubator-dolphinscheduler/pull/2572 <https://github.com/apache/incubator-dolphinscheduler/pull/2572>
Feel free to let me know if there is any suggestion.
Thanks,
Han Gao
> On Apr 28, 2020, at 15:25, xingchun.chen <xi...@qq.com> wrote:
>
> It sounds very good, looking forward to your contribution
>
>
> best wish!
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "hgaol"<dhangao@hotmail.com>;
> 发送时间: 2020年4月28日(星期二) 下午3:11
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: [PROPOSAL][Feature] Support plugin in Alert service
>
>
>
> Hi folks,
>
> I think it will be useful for Alert service supporting the plugin management. I’ve create an issue in Github and described about it, here <https://github.com/apache/incubator-dolphinscheduler/issues/2553> is the link. Copy here for convenience.
>
> > I think the plugin feature is needed by the customers, because different user may have different environment. For example, I want to use SMS to send alert information, but the SMS API is provided by some system we don't know before, and our DS has already deployed. In this case, I just want to implement the plugin code, and don't change the source code.
> >
> > Another benefits are developers can implement different plugin, like slack, wechat, telegram, github issue, and etc...
> >
> > Describe the solution you'd like
> > A clear and concise description of what you want to happen.
> >
> > A better way is that user do not need to change the source code to support new alert method. Instead, they just need to implement some interface, and compliance with the convention, then package all(include dependencies) in a single jar file, and put the jar file in some specific place, such as $DS_HOME/plugins. After that, when start Alert service, the plugins can be recognized and loaded. So user can choose the alert method in the portal.
> >
> > Currently, I've done some investigations, and if this feature is accepted, I think maybe I can implement it.
> >
> > Firstly, I'd better to adapt the current function, which is send mail and something like that.
> >
> > Then, we can add some more features, such as Alert handler instance, which is created to describe the alert information(email/slack/wechat/..., to whom, the title, content, and etc…).
> >
> > BTW, email and webhook are used frequently. I think they can be the default plugin, and written in the source code, instead of a jar file.
>
> Thanks,
> hgaol