You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by 许昌群 <ch...@17zuoye.com> on 2020/06/04 07:03:32 UTC
答复: [Poprosal] DolphinScheduler Plugin
Hi all
Regarding the alarm, I have some other ideas, please refer to it.
At present, there are many types of message notification components, such as email/WeChat/SMS, or some notification components developed by the company.If it is not realistic to connect all the components, can we change the way of thinking and flip this process? We provide an interface for sending information, allowing users to interface with it.
Described as follows:
1. Define the sending interface of the message in DS, such as sendMsg
2. Load the user-configured interface implementation, for example, http implementation is http://192.168.1.1:8080/sendMsg/email
3. Define the message sending format, {"tos":"softxcq@sina.com", "content":"taskA failed."}
4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={"tos":"softxcq@sina.com", "content":"taskA failed."}
If there are other alarm methods, add an http interface
Finally, you only need to provide several default alarm functions in DS to transfer other message notification functions to users.Although you will lose some ease of use out of the box, it will increase flexibility. You can implement your own system to ensure that the DS alarm logic is simple and easy to use.
________________________________
发件人: JUN GAO <ga...@gmail.com>
发送时间: 2020年6月3日 12:13:21
收件人: dev
主题: Re: [Poprosal] DolphinScheduler Plugin
1、Alert is very independent and common function, why should we to implement
it as plugin?
I think Alert should be divided into two parts. One is that the server
collects various alarm information. The other is to pass these warning
contents through optional methods (such as email, WeChat). I think the
specific alarm method should not be implemented by the AlertServer server.
It should be abstracted into an interface and implemented by a concrete
plug-in implementation layer. If someone understands and is familiar with
other alarm methods, he can implement other alarm plug-ins based on the
plug-in design. If he wants, he can contribute this plug-in to us.
Plug-inization is just a way to decouple projects.
2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty? If
it is a thirdparty, it may generally be managed by its own git, and it will
not be placed in the dolphinscheduler related directory.
Plug-inization is just a way to decouple projects. All Plugin is part
of dolphinscheduler.
And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
@daolidong We can FYI this.
Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> Good proposal, but I have some question as following:
> 1、Alert is very independent and common function, why should we to
> implement it as plugin?
> 2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty?
> If it is a thirdparty, it may generally be managed by its own git, and it
> will not be placed in the dolphinscheduler related directory.
>
> And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org
> ---------------
>
>
>
> > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
回复： Re: [Poprosal] DolphinScheduler Plugin
Posted by 向偲彪 <xi...@qq.com>.
alpacajs can let json data be displayed in the form of form, but it lacks the method binding for each field and how to submit the entire form data documentation. Does anyone have used alpacajs or a friend who is familiar with alpacajs?
DolphinScheduler(Incubator) committer
Caibiao Xiang 向偲彪
------------------ 原始邮件 ------------------
发件人: "JUN GAO"<gaojun2048@gmail.com>;
发送时间: 2020年6月11日(星期四) 下午3:48
收件人: "dev"<dev@dolphinscheduler.apache.org>;"leonbao"<leonbao@apache.org>;"lgcareer2019"<lgcareer2019@outlook.com>;"GabryWu"<wu_shao_jie@qq.com>;"Xiaochun Liu"<liuxiaochun@apache.org>;"许昌群"<changqun.xu@17zuoye.com>;
主题: Re: Re: [Poprosal] DolphinScheduler Plugin
@leonbao@apache.org <leonbao@apache.org> @lgcareer2019@outlook.com
<lgcareer2019@outlook.com> @GabryWu <wu_shao_jie@qq.com> @Xiaochun Liu
<liuxiaochun@apache.org> @Han Gao @许昌群 <changqun.xu@17zuoye.com>
Thank you eveny one.
I see that most of the friends who participate in this proposal discussion
support it. Therefore, I will continue the work of DS plug-in design. Next,
I applied to add the incubator-dolphinscheduler-maven-plugin repository to
apache, and applied to create a branch on the incubator-dolphinscheduler
repository for the next plug-in development and collaboration. And then I
will open this todo list.
leon bao <leonbao@apache.org> 于2020年6月10日周三 下午6:19写道：
> As a developer, I think
> 1. A good architecture allows a modification that will not affect the
> operation of other modules, which is not possible in the current DS. For
> example, adding new task types may affect the execution of other tasks
> because they are all in one module. .
> If good isolation is achieved, any modification to a module will not affect
> the normal operation of other modules in the system.
> 2. We need an architecture so that developers and the community stay in
> sync, and each version of the community has the least impact on developers.
> This requires a more flexible architecture and will attract more
> developers.
>
> so i think the plug-in is suitable for DS.
>
>
> lgcareer2019@outlook.com <lgcareer2019@outlook.com> 于2020年6月10日周三
> 下午4:48写道：
>
> > @jun gao
> > So great to decouple the upper core logic of DS from the specific
> > implementation.
> > I think this design is more scalable and developers can cooperate in
> > develop more easily.So I agree with it.
> >
> > @wu shaoj @changqun
> > I saw you may be have other options that flexible plugin mechanism is
> > complex and in a low priority.
> >
> > I think we can redesign the alert service in this time first,and I saw
> jun
> > gao have already done a lot of works.
> >
> >
> >
> > DolphinScheduler(Incubator) PPMC
> > Gang Li 李岗
> >
> > lgcareer2019@outlook.com<mailto:lgcareer2019@outlook.com>
> >
> > 发件人： wu shaoj
> > 发送时间： 2020-06-04 17:44
> > 收件人： dev@dolphinscheduler.apache.org
> > 主题： Re: 答复: [Poprosal] DolphinScheduler Plugin
> > Yes, we should forcus on the job with high priority .
> > No need to satisfy all customer requirement which might be unreasonable
> >
> > On 2020/6/4, 17:41, "Hemin Wen" <wenhemin@apache.org> wrote:
> >
> > Very detailed design.
> > 1. About alert module, I agree with @许昌群.
> > Feedback from the community on the alert module, the alert module
> > function
> > is relatively stable,
> > and the current notification function plus webhook whether it can
> meet
> > the
> > needs.
> >
> > 2.It is not recommended that the UI be modularized, this will
> increase
> > the
> > complexity of module development and debugging,
> > If there is more complicated UI interaction, it will cause trouble to
> > the
> > development, Not only need to understand UI development,
> > but also understand modular configuration
> >
> > --------------------
> > DolphinScheduler(Incubator) Commtter
> > Hemin Wen 温合民
> > wenhemin@apache.org
> > --------------------
> >
> >
> > Han Gao <dhangao@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> >
> > > Hi folks,
> > >
> > > I prefer plugin way. @许昌群<mailto:changqun.xu@17zuoye.com> provide
> an
> > > alert way like webhook(pls correct me if I'm wrong), which we can
> > support
> > > as an official plugin.
> > >
> > > The difference between webhook and plugin is you should start a
> > server to
> > > accept alert request in webhook, but in plugin way you don't need
> to
> > do
> > > that.
> > >
> > > IMO, I prefer implement some official plugins in AlertServer
> package
> > > directly, as for others, we could put it in a plugin folder. I've
> > > implemented a plugin class loader in DS common package, which used
> > to load
> > > plugin and avoid package conflict. Put official plugins directly in
> > > AlertServer package can use the common packages, and will cost less
> > memory
> > > for class meta data.
> > >
> > > But put all plugin into a specific folder is fine too, because it
> > looks
> > > clean.
> > >
> > > BTW, I think there is no conflict to support both webhook and
> plugin.
> > >
> > > Thanks,
> > > Han Gao
> > >
> > > ________________________________
> > > From: wu shaoj <gabrywu@apache.org>
> > > Sent: Thursday, June 4, 2020 15:13
> > > To: dev@dolphinscheduler.apache.org <
> dev@dolphinscheduler.apache.org
> > >;
> > > 许昌群 <changqun.xu@17zuoye.com>
> > > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> > >
> > > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and
> in
> > a low
> > > priority I think.
> > >
> > > On 2020/6/4, 15:04, "许昌群" <changqun.xu@17zuoye.com> wrote:
> > >
> > > Hi all
> > >
> > > Regarding the alarm, I have some other ideas, please refer to
> it.
> > >
> > > At present, there are many types of message notification
> > components,
> > > such as email/WeChat/SMS, or some notification components developed
> > by the
> > > company.If it is not realistic to connect all the components, can
> we
> > change
> > > the way of thinking and flip this process? We provide an interface
> > for
> > > sending information, allowing users to interface with it.
> > >
> > > Described as follows:
> > > 1. Define the sending interface of the message in DS, such as
> > sendMsg
> > > 2. Load the user-configured interface implementation, for
> > example,
> > > http implementation is http://192.168.1.1:8080/sendMsg/email
> > > 3. Define the message sending format, {"tos":"softxcq@sina.com
> ",
> > > "content":"taskA failed."}
> > > 4. Send http request.
> > http://192.168.1.1:8080/sendMsg/email?data={
> > > "tos":"softxcq@sina.com", "content":"taskA failed."}
> > >
> > > If there are other alarm methods, add an http interface
> > >
> > > Finally, you only need to provide several default alarm
> > functions in
> > > DS to transfer other message notification functions to
> > users.Although you
> > > will lose some ease of use out of the box, it will increase
> > flexibility.
> > > You can implement your own system to ensure that the DS alarm logic
> > is
> > > simple and easy to use.
> > >
> > >
> > > ________________________________
> > > 发件人: JUN GAO <gaojun2048@gmail.com>
> > > 发送时间: 2020年6月3日 12:13:21
> > > 收件人: dev
> > > 主题: Re: [Poprosal] DolphinScheduler Plugin
> > >
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement
> > > it as plugin?
> > >
> > > I think Alert should be divided into two parts. One is that the
> > server
> > > collects various alarm information. The other is to pass these
> > warning
> > > contents through optional methods (such as email, WeChat). I
> > think the
> > > specific alarm method should not be implemented by the
> > AlertServer
> > > server.
> > > It should be abstracted into an interface and implemented by a
> > concrete
> > > plug-in implementation layer. If someone understands and is
> > familiar
> > > with
> > > other alarm methods, he can implement other alarm plug-ins
> based
> > on the
> > > plug-in design. If he wants, he can contribute this plug-in to
> > us.
> > > Plug-inization is just a way to decouple projects.
> > >
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > > thirdparty? If
> > > it is a thirdparty, it may generally be managed by its own git,
> > and it
> > > will
> > > not be placed in the dolphinscheduler related directory.
> > >
> > > Plug-inization is just a way to decouple projects. All Plugin
> is
> > part
> > > of dolphinscheduler.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> > powerful.
> > >
> > > @daolidong We can FYI this.
> > >
> > > Xiaochun Liu <liuxiaochun@apache.org> 于2020年6月3日周三 上午11:35写道：
> > >
> > > > Good proposal, but I have some question as following:
> > > > 1、Alert is very independent and common function, why should
> we
> > to
> > > > implement it as plugin?
> > > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > > thirdparty?
> > > > If it is a thirdparty, it may generally be managed by its own
> > git,
> > > and it
> > > > will not be placed in the dolphinscheduler related directory.
> > > >
> > > > And we can use jsonschema to solve the dynamic ui, for
> example:
> > > alpacajs(
> > > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> > powerful.
> > > >
> > > > Best Regards
> > > > ---------------
> > > > DolphinScheduler(Incubator) Committer
> > > > Xiaochun Liu 刘小春
> > > > liuxiaochun@apache.org
> > > > ---------------
> > > >
> > > >
> > > >
> > > > > 2020年6月2日 下午3:04，JUN GAO <gaojun2048@gmail.com> 写道：
> > > > >
> > > > >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > > >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > > >
> > >
> > >
> > > --
> > >
> > > DolphinScheduler(Incubator) PPMC
> > > Jun Gao 高俊
> > > gaojun2048@gmail.com
> > >
> >
>
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: Re: [Poprosal] DolphinScheduler Plugin
Posted by JUN GAO <ga...@gmail.com>.
@leonbao@apache.org <le...@apache.org> @lgcareer2019@outlook.com
<lg...@outlook.com> @GabryWu <wu...@qq.com> @Xiaochun Liu
<li...@apache.org> @Han Gao @许昌群 <ch...@17zuoye.com>
Thank you eveny one.
I see that most of the friends who participate in this proposal discussion
support it. Therefore, I will continue the work of DS plug-in design. Next,
I applied to add the incubator-dolphinscheduler-maven-plugin repository to
apache, and applied to create a branch on the incubator-dolphinscheduler
repository for the next plug-in development and collaboration. And then I
will open this todo list.
leon bao <le...@apache.org> 于2020年6月10日周三 下午6:19写道：
> As a developer, I think
> 1. A good architecture allows a modification that will not affect the
> operation of other modules, which is not possible in the current DS. For
> example, adding new task types may affect the execution of other tasks
> because they are all in one module. .
> If good isolation is achieved, any modification to a module will not affect
> the normal operation of other modules in the system.
> 2. We need an architecture so that developers and the community stay in
> sync, and each version of the community has the least impact on developers.
> This requires a more flexible architecture and will attract more
> developers.
>
> so i think the plug-in is suitable for DS.
>
>
> lgcareer2019@outlook.com <lg...@outlook.com> 于2020年6月10日周三
> 下午4:48写道：
>
> > @jun gao
> > So great to decouple the upper core logic of DS from the specific
> > implementation.
> > I think this design is more scalable and developers can cooperate in
> > develop more easily.So I agree with it.
> >
> > @wu shaoj @changqun
> > I saw you may be have other options that flexible plugin mechanism is
> > complex and in a low priority.
> >
> > I think we can redesign the alert service in this time first,and I saw
> jun
> > gao have already done a lot of works.
> >
> >
> >
> > DolphinScheduler(Incubator) PPMC
> > Gang Li 李岗
> >
> > lgcareer2019@outlook.com<ma...@outlook.com>
> >
> > 发件人： wu shaoj
> > 发送时间： 2020-06-04 17:44
> > 收件人： dev@dolphinscheduler.apache.org
> > 主题： Re: 答复: [Poprosal] DolphinScheduler Plugin
> > Yes, we should forcus on the job with high priority .
> > No need to satisfy all customer requirement which might be unreasonable
> >
> > On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
> >
> > Very detailed design.
> > 1. About alert module, I agree with @许昌群.
> > Feedback from the community on the alert module, the alert module
> > function
> > is relatively stable,
> > and the current notification function plus webhook whether it can
> meet
> > the
> > needs.
> >
> > 2.It is not recommended that the UI be modularized, this will
> increase
> > the
> > complexity of module development and debugging,
> > If there is more complicated UI interaction, it will cause trouble to
> > the
> > development, Not only need to understand UI development,
> > but also understand modular configuration
> >
> > --------------------
> > DolphinScheduler(Incubator) Commtter
> > Hemin Wen 温合民
> > wenhemin@apache.org
> > --------------------
> >
> >
> > Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> >
> > > Hi folks,
> > >
> > > I prefer plugin way. @许昌群<ma...@17zuoye.com> provide
> an
> > > alert way like webhook(pls correct me if I'm wrong), which we can
> > support
> > > as an official plugin.
> > >
> > > The difference between webhook and plugin is you should start a
> > server to
> > > accept alert request in webhook, but in plugin way you don't need
> to
> > do
> > > that.
> > >
> > > IMO, I prefer implement some official plugins in AlertServer
> package
> > > directly, as for others, we could put it in a plugin folder. I've
> > > implemented a plugin class loader in DS common package, which used
> > to load
> > > plugin and avoid package conflict. Put official plugins directly in
> > > AlertServer package can use the common packages, and will cost less
> > memory
> > > for class meta data.
> > >
> > > But put all plugin into a specific folder is fine too, because it
> > looks
> > > clean.
> > >
> > > BTW, I think there is no conflict to support both webhook and
> plugin.
> > >
> > > Thanks,
> > > Han Gao
> > >
> > > ________________________________
> > > From: wu shaoj <ga...@apache.org>
> > > Sent: Thursday, June 4, 2020 15:13
> > > To: dev@dolphinscheduler.apache.org <
> dev@dolphinscheduler.apache.org
> > >;
> > > 许昌群 <ch...@17zuoye.com>
> > > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> > >
> > > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and
> in
> > a low
> > > priority I think.
> > >
> > > On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
> > >
> > > Hi all
> > >
> > > Regarding the alarm, I have some other ideas, please refer to
> it.
> > >
> > > At present, there are many types of message notification
> > components,
> > > such as email/WeChat/SMS, or some notification components developed
> > by the
> > > company.If it is not realistic to connect all the components, can
> we
> > change
> > > the way of thinking and flip this process? We provide an interface
> > for
> > > sending information, allowing users to interface with it.
> > >
> > > Described as follows:
> > > 1. Define the sending interface of the message in DS, such as
> > sendMsg
> > > 2. Load the user-configured interface implementation, for
> > example,
> > > http implementation is http://192.168.1.1:8080/sendMsg/email
> > > 3. Define the message sending format, {"tos":"softxcq@sina.com
> ",
> > > "content":"taskA failed."}
> > > 4. Send http request.
> > http://192.168.1.1:8080/sendMsg/email?data={
> > > "tos":"softxcq@sina.com", "content":"taskA failed."}
> > >
> > > If there are other alarm methods, add an http interface
> > >
> > > Finally, you only need to provide several default alarm
> > functions in
> > > DS to transfer other message notification functions to
> > users.Although you
> > > will lose some ease of use out of the box, it will increase
> > flexibility.
> > > You can implement your own system to ensure that the DS alarm logic
> > is
> > > simple and easy to use.
> > >
> > >
> > > ________________________________
> > > 发件人: JUN GAO <ga...@gmail.com>
> > > 发送时间: 2020年6月3日 12:13:21
> > > 收件人: dev
> > > 主题: Re: [Poprosal] DolphinScheduler Plugin
> > >
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement
> > > it as plugin?
> > >
> > > I think Alert should be divided into two parts. One is that the
> > server
> > > collects various alarm information. The other is to pass these
> > warning
> > > contents through optional methods (such as email, WeChat). I
> > think the
> > > specific alarm method should not be implemented by the
> > AlertServer
> > > server.
> > > It should be abstracted into an interface and implemented by a
> > concrete
> > > plug-in implementation layer. If someone understands and is
> > familiar
> > > with
> > > other alarm methods, he can implement other alarm plug-ins
> based
> > on the
> > > plug-in design. If he wants, he can contribute this plug-in to
> > us.
> > > Plug-inization is just a way to decouple projects.
> > >
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > > thirdparty? If
> > > it is a thirdparty, it may generally be managed by its own git,
> > and it
> > > will
> > > not be placed in the dolphinscheduler related directory.
> > >
> > > Plug-inization is just a way to decouple projects. All Plugin
> is
> > part
> > > of dolphinscheduler.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> > powerful.
> > >
> > > @daolidong We can FYI this.
> > >
> > > Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> > >
> > > > Good proposal, but I have some question as following:
> > > > 1、Alert is very independent and common function, why should
> we
> > to
> > > > implement it as plugin?
> > > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > > thirdparty?
> > > > If it is a thirdparty, it may generally be managed by its own
> > git,
> > > and it
> > > > will not be placed in the dolphinscheduler related directory.
> > > >
> > > > And we can use jsonschema to solve the dynamic ui, for
> example:
> > > alpacajs(
> > > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> > powerful.
> > > >
> > > > Best Regards
> > > > ---------------
> > > > DolphinScheduler(Incubator) Committer
> > > > Xiaochun Liu 刘小春
> > > > liuxiaochun@apache.org
> > > > ---------------
> > > >
> > > >
> > > >
> > > > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > > > >
> > > > >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > > >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > > >
> > >
> > >
> > > --
> > >
> > > DolphinScheduler(Incubator) PPMC
> > > Jun Gao 高俊
> > > gaojun2048@gmail.com
> > >
> >
>
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: Re: [Poprosal] DolphinScheduler Plugin
Posted by leon bao <le...@apache.org>.
As a developer, I think
1. A good architecture allows a modification that will not affect the
operation of other modules, which is not possible in the current DS. For
example, adding new task types may affect the execution of other tasks
because they are all in one module. .
If good isolation is achieved, any modification to a module will not affect
the normal operation of other modules in the system.
2. We need an architecture so that developers and the community stay in
sync, and each version of the community has the least impact on developers.
This requires a more flexible architecture and will attract more developers.
so i think the plug-in is suitable for DS.
lgcareer2019@outlook.com <lg...@outlook.com> 于2020年6月10日周三 下午4:48写道：
> @jun gao
> So great to decouple the upper core logic of DS from the specific
> implementation.
> I think this design is more scalable and developers can cooperate in
> develop more easily.So I agree with it.
>
> @wu shaoj @changqun
> I saw you may be have other options that flexible plugin mechanism is
> complex and in a low priority.
>
> I think we can redesign the alert service in this time first,and I saw jun
> gao have already done a lot of works.
>
>
>
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> 发件人： wu shaoj
> 发送时间： 2020-06-04 17:44
> 收件人： dev@dolphinscheduler.apache.org
> 主题： Re: 答复: [Poprosal] DolphinScheduler Plugin
> Yes, we should forcus on the job with high priority .
> No need to satisfy all customer requirement which might be unreasonable
>
> On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
>
> Very detailed design.
> 1. About alert module, I agree with @许昌群.
> Feedback from the community on the alert module, the alert module
> function
> is relatively stable,
> and the current notification function plus webhook whether it can meet
> the
> needs.
>
> 2.It is not recommended that the UI be modularized, this will increase
> the
> complexity of module development and debugging,
> If there is more complicated UI interaction, it will cause trouble to
> the
> development, Not only need to understand UI development,
> but also understand modular configuration
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
>
> > Hi folks,
> >
> > I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> > alert way like webhook(pls correct me if I'm wrong), which we can
> support
> > as an official plugin.
> >
> > The difference between webhook and plugin is you should start a
> server to
> > accept alert request in webhook, but in plugin way you don't need to
> do
> > that.
> >
> > IMO, I prefer implement some official plugins in AlertServer package
> > directly, as for others, we could put it in a plugin folder. I've
> > implemented a plugin class loader in DS common package, which used
> to load
> > plugin and avoid package conflict. Put official plugins directly in
> > AlertServer package can use the common packages, and will cost less
> memory
> > for class meta data.
> >
> > But put all plugin into a specific folder is fine too, because it
> looks
> > clean.
> >
> > BTW, I think there is no conflict to support both webhook and plugin.
> >
> > Thanks,
> > Han Gao
> >
> > ________________________________
> > From: wu shaoj <ga...@apache.org>
> > Sent: Thursday, June 4, 2020 15:13
> > To: dev@dolphinscheduler.apache.org <dev@dolphinscheduler.apache.org
> >;
> > 许昌群 <ch...@17zuoye.com>
> > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> >
> > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in
> a low
> > priority I think.
> >
> > On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
> >
> > Hi all
> >
> > Regarding the alarm, I have some other ideas, please refer to it.
> >
> > At present, there are many types of message notification
> components,
> > such as email/WeChat/SMS, or some notification components developed
> by the
> > company.If it is not realistic to connect all the components, can we
> change
> > the way of thinking and flip this process? We provide an interface
> for
> > sending information, allowing users to interface with it.
> >
> > Described as follows:
> > 1. Define the sending interface of the message in DS, such as
> sendMsg
> > 2. Load the user-configured interface implementation, for
> example,
> > http implementation is http://192.168.1.1:8080/sendMsg/email
> > 3. Define the message sending format, {"tos":"softxcq@sina.com",
> > "content":"taskA failed."}
> > 4. Send http request.
> http://192.168.1.1:8080/sendMsg/email?data={
> > "tos":"softxcq@sina.com", "content":"taskA failed."}
> >
> > If there are other alarm methods, add an http interface
> >
> > Finally, you only need to provide several default alarm
> functions in
> > DS to transfer other message notification functions to
> users.Although you
> > will lose some ease of use out of the box, it will increase
> flexibility.
> > You can implement your own system to ensure that the DS alarm logic
> is
> > simple and easy to use.
> >
> >
> > ________________________________
> > 发件人: JUN GAO <ga...@gmail.com>
> > 发送时间: 2020年6月3日 12:13:21
> > 收件人: dev
> > 主题: Re: [Poprosal] DolphinScheduler Plugin
> >
> > 1、Alert is very independent and common function, why should we to
> > implement
> > it as plugin?
> >
> > I think Alert should be divided into two parts. One is that the
> server
> > collects various alarm information. The other is to pass these
> warning
> > contents through optional methods (such as email, WeChat). I
> think the
> > specific alarm method should not be implemented by the
> AlertServer
> > server.
> > It should be abstracted into an interface and implemented by a
> concrete
> > plug-in implementation layer. If someone understands and is
> familiar
> > with
> > other alarm methods, he can implement other alarm plug-ins based
> on the
> > plug-in design. If he wants, he can contribute this plug-in to
> us.
> > Plug-inization is just a way to decouple projects.
> >
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty? If
> > it is a thirdparty, it may generally be managed by its own git,
> and it
> > will
> > not be placed in the dolphinscheduler related directory.
> >
> > Plug-inization is just a way to decouple projects. All Plugin is
> part
> > of dolphinscheduler.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> >
> > @daolidong We can FYI this.
> >
> > Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> >
> > > Good proposal, but I have some question as following:
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement it as plugin?
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty?
> > > If it is a thirdparty, it may generally be managed by its own
> git,
> > and it
> > > will not be placed in the dolphinscheduler related directory.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> > >
> > > Best Regards
> > > ---------------
> > > DolphinScheduler(Incubator) Committer
> > > Xiaochun Liu 刘小春
> > > liuxiaochun@apache.org
> > > ---------------
> > >
> > >
> > >
> > > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > > >
> > > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > >
> >
> >
> > --
> >
> > DolphinScheduler(Incubator) PPMC
> > Jun Gao 高俊
> > gaojun2048@gmail.com
> >
>
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: Re: [Poprosal] DolphinScheduler Plugin
Posted by "lgcareer2019@outlook.com" <lg...@outlook.com>.
@jun gao
So great to decouple the upper core logic of DS from the specific implementation.
I think this design is more scalable and developers can cooperate in develop more easily.So I agree with it.
@wu shaoj @changqun
I saw you may be have other options that flexible plugin mechanism is complex and in a low priority.
I think we can redesign the alert service in this time first,and I saw jun gao have already done a lot of works.
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： wu shaoj
发送时间： 2020-06-04 17:44
收件人： dev@dolphinscheduler.apache.org
主题： Re: 答复: [Poprosal] DolphinScheduler Plugin
Yes, we should forcus on the job with high priority .
No need to satisfy all customer requirement which might be unreasonable
On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
Very detailed design.
1. About alert module, I agree with @许昌群.
Feedback from the community on the alert module, the alert module function
is relatively stable,
and the current notification function plus webhook whether it can meet the
needs.
2.It is not recommended that the UI be modularized, this will increase the
complexity of module development and debugging,
If there is more complicated UI interaction, it will cause trouble to the
development, Not only need to understand UI development,
but also understand modular configuration
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> Hi folks,
>
> I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> alert way like webhook(pls correct me if I'm wrong), which we can support
> as an official plugin.
>
> The difference between webhook and plugin is you should start a server to
> accept alert request in webhook, but in plugin way you don't need to do
> that.
>
> IMO, I prefer implement some official plugins in AlertServer package
> directly, as for others, we could put it in a plugin folder. I've
> implemented a plugin class loader in DS common package, which used to load
> plugin and avoid package conflict. Put official plugins directly in
> AlertServer package can use the common packages, and will cost less memory
> for class meta data.
>
> But put all plugin into a specific folder is fine too, because it looks
> clean.
>
> BTW, I think there is no conflict to support both webhook and plugin.
>
> Thanks,
> Han Gao
>
> ________________________________
> From: wu shaoj <ga...@apache.org>
> Sent: Thursday, June 4, 2020 15:13
> To: dev@dolphinscheduler.apache.org <de...@dolphinscheduler.apache.org>;
> 许昌群 <ch...@17zuoye.com>
> Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
>
> Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low
> priority I think.
>
> On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
>
> Hi all
>
> Regarding the alarm, I have some other ideas, please refer to it.
>
> At present, there are many types of message notification components,
> such as email/WeChat/SMS, or some notification components developed by the
> company.If it is not realistic to connect all the components, can we change
> the way of thinking and flip this process? We provide an interface for
> sending information, allowing users to interface with it.
>
> Described as follows:
> 1. Define the sending interface of the message in DS, such as sendMsg
> 2. Load the user-configured interface implementation, for example,
> http implementation is http://192.168.1.1:8080/sendMsg/email
> 3. Define the message sending format, {"tos":"softxcq@sina.com",
> "content":"taskA failed."}
> 4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={
> "tos":"softxcq@sina.com", "content":"taskA failed."}
>
> If there are other alarm methods, add an http interface
>
> Finally, you only need to provide several default alarm functions in
> DS to transfer other message notification functions to users.Although you
> will lose some ease of use out of the box, it will increase flexibility.
> You can implement your own system to ensure that the DS alarm logic is
> simple and easy to use.
>
>
> ________________________________
> 发件人: JUN GAO <ga...@gmail.com>
> 发送时间: 2020年6月3日 12:13:21
> 收件人: dev
> 主题: Re: [Poprosal] DolphinScheduler Plugin
>
> 1、Alert is very independent and common function, why should we to
> implement
> it as plugin?
>
> I think Alert should be divided into two parts. One is that the server
> collects various alarm information. The other is to pass these warning
> contents through optional methods (such as email, WeChat). I think the
> specific alarm method should not be implemented by the AlertServer
> server.
> It should be abstracted into an interface and implemented by a concrete
> plug-in implementation layer. If someone understands and is familiar
> with
> other alarm methods, he can implement other alarm plug-ins based on the
> plug-in design. If he wants, he can contribute this plug-in to us.
> Plug-inization is just a way to decouple projects.
>
> 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty? If
> it is a thirdparty, it may generally be managed by its own git, and it
> will
> not be placed in the dolphinscheduler related directory.
>
> Plug-inization is just a way to decouple projects. All Plugin is part
> of dolphinscheduler.
>
> And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> @daolidong We can FYI this.
>
> Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
>
> > Good proposal, but I have some question as following:
> > 1、Alert is very independent and common function, why should we to
> > implement it as plugin?
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty?
> > If it is a thirdparty, it may generally be managed by its own git,
> and it
> > will not be placed in the dolphinscheduler related directory.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) Committer
> > Xiaochun Liu 刘小春
> > liuxiaochun@apache.org
> > ---------------
> >
> >
> >
> > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > >
> > > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> >
>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by wu shaoj <ga...@apache.org>.
+1
On 2020/6/5, 11:51, "JUN GAO" <ga...@gmail.com> wrote:
Yes, Function development is very important, but I think the adjustment of
the architecture is a work that should be more valued. Because the more
work is done on an inappropriate architecture, the higher the cost of
future transformation.
The current code architecture is not friendly to the scalability of DS, I
think we need to do more work in this area. And plug-in is also a very
important feature of DS.
wu shaoj <ga...@apache.org> 于2020年6月4日周四 下午5:50写道：
> Yes, we should forcus on the job with high priority .
> No need to satisfy all customer requirement which might be unreasonable
>
> On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
>
> Very detailed design.
> 1. About alert module, I agree with @许昌群.
> Feedback from the community on the alert module, the alert module
> function
> is relatively stable,
> and the current notification function plus webhook whether it can meet
> the
> needs.
>
> 2.It is not recommended that the UI be modularized, this will increase
> the
> complexity of module development and debugging,
> If there is more complicated UI interaction, it will cause trouble to
> the
> development, Not only need to understand UI development,
> but also understand modular configuration
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
>
> > Hi folks,
> >
> > I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> > alert way like webhook(pls correct me if I'm wrong), which we can
> support
> > as an official plugin.
> >
> > The difference between webhook and plugin is you should start a
> server to
> > accept alert request in webhook, but in plugin way you don't need to
> do
> > that.
> >
> > IMO, I prefer implement some official plugins in AlertServer package
> > directly, as for others, we could put it in a plugin folder. I've
> > implemented a plugin class loader in DS common package, which used
> to load
> > plugin and avoid package conflict. Put official plugins directly in
> > AlertServer package can use the common packages, and will cost less
> memory
> > for class meta data.
> >
> > But put all plugin into a specific folder is fine too, because it
> looks
> > clean.
> >
> > BTW, I think there is no conflict to support both webhook and plugin.
> >
> > Thanks,
> > Han Gao
> >
> > ________________________________
> > From: wu shaoj <ga...@apache.org>
> > Sent: Thursday, June 4, 2020 15:13
> > To: dev@dolphinscheduler.apache.org <dev@dolphinscheduler.apache.org
> >;
> > 许昌群 <ch...@17zuoye.com>
> > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> >
> > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in
> a low
> > priority I think.
> >
> > On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
> >
> > Hi all
> >
> > Regarding the alarm, I have some other ideas, please refer to it.
> >
> > At present, there are many types of message notification
> components,
> > such as email/WeChat/SMS, or some notification components developed
> by the
> > company.If it is not realistic to connect all the components, can we
> change
> > the way of thinking and flip this process? We provide an interface
> for
> > sending information, allowing users to interface with it.
> >
> > Described as follows:
> > 1. Define the sending interface of the message in DS, such as
> sendMsg
> > 2. Load the user-configured interface implementation, for
> example,
> > http implementation is http://192.168.1.1:8080/sendMsg/email
> > 3. Define the message sending format, {"tos":"softxcq@sina.com",
> > "content":"taskA failed."}
> > 4. Send http request.
> http://192.168.1.1:8080/sendMsg/email?data={
> > "tos":"softxcq@sina.com", "content":"taskA failed."}
> >
> > If there are other alarm methods, add an http interface
> >
> > Finally, you only need to provide several default alarm
> functions in
> > DS to transfer other message notification functions to
> users.Although you
> > will lose some ease of use out of the box, it will increase
> flexibility.
> > You can implement your own system to ensure that the DS alarm logic
> is
> > simple and easy to use.
> >
> >
> > ________________________________
> > 发件人: JUN GAO <ga...@gmail.com>
> > 发送时间: 2020年6月3日 12:13:21
> > 收件人: dev
> > 主题: Re: [Poprosal] DolphinScheduler Plugin
> >
> > 1、Alert is very independent and common function, why should we to
> > implement
> > it as plugin?
> >
> > I think Alert should be divided into two parts. One is that the
> server
> > collects various alarm information. The other is to pass these
> warning
> > contents through optional methods (such as email, WeChat). I
> think the
> > specific alarm method should not be implemented by the
> AlertServer
> > server.
> > It should be abstracted into an interface and implemented by a
> concrete
> > plug-in implementation layer. If someone understands and is
> familiar
> > with
> > other alarm methods, he can implement other alarm plug-ins based
> on the
> > plug-in design. If he wants, he can contribute this plug-in to
> us.
> > Plug-inization is just a way to decouple projects.
> >
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty? If
> > it is a thirdparty, it may generally be managed by its own git,
> and it
> > will
> > not be placed in the dolphinscheduler related directory.
> >
> > Plug-inization is just a way to decouple projects. All Plugin is
> part
> > of dolphinscheduler.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> >
> > @daolidong We can FYI this.
> >
> > Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> >
> > > Good proposal, but I have some question as following:
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement it as plugin?
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty?
> > > If it is a thirdparty, it may generally be managed by its own
> git,
> > and it
> > > will not be placed in the dolphinscheduler related directory.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> > >
> > > Best Regards
> > > ---------------
> > > DolphinScheduler(Incubator) Committer
> > > Xiaochun Liu 刘小春
> > > liuxiaochun@apache.org
> > > ---------------
> > >
> > >
> > >
> > > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > > >
> > > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > >
> >
> >
> > --
> >
> > DolphinScheduler(Incubator) PPMC
> > Jun Gao 高俊
> > gaojun2048@gmail.com
> >
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by JUN GAO <ga...@gmail.com>.
Yes, Function development is very important, but I think the adjustment of
the architecture is a work that should be more valued. Because the more
work is done on an inappropriate architecture, the higher the cost of
future transformation.
The current code architecture is not friendly to the scalability of DS, I
think we need to do more work in this area. And plug-in is also a very
important feature of DS.
wu shaoj <ga...@apache.org> 于2020年6月4日周四 下午5:50写道：
> Yes, we should forcus on the job with high priority .
> No need to satisfy all customer requirement which might be unreasonable
>
> On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
>
> Very detailed design.
> 1. About alert module, I agree with @许昌群.
> Feedback from the community on the alert module, the alert module
> function
> is relatively stable,
> and the current notification function plus webhook whether it can meet
> the
> needs.
>
> 2.It is not recommended that the UI be modularized, this will increase
> the
> complexity of module development and debugging,
> If there is more complicated UI interaction, it will cause trouble to
> the
> development, Not only need to understand UI development,
> but also understand modular configuration
>
> --------------------
> DolphinScheduler(Incubator) Commtter
> Hemin Wen 温合民
> wenhemin@apache.org
> --------------------
>
>
> Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
>
> > Hi folks,
> >
> > I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> > alert way like webhook(pls correct me if I'm wrong), which we can
> support
> > as an official plugin.
> >
> > The difference between webhook and plugin is you should start a
> server to
> > accept alert request in webhook, but in plugin way you don't need to
> do
> > that.
> >
> > IMO, I prefer implement some official plugins in AlertServer package
> > directly, as for others, we could put it in a plugin folder. I've
> > implemented a plugin class loader in DS common package, which used
> to load
> > plugin and avoid package conflict. Put official plugins directly in
> > AlertServer package can use the common packages, and will cost less
> memory
> > for class meta data.
> >
> > But put all plugin into a specific folder is fine too, because it
> looks
> > clean.
> >
> > BTW, I think there is no conflict to support both webhook and plugin.
> >
> > Thanks,
> > Han Gao
> >
> > ________________________________
> > From: wu shaoj <ga...@apache.org>
> > Sent: Thursday, June 4, 2020 15:13
> > To: dev@dolphinscheduler.apache.org <dev@dolphinscheduler.apache.org
> >;
> > 许昌群 <ch...@17zuoye.com>
> > Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
> >
> > Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in
> a low
> > priority I think.
> >
> > On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
> >
> > Hi all
> >
> > Regarding the alarm, I have some other ideas, please refer to it.
> >
> > At present, there are many types of message notification
> components,
> > such as email/WeChat/SMS, or some notification components developed
> by the
> > company.If it is not realistic to connect all the components, can we
> change
> > the way of thinking and flip this process? We provide an interface
> for
> > sending information, allowing users to interface with it.
> >
> > Described as follows:
> > 1. Define the sending interface of the message in DS, such as
> sendMsg
> > 2. Load the user-configured interface implementation, for
> example,
> > http implementation is http://192.168.1.1:8080/sendMsg/email
> > 3. Define the message sending format, {"tos":"softxcq@sina.com",
> > "content":"taskA failed."}
> > 4. Send http request.
> http://192.168.1.1:8080/sendMsg/email?data={
> > "tos":"softxcq@sina.com", "content":"taskA failed."}
> >
> > If there are other alarm methods, add an http interface
> >
> > Finally, you only need to provide several default alarm
> functions in
> > DS to transfer other message notification functions to
> users.Although you
> > will lose some ease of use out of the box, it will increase
> flexibility.
> > You can implement your own system to ensure that the DS alarm logic
> is
> > simple and easy to use.
> >
> >
> > ________________________________
> > 发件人: JUN GAO <ga...@gmail.com>
> > 发送时间: 2020年6月3日 12:13:21
> > 收件人: dev
> > 主题: Re: [Poprosal] DolphinScheduler Plugin
> >
> > 1、Alert is very independent and common function, why should we to
> > implement
> > it as plugin?
> >
> > I think Alert should be divided into two parts. One is that the
> server
> > collects various alarm information. The other is to pass these
> warning
> > contents through optional methods (such as email, WeChat). I
> think the
> > specific alarm method should not be implemented by the
> AlertServer
> > server.
> > It should be abstracted into an interface and implemented by a
> concrete
> > plug-in implementation layer. If someone understands and is
> familiar
> > with
> > other alarm methods, he can implement other alarm plug-ins based
> on the
> > plug-in design. If he wants, he can contribute this plug-in to
> us.
> > Plug-inization is just a way to decouple projects.
> >
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty? If
> > it is a thirdparty, it may generally be managed by its own git,
> and it
> > will
> > not be placed in the dolphinscheduler related directory.
> >
> > Plug-inization is just a way to decouple projects. All Plugin is
> part
> > of dolphinscheduler.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> >
> > @daolidong We can FYI this.
> >
> > Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> >
> > > Good proposal, but I have some question as following:
> > > 1、Alert is very independent and common function, why should we
> to
> > > implement it as plugin?
> > > 2、The plug-in you mentioned is part of dolphinscheduler or a
> > thirdparty?
> > > If it is a thirdparty, it may generally be managed by its own
> git,
> > and it
> > > will not be placed in the dolphinscheduler related directory.
> > >
> > > And we can use jsonschema to solve the dynamic ui, for example:
> > alpacajs(
> > > http://alpacajs.org/ <http://alpacajs.org/>), it is very
> powerful.
> > >
> > > Best Regards
> > > ---------------
> > > DolphinScheduler(Incubator) Committer
> > > Xiaochun Liu 刘小春
> > > liuxiaochun@apache.org
> > > ---------------
> > >
> > >
> > >
> > > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > > >
> > > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > >
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> > >
> >
> >
> > --
> >
> > DolphinScheduler(Incubator) PPMC
> > Jun Gao 高俊
> > gaojun2048@gmail.com
> >
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by wu shaoj <ga...@apache.org>.
Yes, we should forcus on the job with high priority .
No need to satisfy all customer requirement which might be unreasonable
On 2020/6/4, 17:41, "Hemin Wen" <we...@apache.org> wrote:
Very detailed design.
1. About alert module, I agree with @许昌群.
Feedback from the community on the alert module, the alert module function
is relatively stable,
and the current notification function plus webhook whether it can meet the
needs.
2.It is not recommended that the UI be modularized, this will increase the
complexity of module development and debugging,
If there is more complicated UI interaction, it will cause trouble to the
development, Not only need to understand UI development,
but also understand modular configuration
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> Hi folks,
>
> I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> alert way like webhook(pls correct me if I'm wrong), which we can support
> as an official plugin.
>
> The difference between webhook and plugin is you should start a server to
> accept alert request in webhook, but in plugin way you don't need to do
> that.
>
> IMO, I prefer implement some official plugins in AlertServer package
> directly, as for others, we could put it in a plugin folder. I've
> implemented a plugin class loader in DS common package, which used to load
> plugin and avoid package conflict. Put official plugins directly in
> AlertServer package can use the common packages, and will cost less memory
> for class meta data.
>
> But put all plugin into a specific folder is fine too, because it looks
> clean.
>
> BTW, I think there is no conflict to support both webhook and plugin.
>
> Thanks,
> Han Gao
>
> ________________________________
> From: wu shaoj <ga...@apache.org>
> Sent: Thursday, June 4, 2020 15:13
> To: dev@dolphinscheduler.apache.org <de...@dolphinscheduler.apache.org>;
> 许昌群 <ch...@17zuoye.com>
> Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
>
> Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low
> priority I think.
>
> On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
>
> Hi all
>
> Regarding the alarm, I have some other ideas, please refer to it.
>
> At present, there are many types of message notification components,
> such as email/WeChat/SMS, or some notification components developed by the
> company.If it is not realistic to connect all the components, can we change
> the way of thinking and flip this process? We provide an interface for
> sending information, allowing users to interface with it.
>
> Described as follows:
> 1. Define the sending interface of the message in DS, such as sendMsg
> 2. Load the user-configured interface implementation, for example,
> http implementation is http://192.168.1.1:8080/sendMsg/email
> 3. Define the message sending format, {"tos":"softxcq@sina.com",
> "content":"taskA failed."}
> 4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={
> "tos":"softxcq@sina.com", "content":"taskA failed."}
>
> If there are other alarm methods, add an http interface
>
> Finally, you only need to provide several default alarm functions in
> DS to transfer other message notification functions to users.Although you
> will lose some ease of use out of the box, it will increase flexibility.
> You can implement your own system to ensure that the DS alarm logic is
> simple and easy to use.
>
>
> ________________________________
> 发件人: JUN GAO <ga...@gmail.com>
> 发送时间: 2020年6月3日 12:13:21
> 收件人: dev
> 主题: Re: [Poprosal] DolphinScheduler Plugin
>
> 1、Alert is very independent and common function, why should we to
> implement
> it as plugin?
>
> I think Alert should be divided into two parts. One is that the server
> collects various alarm information. The other is to pass these warning
> contents through optional methods (such as email, WeChat). I think the
> specific alarm method should not be implemented by the AlertServer
> server.
> It should be abstracted into an interface and implemented by a concrete
> plug-in implementation layer. If someone understands and is familiar
> with
> other alarm methods, he can implement other alarm plug-ins based on the
> plug-in design. If he wants, he can contribute this plug-in to us.
> Plug-inization is just a way to decouple projects.
>
> 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty? If
> it is a thirdparty, it may generally be managed by its own git, and it
> will
> not be placed in the dolphinscheduler related directory.
>
> Plug-inization is just a way to decouple projects. All Plugin is part
> of dolphinscheduler.
>
> And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> @daolidong We can FYI this.
>
> Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
>
> > Good proposal, but I have some question as following:
> > 1、Alert is very independent and common function, why should we to
> > implement it as plugin?
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty?
> > If it is a thirdparty, it may generally be managed by its own git,
> and it
> > will not be placed in the dolphinscheduler related directory.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) Committer
> > Xiaochun Liu 刘小春
> > liuxiaochun@apache.org
> > ---------------
> >
> >
> >
> > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > >
> > > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> >
>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by Hemin Wen <we...@apache.org>.
Very detailed design.
1. About alert module, I agree with @许昌群.
Feedback from the community on the alert module, the alert module function
is relatively stable,
and the current notification function plus webhook whether it can meet the
needs.
2.It is not recommended that the UI be modularized, this will increase the
complexity of module development and debugging,
If there is more complicated UI interaction, it will cause trouble to the
development, Not only need to understand UI development,
but also understand modular configuration
--------------------
DolphinScheduler(Incubator) Commtter
Hemin Wen 温合民
wenhemin@apache.org
--------------------
Han Gao <dh...@hotmail.com> 于2020年6月4日周四 下午4:42写道：
> Hi folks,
>
> I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an
> alert way like webhook(pls correct me if I'm wrong), which we can support
> as an official plugin.
>
> The difference between webhook and plugin is you should start a server to
> accept alert request in webhook, but in plugin way you don't need to do
> that.
>
> IMO, I prefer implement some official plugins in AlertServer package
> directly, as for others, we could put it in a plugin folder. I've
> implemented a plugin class loader in DS common package, which used to load
> plugin and avoid package conflict. Put official plugins directly in
> AlertServer package can use the common packages, and will cost less memory
> for class meta data.
>
> But put all plugin into a specific folder is fine too, because it looks
> clean.
>
> BTW, I think there is no conflict to support both webhook and plugin.
>
> Thanks,
> Han Gao
>
> ________________________________
> From: wu shaoj <ga...@apache.org>
> Sent: Thursday, June 4, 2020 15:13
> To: dev@dolphinscheduler.apache.org <de...@dolphinscheduler.apache.org>;
> 许昌群 <ch...@17zuoye.com>
> Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
>
> Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low
> priority I think.
>
> On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
>
> Hi all
>
> Regarding the alarm, I have some other ideas, please refer to it.
>
> At present, there are many types of message notification components,
> such as email/WeChat/SMS, or some notification components developed by the
> company.If it is not realistic to connect all the components, can we change
> the way of thinking and flip this process? We provide an interface for
> sending information, allowing users to interface with it.
>
> Described as follows:
> 1. Define the sending interface of the message in DS, such as sendMsg
> 2. Load the user-configured interface implementation, for example,
> http implementation is http://192.168.1.1:8080/sendMsg/email
> 3. Define the message sending format, {"tos":"softxcq@sina.com",
> "content":"taskA failed."}
> 4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={
> "tos":"softxcq@sina.com", "content":"taskA failed."}
>
> If there are other alarm methods, add an http interface
>
> Finally, you only need to provide several default alarm functions in
> DS to transfer other message notification functions to users.Although you
> will lose some ease of use out of the box, it will increase flexibility.
> You can implement your own system to ensure that the DS alarm logic is
> simple and easy to use.
>
>
> ________________________________
> 发件人: JUN GAO <ga...@gmail.com>
> 发送时间: 2020年6月3日 12:13:21
> 收件人: dev
> 主题: Re: [Poprosal] DolphinScheduler Plugin
>
> 1、Alert is very independent and common function, why should we to
> implement
> it as plugin?
>
> I think Alert should be divided into two parts. One is that the server
> collects various alarm information. The other is to pass these warning
> contents through optional methods (such as email, WeChat). I think the
> specific alarm method should not be implemented by the AlertServer
> server.
> It should be abstracted into an interface and implemented by a concrete
> plug-in implementation layer. If someone understands and is familiar
> with
> other alarm methods, he can implement other alarm plug-ins based on the
> plug-in design. If he wants, he can contribute this plug-in to us.
> Plug-inization is just a way to decouple projects.
>
> 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty? If
> it is a thirdparty, it may generally be managed by its own git, and it
> will
> not be placed in the dolphinscheduler related directory.
>
> Plug-inization is just a way to decouple projects. All Plugin is part
> of dolphinscheduler.
>
> And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> @daolidong We can FYI this.
>
> Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
>
> > Good proposal, but I have some question as following:
> > 1、Alert is very independent and common function, why should we to
> > implement it as plugin?
> > 2、The plug-in you mentioned is part of dolphinscheduler or a
> thirdparty?
> > If it is a thirdparty, it may generally be managed by its own git,
> and it
> > will not be placed in the dolphinscheduler related directory.
> >
> > And we can use jsonschema to solve the dynamic ui, for example:
> alpacajs(
> > http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) Committer
> > Xiaochun Liu 刘小春
> > liuxiaochun@apache.org
> > ---------------
> >
> >
> >
> > > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> > >
> > > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
> >
>
>
> --
>
> DolphinScheduler(Incubator) PPMC
> Jun Gao 高俊
> gaojun2048@gmail.com
>
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by Han Gao <dh...@hotmail.com>.
Hi folks,
I prefer plugin way. @许昌群<ma...@17zuoye.com> provide an alert way like webhook(pls correct me if I'm wrong), which we can support as an official plugin.
The difference between webhook and plugin is you should start a server to accept alert request in webhook, but in plugin way you don't need to do that.
IMO, I prefer implement some official plugins in AlertServer package directly, as for others, we could put it in a plugin folder. I've implemented a plugin class loader in DS common package, which used to load plugin and avoid package conflict. Put official plugins directly in AlertServer package can use the common packages, and will cost less memory for class meta data.
But put all plugin into a specific folder is fine too, because it looks clean.
BTW, I think there is no conflict to support both webhook and plugin.
Thanks,
Han Gao
________________________________
From: wu shaoj <ga...@apache.org>
Sent: Thursday, June 4, 2020 15:13
To: dev@dolphinscheduler.apache.org <de...@dolphinscheduler.apache.org>; 许昌群 <ch...@17zuoye.com>
Subject: Re: 答复: [Poprosal] DolphinScheduler Plugin
Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low priority I think.
On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
Hi all
Regarding the alarm, I have some other ideas, please refer to it.
At present, there are many types of message notification components, such as email/WeChat/SMS, or some notification components developed by the company.If it is not realistic to connect all the components, can we change the way of thinking and flip this process? We provide an interface for sending information, allowing users to interface with it.
Described as follows:
1. Define the sending interface of the message in DS, such as sendMsg
2. Load the user-configured interface implementation, for example, http implementation is http://192.168.1.1:8080/sendMsg/email
3. Define the message sending format, {"tos":"softxcq@sina.com", "content":"taskA failed."}
4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={"tos":"softxcq@sina.com", "content":"taskA failed."}
If there are other alarm methods, add an http interface
Finally, you only need to provide several default alarm functions in DS to transfer other message notification functions to users.Although you will lose some ease of use out of the box, it will increase flexibility. You can implement your own system to ensure that the DS alarm logic is simple and easy to use.
________________________________
发件人: JUN GAO <ga...@gmail.com>
发送时间: 2020年6月3日 12:13:21
收件人: dev
主题: Re: [Poprosal] DolphinScheduler Plugin
1、Alert is very independent and common function, why should we to implement
it as plugin?
I think Alert should be divided into two parts. One is that the server
collects various alarm information. The other is to pass these warning
contents through optional methods (such as email, WeChat). I think the
specific alarm method should not be implemented by the AlertServer server.
It should be abstracted into an interface and implemented by a concrete
plug-in implementation layer. If someone understands and is familiar with
other alarm methods, he can implement other alarm plug-ins based on the
plug-in design. If he wants, he can contribute this plug-in to us.
Plug-inization is just a way to decouple projects.
2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty? If
it is a thirdparty, it may generally be managed by its own git, and it will
not be placed in the dolphinscheduler related directory.
Plug-inization is just a way to decouple projects. All Plugin is part
of dolphinscheduler.
And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
@daolidong We can FYI this.
Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> Good proposal, but I have some question as following:
> 1、Alert is very independent and common function, why should we to
> implement it as plugin?
> 2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty?
> If it is a thirdparty, it may generally be managed by its own git, and it
> will not be placed in the dolphinscheduler related directory.
>
> And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org
> ---------------
>
>
>
> > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: 答复: [Poprosal] DolphinScheduler Plugin
Posted by wu shaoj <ga...@apache.org>.
Yes , aggree with @许昌群 , flexible plugin mechanism is complex and in a low priority I think.
On 2020/6/4, 15:04, "许昌群" <ch...@17zuoye.com> wrote:
Hi all
Regarding the alarm, I have some other ideas, please refer to it.
At present, there are many types of message notification components, such as email/WeChat/SMS, or some notification components developed by the company.If it is not realistic to connect all the components, can we change the way of thinking and flip this process? We provide an interface for sending information, allowing users to interface with it.
Described as follows:
1. Define the sending interface of the message in DS, such as sendMsg
2. Load the user-configured interface implementation, for example, http implementation is http://192.168.1.1:8080/sendMsg/email
3. Define the message sending format, {"tos":"softxcq@sina.com", "content":"taskA failed."}
4. Send http request. http://192.168.1.1:8080/sendMsg/email?data={"tos":"softxcq@sina.com", "content":"taskA failed."}
If there are other alarm methods, add an http interface
Finally, you only need to provide several default alarm functions in DS to transfer other message notification functions to users.Although you will lose some ease of use out of the box, it will increase flexibility. You can implement your own system to ensure that the DS alarm logic is simple and easy to use.
________________________________
发件人: JUN GAO <ga...@gmail.com>
发送时间: 2020年6月3日 12:13:21
收件人: dev
主题: Re: [Poprosal] DolphinScheduler Plugin
1、Alert is very independent and common function, why should we to implement
it as plugin?
I think Alert should be divided into two parts. One is that the server
collects various alarm information. The other is to pass these warning
contents through optional methods (such as email, WeChat). I think the
specific alarm method should not be implemented by the AlertServer server.
It should be abstracted into an interface and implemented by a concrete
plug-in implementation layer. If someone understands and is familiar with
other alarm methods, he can implement other alarm plug-ins based on the
plug-in design. If he wants, he can contribute this plug-in to us.
Plug-inization is just a way to decouple projects.
2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty? If
it is a thirdparty, it may generally be managed by its own git, and it will
not be placed in the dolphinscheduler related directory.
Plug-inization is just a way to decouple projects. All Plugin is part
of dolphinscheduler.
And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
@daolidong We can FYI this.
Xiaochun Liu <li...@apache.org> 于2020年6月3日周三 上午11:35写道：
> Good proposal, but I have some question as following:
> 1、Alert is very independent and common function, why should we to
> implement it as plugin?
> 2、The plug-in you mentioned is part of dolphinscheduler or a thirdparty?
> If it is a thirdparty, it may generally be managed by its own git, and it
> will not be placed in the dolphinscheduler related directory.
>
> And we can use jsonschema to solve the dynamic ui, for example: alpacajs(
> http://alpacajs.org/ <http://alpacajs.org/>), it is very powerful.
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org
> ---------------
>
>
>
> > 2020年6月2日 下午3:04，JUN GAO <ga...@gmail.com> 写道：
> >
> > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1 <
> https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1>
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com