You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by qiao zhanwei <qi...@outlook.com> on 2019/12/10 06:24:12 UTC
Aproposal for DolphinScheduler Simplified Deployment
Hello All ,
Now DolphinScheduler has so many Configuration files
for example :
dolphinscheduler-alert :
alert.properties
dolphinscheduler-api :
application-api.properties
application-combined.properties
dolphinscheduler-common :
hadoop.properties
common.properties
quartz.properties
zookeeper.properties
dolphinscheduler-dao :
application-dao.properties
dolphinscheduler-server :
application-master.properties
application-master.properties
master.properties
worker.properties
.dolphinscheduler_env.sh
Can we simplify deployment ?
Main point :
1 configuration file simplification and merged configuration file
2 master server remove port
3 support offline installation，remove kazoo dependencies in install and monitor
4 instll.sh script simplification
―――――――――――――
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
Re: Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by qiao zhanwei <qi...@outlook.com>.
DS does not support offline installation now, because kazoo needs to be installed, remove kazoo,
and use Java programs. What do you think?
―――――――――――――
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
From: 李 岗<ma...@outlook.com>
Date: 2019-12-11 18:23
To: dev<ma...@dolphinscheduler.apache.org>
Subject: Re: Re: Aproposal for DolphinScheduler Simplified Deployment
I think from another angle,Master and Worker as key services,I think not to redeployment during normal execution.
If tasks are still running,redeploy master and worker may be lead to missed the timed task.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: guo jiwei<ma...@gmail.com>
Date: 2019-12-11 18:11
To: dev<ma...@dolphinscheduler.apache.org>
Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
To ligang.
redeploy is simple, but what about the latency of alert ?
it's easy to redeploy master server to update alert
On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
> I think the alert module can be retained. Currently, it only supports
> email and webchat,
> but more alarm modes can be added in the future.
>
> At present，alert is a independent service. the alert service only consumes
> alarm information in the database, other services produce these alarm
> information.
> If a new alarm mode is added, It is only necessary to redeploy the alert
> service.
>
>
>
> ________________________________
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> 发件人： qiao zhanwei<ma...@outlook.com>
> 发送时间： 2019-12-10 14:24
> 收件人： dev<ma...@dolphinscheduler.apache.org>
> 主题： Aproposal for DolphinScheduler Simplified Deployment
>
> Hello All ,
>
> Now DolphinScheduler has so many Configuration files
>
> for example :
>
> dolphinscheduler-alert :
> alert.properties
>
> dolphinscheduler-api :
> application-api.properties
> application-combined.properties
>
>
> dolphinscheduler-common :
> hadoop.properties
> common.properties
> quartz.properties
> zookeeper.properties
>
> dolphinscheduler-dao :
> application-dao.properties
>
> dolphinscheduler-server :
> application-master.properties
> application-master.properties
> master.properties
> worker.properties
>
> .dolphinscheduler_env.sh
>
> Can we simplify deployment ?
>
> Main point :
>
> 1 configuration file simplification and merged configuration file
> 2 master server remove port
> 3 support offline installation，remove kazoo dependencies in install and
> monitor
> 4 instll.sh script simplification
>
>
> ―――――――――――――
> DolphinScheduler(Incubator) PPMC
> Zhanwei Qiao 乔占卫
>
> qiaozhanwei@outlook.com
>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by chendapao <fe...@163.com>.
Hello：
I have developed a set of procedures for simplifying operation and maintenance for DS.
I maintain the TiDB cluster and CM/CDH cluster daily, and got some inspiration.
I use Ansible to develop this program.
My current idea is based on the DevOps, and does not make any adjustments to the architecture.
Features:
Install all roles fully automatically (JDK, DB, ZK, BIG DATA relate, DS relate, LOG collect)
Install online / Install offline (The offline way is to manually download the package I prepared no need verification, Just put it in the specified directory)
Start and stop, various roles.
Remote role configuration modification.
Butt BIG DATA.
Cluster rolling upgrade, Cluster single role upgrade.
Cluster expansion and reduction. (such as switch from cluster to single node)
Log collect based on GaryLog.
Destroy cluster.
Adjustment:
Adjust configuration files to two categories, main and other. The main thing is to run, others are used for tuning. (No add or del conf, just tell users to conf priority)
Ignore previous install.
Use ZK native method instead of Kazoo.
Added original patches for start and stop scripts.
Advantage:
The previous installation method is only installation, The user cannot get the correct error prompt during installation or other operations.This program can automatically avoid various minor problems during installation and will tell the user what the error is.
Prepare for future rolling upgrades.
When a user needs to troubleshoot or upgrade, he can simply and comprehensively describe his cluster status.
These are what I currently think, Already implement [Automatic installation 70%] [Remote role configuration modification 100%] [Start and stop, various roles 100%], This is the address of the program https://github.com/feloxx/ds-yibasuo, Welcome everyone to criticize and fix.
chendapao
feloxx@163.com
On 12/11/2019 20:26，leon bao<le...@apache.org> wrote：
As an open source project, I think it's important to stay open and
extensible.
Our current alert does not have many shortcomings except for scanning the
database with one thread, at the same time, the alert service as a separate
module provides better scalability.
So I don't think it is necessary to merge alert into other modules without
much benefit.
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
To xiaochun.
it's not a good way.
Alert must be trigger by who is scheduling the task, in DS, it's
MasterServer. Because, only by who is scheduling the task, it can know the
task status in time. If the task is timeout, trigger timeout alert in time
is very important for users.
this is why we have to move alert into server module.
Alert implementation will be refactor in the future. not scan db anymore.
On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
wrote:
To guo jiwei:
Why not put together with the api server, the alert server function is
very small,
and the load will not be very high. If logs are stored together in the
future,
we can combine alert server, log server, api server together, these can
be
called management server.
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
To ligang.
it's right.
But alert server is only a small function. we define it as an
individual module and as a server. do you thing alert is expensive or
taking more resource ? if not , why a single module ?
And alert server trigger task event by scanning db, do you think it
is a
nice way ?
Moving into server module is only our first step for simplifying user
deployment. Extension of alert can be updated via redeploy server and
it's
not a frequent operation.
As the architecture changes, alert implementation will change.
On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com> wrote:
I think from another angle,Master and Worker as key services,I think
not
to redeployment during normal execution.
If tasks are still running,redeploy master and worker may be lead to
missed the timed task.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: guo jiwei<ma...@gmail.com>
Date: 2019-12-11 18:11
To: dev<ma...@dolphinscheduler.apache.org>
Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
To ligang.
redeploy is simple, but what about the latency of alert ?
it's easy to redeploy master server to update alert
On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
I think the alert module can be retained. Currently, it only supports
email and webchat,
but more alarm modes can be added in the future.
At present，alert is a independent service. the alert service only
consumes
alarm information in the database, other services produce these alarm
information.
If a new alarm mode is added, It is only necessary to redeploy the
alert
service.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： qiao zhanwei<ma...@outlook.com>
发送时间： 2019-12-10 14:24
收件人： dev<ma...@dolphinscheduler.apache.org>
主题： Aproposal for DolphinScheduler Simplified Deployment
Hello All ,
Now DolphinScheduler has so many Configuration files
for example :
dolphinscheduler-alert :
alert.properties
dolphinscheduler-api :
application-api.properties
application-combined.properties
dolphinscheduler-common :
hadoop.properties
common.properties
quartz.properties
zookeeper.properties
dolphinscheduler-dao :
application-dao.properties
dolphinscheduler-server :
application-master.properties
application-master.properties
master.properties
worker.properties
.dolphinscheduler_env.sh
Can we simplify deployment ?
Main point :
1 configuration file simplification and merged configuration file
2 master server remove port
3 support offline installation，remove kazoo dependencies in install
and
monitor
4 instll.sh script simplification
—————————————
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
To xiaochun.
it's not a good way.
Alert must be trigger by who is scheduling the task, in DS, it's
MasterServer. Because, only by who is scheduling the task, it can know the
task status in time. If the task is timeout, trigger timeout alert in time
is very important for users.
this is why we have to move alert into server module.
Alert implementation will be refactor in the future. not scan db anymore.
On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
wrote:
To guo jiwei:
Why not put together with the api server, the alert server function is
very small,
and the load will not be very high. If logs are stored together in the
future,
we can combine alert server, log server, api server together, these can
be
called management server.
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
To ligang.
it's right.
But alert server is only a small function. we define it as an
individual module and as a server. do you thing alert is expensive or
taking more resource ? if not , why a single module ?
And alert server trigger task event by scanning db, do you think it
is a
nice way ?
Moving into server module is only our first step for simplifying user
deployment. Extension of alert can be updated via redeploy server and
it's
not a frequent operation.
As the architecture changes, alert implementation will change.
On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com> wrote:
I think from another angle,Master and Worker as key services,I think
not
to redeployment during normal execution.
If tasks are still running,redeploy master and worker may be lead to
missed the timed task.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: guo jiwei<ma...@gmail.com>
Date: 2019-12-11 18:11
To: dev<ma...@dolphinscheduler.apache.org>
Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
To ligang.
redeploy is simple, but what about the latency of alert ?
it's easy to redeploy master server to update alert
On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
I think the alert module can be retained. Currently, it only supports
email and webchat,
but more alarm modes can be added in the future.
At present，alert is a independent service. the alert service only
consumes
alarm information in the database, other services produce these alarm
information.
If a new alarm mode is added, It is only necessary to redeploy the
alert
service.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： qiao zhanwei<ma...@outlook.com>
发送时间： 2019-12-10 14:24
收件人： dev<ma...@dolphinscheduler.apache.org>
主题： Aproposal for DolphinScheduler Simplified Deployment
Hello All ,
Now DolphinScheduler has so many Configuration files
for example :
dolphinscheduler-alert :
alert.properties
dolphinscheduler-api :
application-api.properties
application-combined.properties
dolphinscheduler-common :
hadoop.properties
common.properties
quartz.properties
zookeeper.properties
dolphinscheduler-dao :
application-dao.properties
dolphinscheduler-server :
application-master.properties
application-master.properties
master.properties
worker.properties
.dolphinscheduler_env.sh
Can we simplify deployment ?
Main point :
1 configuration file simplification and merged configuration file
2 master server remove port
3 support offline installation，remove kazoo dependencies in install
and
monitor
4 instll.sh script simplification
—————————————
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by chendapao <fe...@163.com>.
Hello：
Adjustment of the configuration file, I have an idea.
Do not adjust the current configuration structure, just tell the user the priority of the configuration.
Divided into two categories, Main and Other.
Main:
run necessary
Other:
run tuning
---
I am in the ds-yibasuo project, Define the following as Main:
- roles ip or host
- dolphinscheduler_version
- deploy_dir
- ansible_user
Other, All use default.
chendapao
feloxx@163.com
On 12/13/2019 13:35，chendapao<fe...@163.com> wrote：
Hello：
Adjustment of the configuration file, I have an idea.
Do not adjust the current configuration structure, just tell the user the priority of the configuration.
Divided into two categories, Main and Other.
Main:
run necessary
Other:
run tuning
---
I am in the ds-yibasuo project, Define the following as Main:
- roles ip or host
- dolphinscheduler_version
- deploy_dir
- ansible_user
Other, All use default.
chendapao
feloxx@163.com
On 12/12/2019 18:45，JUN GAO<ga...@gmail.com> wrote：
I have some idea:
1. Remove the alert service and change it to a tool jar package. Whether it
is the master or the worker, who needs to send a message calls the alert
interface.
2. The alert service no longer scans the db and provides an interface. What
is sent is determined by the caller. The alert need to saving the alert
message and sending the message via email or WeChat.
3. If the master, worker, or other services can call the alert toolkit to
send messages, the alert can exist as an independent module, otherwise it
is not suitable for the common layer, the dao layer, or the service layer.
And if want to extend the function of alert, only need to modify the alert
module.
By doing this, we can reduce one service(alert) and simplify deployment.
What do you guys think？
guo jiwei <te...@gmail.com> 于2019年12月12日周四 上午10:14写道：
To baoliang：
Is alert server distributed ? it's a standalone module with only a
function scanning db and triggering alert .
Trigger event will delay if in an individual module even not scan db.
do you have some good idea to let alert trigger in time ? do you get my
point of in time ?
On Wed, Dec 11, 2019 at 8:26 PM leon bao <le...@apache.org> wrote:
As an open source project, I think it's important to stay open and
extensible.
Our current alert does not have many shortcomings except for scanning the
database with one thread, at the same time, the alert service as a
separate
module provides better scalability.
So I don't think it is necessary to merge alert into other modules
without
much benefit.
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
To xiaochun.
it's not a good way.
Alert must be trigger by who is scheduling the task, in DS, it's
MasterServer. Because, only by who is scheduling the task, it can know
the
task status in time. If the task is timeout, trigger timeout alert in
time
is very important for users.
this is why we have to move alert into server module.
Alert implementation will be refactor in the future. not scan db
anymore.
On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
wrote:
To guo jiwei:
Why not put together with the api server, the alert server function
is
very small,
and the load will not be very high. If logs are stored together in
the
future,
we can combine alert server, log server, api server together, these
can
be
called management server.
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
To ligang.
it's right.
But alert server is only a small function. we define it as an
individual module and as a server. do you thing alert is
expensive
or
taking more resource ? if not , why a single module ?
And alert server trigger task event by scanning db, do you think
it
is a
nice way ?
Moving into server module is only our first step for simplifying
user
deployment. Extension of alert can be updated via redeploy server
and
it's
not a frequent operation.
As the architecture changes, alert implementation will change.
On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com>
wrote:
I think from another angle,Master and Worker as key services,I
think
not
to redeployment during normal execution.
If tasks are still running,redeploy master and worker may be lead
to
missed the timed task.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: guo jiwei<ma...@gmail.com>
Date: 2019-12-11 18:11
To: dev<ma...@dolphinscheduler.apache.org>
Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
To ligang.
redeploy is simple, but what about the latency of alert ?
it's easy to redeploy master server to update alert
On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com>
wrote:
I think the alert module can be retained. Currently, it only
supports
email and webchat,
but more alarm modes can be added in the future.
At present，alert is a independent service. the alert service only
consumes
alarm information in the database, other services produce these
alarm
information.
If a new alarm mode is added, It is only necessary to redeploy
the
alert
service.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： qiao zhanwei<ma...@outlook.com>
发送时间： 2019-12-10 14:24
收件人： dev<ma...@dolphinscheduler.apache.org>
主题： Aproposal for DolphinScheduler Simplified Deployment
Hello All ,
Now DolphinScheduler has so many Configuration files
for example :
dolphinscheduler-alert :
alert.properties
dolphinscheduler-api :
application-api.properties
application-combined.properties
dolphinscheduler-common :
hadoop.properties
common.properties
quartz.properties
zookeeper.properties
dolphinscheduler-dao :
application-dao.properties
dolphinscheduler-server :
application-master.properties
application-master.properties
master.properties
worker.properties
.dolphinscheduler_env.sh
Can we simplify deployment ?
Main point :
1 configuration file simplification and merged configuration
file
2 master server remove port
3 support offline installation，remove kazoo dependencies in
install
and
monitor
4 instll.sh script simplification
—————————————
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
To xiaochun.
it's not a good way.
Alert must be trigger by who is scheduling the task, in DS, it's
MasterServer. Because, only by who is scheduling the task, it can know
the
task status in time. If the task is timeout, trigger timeout alert in
time
is very important for users.
this is why we have to move alert into server module.
Alert implementation will be refactor in the future. not scan db
anymore.
On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
wrote:
To guo jiwei:
Why not put together with the api server, the alert server function
is
very small,
and the load will not be very high. If logs are stored together in
the
future,
we can combine alert server, log server, api server together, these
can
be
called management server.
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
To ligang.
it's right.
But alert server is only a small function. we define it as an
individual module and as a server. do you thing alert is
expensive
or
taking more resource ? if not , why a single module ?
And alert server trigger task event by scanning db, do you think
it
is a
nice way ?
Moving into server module is only our first step for simplifying
user
deployment. Extension of alert can be updated via redeploy server
and
it's
not a frequent operation.
As the architecture changes, alert implementation will change.
On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com>
wrote:
I think from another angle,Master and Worker as key services,I
think
not
to redeployment during normal execution.
If tasks are still running,redeploy master and worker may be lead
to
missed the timed task.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: guo jiwei<ma...@gmail.com>
Date: 2019-12-11 18:11
To: dev<ma...@dolphinscheduler.apache.org>
Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
To ligang.
redeploy is simple, but what about the latency of alert ?
it's easy to redeploy master server to update alert
On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com>
wrote:
I think the alert module can be retained. Currently, it only
supports
email and webchat,
but more alarm modes can be added in the future.
At present，alert is a independent service. the alert service only
consumes
alarm information in the database, other services produce these
alarm
information.
If a new alarm mode is added, It is only necessary to redeploy
the
alert
service.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： qiao zhanwei<ma...@outlook.com>
发送时间： 2019-12-10 14:24
收件人： dev<ma...@dolphinscheduler.apache.org>
主题： Aproposal for DolphinScheduler Simplified Deployment
Hello All ,
Now DolphinScheduler has so many Configuration files
for example :
dolphinscheduler-alert :
alert.properties
dolphinscheduler-api :
application-api.properties
application-combined.properties
dolphinscheduler-common :
hadoop.properties
common.properties
quartz.properties
zookeeper.properties
dolphinscheduler-dao :
application-dao.properties
dolphinscheduler-server :
application-master.properties
application-master.properties
master.properties
worker.properties
.dolphinscheduler_env.sh
Can we simplify deployment ?
Main point :
1 configuration file simplification and merged configuration
file
2 master server remove port
3 support offline installation，remove kazoo dependencies in
install
and
monitor
4 instll.sh script simplification
—————————————
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by chendapao <fe...@163.com>.
Hello：
Adjustment of the configuration file, I have an idea.
Do not adjust the current configuration structure, just tell the user the priority of the configuration.
Divided into two categories, Main and Other.
Main:
run necessary
Other:
run tuning
---
I am in the ds-yibasuo project, Define the following as Main:
- roles ip or host
- dolphinscheduler_version
- deploy_dir
- ansible_user
Other, All use default.
chendapao
feloxx@163.com
On 12/12/2019 18:45，JUN GAO<ga...@gmail.com> wrote：
I have some idea:
1. Remove the alert service and change it to a tool jar package. Whether it
is the master or the worker, who needs to send a message calls the alert
interface.
2. The alert service no longer scans the db and provides an interface. What
is sent is determined by the caller. The alert need to saving the alert
message and sending the message via email or WeChat.
3. If the master, worker, or other services can call the alert toolkit to
send messages, the alert can exist as an independent module, otherwise it
is not suitable for the common layer, the dao layer, or the service layer.
And if want to extend the function of alert, only need to modify the alert
module.
By doing this, we can reduce one service(alert) and simplify deployment.
What do you guys think？
guo jiwei <te...@gmail.com> 于2019年12月12日周四 上午10:14写道：
To baoliang：
Is alert server distributed ? it's a standalone module with only a
function scanning db and triggering alert .
Trigger event will delay if in an individual module even not scan db.
do you have some good idea to let alert trigger in time ? do you get my
point of in time ?
On Wed, Dec 11, 2019 at 8:26 PM leon bao <le...@apache.org> wrote:
As an open source project, I think it's important to stay open and
extensible.
Our current alert does not have many shortcomings except for scanning the
database with one thread, at the same time, the alert service as a
separate
module provides better scalability.
So I don't think it is necessary to merge alert into other modules
without
much benefit.
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
To xiaochun.
it's not a good way.
Alert must be trigger by who is scheduling the task, in DS, it's
MasterServer. Because, only by who is scheduling the task, it can know
the
task status in time. If the task is timeout, trigger timeout alert in
time
is very important for users.
this is why we have to move alert into server module.
Alert implementation will be refactor in the future. not scan db
anymore.
On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
wrote:
To guo jiwei:
Why not put together with the api server, the alert server function
is
very small,
and the load will not be very high. If logs are stored together in
the
future,
we can combine alert server, log server, api server together, these
can
be
called management server.
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
To ligang.
it's right.
But alert server is only a small function. we define it as an
individual module and as a server. do you thing alert is
expensive
or
taking more resource ? if not , why a single module ?
And alert server trigger task event by scanning db, do you think
it
is a
nice way ?
Moving into server module is only our first step for simplifying
user
deployment. Extension of alert can be updated via redeploy server
and
it's
not a frequent operation.
As the architecture changes, alert implementation will change.
On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com>
wrote:
I think from another angle,Master and Worker as key services,I
think
not
to redeployment during normal execution.
If tasks are still running,redeploy master and worker may be lead
to
missed the timed task.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: guo jiwei<ma...@gmail.com>
Date: 2019-12-11 18:11
To: dev<ma...@dolphinscheduler.apache.org>
Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
To ligang.
redeploy is simple, but what about the latency of alert ?
it's easy to redeploy master server to update alert
On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com>
wrote:
I think the alert module can be retained. Currently, it only
supports
email and webchat,
but more alarm modes can be added in the future.
At present，alert is a independent service. the alert service only
consumes
alarm information in the database, other services produce these
alarm
information.
If a new alarm mode is added, It is only necessary to redeploy
the
alert
service.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： qiao zhanwei<ma...@outlook.com>
发送时间： 2019-12-10 14:24
收件人： dev<ma...@dolphinscheduler.apache.org>
主题： Aproposal for DolphinScheduler Simplified Deployment
Hello All ,
Now DolphinScheduler has so many Configuration files
for example :
dolphinscheduler-alert :
alert.properties
dolphinscheduler-api :
application-api.properties
application-combined.properties
dolphinscheduler-common :
hadoop.properties
common.properties
quartz.properties
zookeeper.properties
dolphinscheduler-dao :
application-dao.properties
dolphinscheduler-server :
application-master.properties
application-master.properties
master.properties
worker.properties
.dolphinscheduler_env.sh
Can we simplify deployment ?
Main point :
1 configuration file simplification and merged configuration
file
2 master server remove port
3 support offline installation，remove kazoo dependencies in
install
and
monitor
4 instll.sh script simplification
—————————————
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
To xiaochun.
it's not a good way.
Alert must be trigger by who is scheduling the task, in DS, it's
MasterServer. Because, only by who is scheduling the task, it can know
the
task status in time. If the task is timeout, trigger timeout alert in
time
is very important for users.
this is why we have to move alert into server module.
Alert implementation will be refactor in the future. not scan db
anymore.
On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
wrote:
To guo jiwei:
Why not put together with the api server, the alert server function
is
very small,
and the load will not be very high. If logs are stored together in
the
future,
we can combine alert server, log server, api server together, these
can
be
called management server.
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
To ligang.
it's right.
But alert server is only a small function. we define it as an
individual module and as a server. do you thing alert is
expensive
or
taking more resource ? if not , why a single module ?
And alert server trigger task event by scanning db, do you think
it
is a
nice way ?
Moving into server module is only our first step for simplifying
user
deployment. Extension of alert can be updated via redeploy server
and
it's
not a frequent operation.
As the architecture changes, alert implementation will change.
On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com>
wrote:
I think from another angle,Master and Worker as key services,I
think
not
to redeployment during normal execution.
If tasks are still running,redeploy master and worker may be lead
to
missed the timed task.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: guo jiwei<ma...@gmail.com>
Date: 2019-12-11 18:11
To: dev<ma...@dolphinscheduler.apache.org>
Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
To ligang.
redeploy is simple, but what about the latency of alert ?
it's easy to redeploy master server to update alert
On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com>
wrote:
I think the alert module can be retained. Currently, it only
supports
email and webchat,
but more alarm modes can be added in the future.
At present，alert is a independent service. the alert service only
consumes
alarm information in the database, other services produce these
alarm
information.
If a new alarm mode is added, It is only necessary to redeploy
the
alert
service.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： qiao zhanwei<ma...@outlook.com>
发送时间： 2019-12-10 14:24
收件人： dev<ma...@dolphinscheduler.apache.org>
主题： Aproposal for DolphinScheduler Simplified Deployment
Hello All ,
Now DolphinScheduler has so many Configuration files
for example :
dolphinscheduler-alert :
alert.properties
dolphinscheduler-api :
application-api.properties
application-combined.properties
dolphinscheduler-common :
hadoop.properties
common.properties
quartz.properties
zookeeper.properties
dolphinscheduler-dao :
application-dao.properties
dolphinscheduler-server :
application-master.properties
application-master.properties
master.properties
worker.properties
.dolphinscheduler_env.sh
Can we simplify deployment ?
Main point :
1 configuration file simplification and merged configuration
file
2 master server remove port
3 support offline installation，remove kazoo dependencies in
install
and
monitor
4 instll.sh script simplification
—————————————
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by JUN GAO <ga...@gmail.com>.
I have some idea:
1. Remove the alert service and change it to a tool jar package. Whether it
is the master or the worker, who needs to send a message calls the alert
interface.
2. The alert service no longer scans the db and provides an interface. What
is sent is determined by the caller. The alert need to saving the alert
message and sending the message via email or WeChat.
3. If the master, worker, or other services can call the alert toolkit to
send messages, the alert can exist as an independent module, otherwise it
is not suitable for the common layer, the dao layer, or the service layer.
And if want to extend the function of alert, only need to modify the alert
module.
By doing this, we can reduce one service(alert) and simplify deployment.
What do you guys think？
guo jiwei <te...@gmail.com> 于2019年12月12日周四 上午10:14写道：
> To baoliang：
> Is alert server distributed ? it's a standalone module with only a
> function scanning db and triggering alert .
> Trigger event will delay if in an individual module even not scan db.
> do you have some good idea to let alert trigger in time ? do you get my
> point of in time ?
>
>
> On Wed, Dec 11, 2019 at 8:26 PM leon bao <le...@apache.org> wrote:
>
> > As an open source project, I think it's important to stay open and
> > extensible.
> > Our current alert does not have many shortcomings except for scanning the
> > database with one thread, at the same time, the alert service as a
> separate
> > module provides better scalability.
> > So I don't think it is necessary to merge alert into other modules
> without
> > much benefit.
> >
> > guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
> >
> > > To xiaochun.
> > > it's not a good way.
> > > Alert must be trigger by who is scheduling the task, in DS, it's
> > > MasterServer. Because, only by who is scheduling the task, it can know
> > the
> > > task status in time. If the task is timeout, trigger timeout alert in
> > time
> > > is very important for users.
> > > this is why we have to move alert into server module.
> > > Alert implementation will be refactor in the future. not scan db
> > anymore.
> > >
> > > On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
> > > wrote:
> > >
> > > > To guo jiwei:
> > > >
> > > > Why not put together with the api server, the alert server function
> is
> > > > very small,
> > > > and the load will not be very high. If logs are stored together in
> the
> > > > future,
> > > > we can combine alert server, log server, api server together, these
> can
> > > be
> > > > called management server.
> > > >
> > > > Best Regards
> > > > ---------------
> > > > DolphinScheduler(Incubator) Committer
> > > > Xiaochun Liu 刘小春
> > > > liuxiaochun@apache.org
> > > > ---------------
> > > >
> > > >
> > > >
> > > > > 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
> > > > >
> > > > > To ligang.
> > > > > it's right.
> > > > > But alert server is only a small function. we define it as an
> > > > > individual module and as a server. do you thing alert is
> expensive
> > or
> > > > > taking more resource ? if not , why a single module ?
> > > > > And alert server trigger task event by scanning db, do you think
> it
> > > is a
> > > > > nice way ?
> > > > > Moving into server module is only our first step for simplifying
> > user
> > > > > deployment. Extension of alert can be updated via redeploy server
> and
> > > > it's
> > > > > not a frequent operation.
> > > > > As the architecture changes, alert implementation will change.
> > > > >
> > > > >
> > > > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com>
> > wrote:
> > > > >
> > > > >> I think from another angle,Master and Worker as key services,I
> think
> > > not
> > > > >> to redeployment during normal execution.
> > > > >> If tasks are still running,redeploy master and worker may be lead
> > to
> > > > >> missed the timed task.
> > > > >>
> > > > >> ________________________________
> > > > >> DolphinScheduler(Incubator) PPMC
> > > > >> Gang Li 李岗
> > > > >>
> > > > >> lgcareer2019@outlook.com<ma...@outlook.com>
> > > > >>
> > > > >> From: guo jiwei<ma...@gmail.com>
> > > > >> Date: 2019-12-11 18:11
> > > > >> To: dev<ma...@dolphinscheduler.apache.org>
> > > > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
> > > > >> To ligang.
> > > > >> redeploy is simple, but what about the latency of alert ?
> > > > >> it's easy to redeploy master server to update alert
> > > > >>
> > > > >>
> > > > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com>
> > wrote:
> > > > >>
> > > > >>> I think the alert module can be retained. Currently, it only
> > supports
> > > > >>> email and webchat,
> > > > >>> but more alarm modes can be added in the future.
> > > > >>>
> > > > >>> At present，alert is a independent service. the alert service only
> > > > >> consumes
> > > > >>> alarm information in the database, other services produce these
> > alarm
> > > > >>> information.
> > > > >>> If a new alarm mode is added, It is only necessary to redeploy
> the
> > > > alert
> > > > >>> service.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> ________________________________
> > > > >>> DolphinScheduler(Incubator) PPMC
> > > > >>> Gang Li 李岗
> > > > >>>
> > > > >>> lgcareer2019@outlook.com<ma...@outlook.com>
> > > > >>>
> > > > >>> 发件人： qiao zhanwei<ma...@outlook.com>
> > > > >>> 发送时间： 2019-12-10 14:24
> > > > >>> 收件人： dev<ma...@dolphinscheduler.apache.org>
> > > > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment
> > > > >>>
> > > > >>> Hello All ,
> > > > >>>
> > > > >>> Now DolphinScheduler has so many Configuration files
> > > > >>>
> > > > >>> for example :
> > > > >>>
> > > > >>> dolphinscheduler-alert :
> > > > >>> alert.properties
> > > > >>>
> > > > >>> dolphinscheduler-api :
> > > > >>> application-api.properties
> > > > >>> application-combined.properties
> > > > >>>
> > > > >>>
> > > > >>> dolphinscheduler-common :
> > > > >>> hadoop.properties
> > > > >>> common.properties
> > > > >>> quartz.properties
> > > > >>> zookeeper.properties
> > > > >>>
> > > > >>> dolphinscheduler-dao :
> > > > >>> application-dao.properties
> > > > >>>
> > > > >>> dolphinscheduler-server :
> > > > >>> application-master.properties
> > > > >>> application-master.properties
> > > > >>> master.properties
> > > > >>> worker.properties
> > > > >>>
> > > > >>> .dolphinscheduler_env.sh
> > > > >>>
> > > > >>> Can we simplify deployment ?
> > > > >>>
> > > > >>> Main point :
> > > > >>>
> > > > >>> 1 configuration file simplification and merged configuration
> file
> > > > >>> 2 master server remove port
> > > > >>> 3 support offline installation，remove kazoo dependencies in
> install
> > > and
> > > > >>> monitor
> > > > >>> 4 instll.sh script simplification
> > > > >>>
> > > > >>>
> > > > >>> —————————————
> > > > >>> DolphinScheduler(Incubator) PPMC
> > > > >>> Zhanwei Qiao 乔占卫
> > > > >>>
> > > > >>> qiaozhanwei@outlook.com
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
> >
> > --
> > DolphinScheduler(Incubator) PPMC
> > BaoLiang 鲍亮
> > leonbao@apache.org
> >
> > guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
> >
> > > To xiaochun.
> > > it's not a good way.
> > > Alert must be trigger by who is scheduling the task, in DS, it's
> > > MasterServer. Because, only by who is scheduling the task, it can know
> > the
> > > task status in time. If the task is timeout, trigger timeout alert in
> > time
> > > is very important for users.
> > > this is why we have to move alert into server module.
> > > Alert implementation will be refactor in the future. not scan db
> > anymore.
> > >
> > > On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
> > > wrote:
> > >
> > > > To guo jiwei:
> > > >
> > > > Why not put together with the api server, the alert server function
> is
> > > > very small,
> > > > and the load will not be very high. If logs are stored together in
> the
> > > > future,
> > > > we can combine alert server, log server, api server together, these
> can
> > > be
> > > > called management server.
> > > >
> > > > Best Regards
> > > > ---------------
> > > > DolphinScheduler(Incubator) Committer
> > > > Xiaochun Liu 刘小春
> > > > liuxiaochun@apache.org
> > > > ---------------
> > > >
> > > >
> > > >
> > > > > 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
> > > > >
> > > > > To ligang.
> > > > > it's right.
> > > > > But alert server is only a small function. we define it as an
> > > > > individual module and as a server. do you thing alert is
> expensive
> > or
> > > > > taking more resource ? if not , why a single module ?
> > > > > And alert server trigger task event by scanning db, do you think
> it
> > > is a
> > > > > nice way ?
> > > > > Moving into server module is only our first step for simplifying
> > user
> > > > > deployment. Extension of alert can be updated via redeploy server
> and
> > > > it's
> > > > > not a frequent operation.
> > > > > As the architecture changes, alert implementation will change.
> > > > >
> > > > >
> > > > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com>
> > wrote:
> > > > >
> > > > >> I think from another angle,Master and Worker as key services,I
> think
> > > not
> > > > >> to redeployment during normal execution.
> > > > >> If tasks are still running,redeploy master and worker may be lead
> > to
> > > > >> missed the timed task.
> > > > >>
> > > > >> ________________________________
> > > > >> DolphinScheduler(Incubator) PPMC
> > > > >> Gang Li 李岗
> > > > >>
> > > > >> lgcareer2019@outlook.com<ma...@outlook.com>
> > > > >>
> > > > >> From: guo jiwei<ma...@gmail.com>
> > > > >> Date: 2019-12-11 18:11
> > > > >> To: dev<ma...@dolphinscheduler.apache.org>
> > > > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
> > > > >> To ligang.
> > > > >> redeploy is simple, but what about the latency of alert ?
> > > > >> it's easy to redeploy master server to update alert
> > > > >>
> > > > >>
> > > > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com>
> > wrote:
> > > > >>
> > > > >>> I think the alert module can be retained. Currently, it only
> > supports
> > > > >>> email and webchat,
> > > > >>> but more alarm modes can be added in the future.
> > > > >>>
> > > > >>> At present，alert is a independent service. the alert service only
> > > > >> consumes
> > > > >>> alarm information in the database, other services produce these
> > alarm
> > > > >>> information.
> > > > >>> If a new alarm mode is added, It is only necessary to redeploy
> the
> > > > alert
> > > > >>> service.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> ________________________________
> > > > >>> DolphinScheduler(Incubator) PPMC
> > > > >>> Gang Li 李岗
> > > > >>>
> > > > >>> lgcareer2019@outlook.com<ma...@outlook.com>
> > > > >>>
> > > > >>> 发件人： qiao zhanwei<ma...@outlook.com>
> > > > >>> 发送时间： 2019-12-10 14:24
> > > > >>> 收件人： dev<ma...@dolphinscheduler.apache.org>
> > > > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment
> > > > >>>
> > > > >>> Hello All ,
> > > > >>>
> > > > >>> Now DolphinScheduler has so many Configuration files
> > > > >>>
> > > > >>> for example :
> > > > >>>
> > > > >>> dolphinscheduler-alert :
> > > > >>> alert.properties
> > > > >>>
> > > > >>> dolphinscheduler-api :
> > > > >>> application-api.properties
> > > > >>> application-combined.properties
> > > > >>>
> > > > >>>
> > > > >>> dolphinscheduler-common :
> > > > >>> hadoop.properties
> > > > >>> common.properties
> > > > >>> quartz.properties
> > > > >>> zookeeper.properties
> > > > >>>
> > > > >>> dolphinscheduler-dao :
> > > > >>> application-dao.properties
> > > > >>>
> > > > >>> dolphinscheduler-server :
> > > > >>> application-master.properties
> > > > >>> application-master.properties
> > > > >>> master.properties
> > > > >>> worker.properties
> > > > >>>
> > > > >>> .dolphinscheduler_env.sh
> > > > >>>
> > > > >>> Can we simplify deployment ?
> > > > >>>
> > > > >>> Main point :
> > > > >>>
> > > > >>> 1 configuration file simplification and merged configuration
> file
> > > > >>> 2 master server remove port
> > > > >>> 3 support offline installation，remove kazoo dependencies in
> install
> > > and
> > > > >>> monitor
> > > > >>> 4 instll.sh script simplification
> > > > >>>
> > > > >>>
> > > > >>> —————————————
> > > > >>> DolphinScheduler(Incubator) PPMC
> > > > >>> Zhanwei Qiao 乔占卫
> > > > >>>
> > > > >>> qiaozhanwei@outlook.com
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
> >
> > --
> > DolphinScheduler(Incubator) PPMC
> > BaoLiang 鲍亮
> > leonbao@apache.org
> >
>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by guo jiwei <te...@gmail.com>.
To baoliang：
Is alert server distributed ? it's a standalone module with only a
function scanning db and triggering alert .
Trigger event will delay if in an individual module even not scan db.
do you have some good idea to let alert trigger in time ? do you get my
point of in time ?
On Wed, Dec 11, 2019 at 8:26 PM leon bao <le...@apache.org> wrote:
> As an open source project, I think it's important to stay open and
> extensible.
> Our current alert does not have many shortcomings except for scanning the
> database with one thread, at the same time, the alert service as a separate
> module provides better scalability.
> So I don't think it is necessary to merge alert into other modules without
> much benefit.
>
> guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
>
> > To xiaochun.
> > it's not a good way.
> > Alert must be trigger by who is scheduling the task, in DS, it's
> > MasterServer. Because, only by who is scheduling the task, it can know
> the
> > task status in time. If the task is timeout, trigger timeout alert in
> time
> > is very important for users.
> > this is why we have to move alert into server module.
> > Alert implementation will be refactor in the future. not scan db
> anymore.
> >
> > On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
> > wrote:
> >
> > > To guo jiwei:
> > >
> > > Why not put together with the api server, the alert server function is
> > > very small,
> > > and the load will not be very high. If logs are stored together in the
> > > future,
> > > we can combine alert server, log server, api server together, these can
> > be
> > > called management server.
> > >
> > > Best Regards
> > > ---------------
> > > DolphinScheduler(Incubator) Committer
> > > Xiaochun Liu 刘小春
> > > liuxiaochun@apache.org
> > > ---------------
> > >
> > >
> > >
> > > > 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
> > > >
> > > > To ligang.
> > > > it's right.
> > > > But alert server is only a small function. we define it as an
> > > > individual module and as a server. do you thing alert is expensive
> or
> > > > taking more resource ? if not , why a single module ?
> > > > And alert server trigger task event by scanning db, do you think it
> > is a
> > > > nice way ?
> > > > Moving into server module is only our first step for simplifying
> user
> > > > deployment. Extension of alert can be updated via redeploy server and
> > > it's
> > > > not a frequent operation.
> > > > As the architecture changes, alert implementation will change.
> > > >
> > > >
> > > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com>
> wrote:
> > > >
> > > >> I think from another angle,Master and Worker as key services,I think
> > not
> > > >> to redeployment during normal execution.
> > > >> If tasks are still running,redeploy master and worker may be lead
> to
> > > >> missed the timed task.
> > > >>
> > > >> ________________________________
> > > >> DolphinScheduler(Incubator) PPMC
> > > >> Gang Li 李岗
> > > >>
> > > >> lgcareer2019@outlook.com<ma...@outlook.com>
> > > >>
> > > >> From: guo jiwei<ma...@gmail.com>
> > > >> Date: 2019-12-11 18:11
> > > >> To: dev<ma...@dolphinscheduler.apache.org>
> > > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
> > > >> To ligang.
> > > >> redeploy is simple, but what about the latency of alert ?
> > > >> it's easy to redeploy master server to update alert
> > > >>
> > > >>
> > > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com>
> wrote:
> > > >>
> > > >>> I think the alert module can be retained. Currently, it only
> supports
> > > >>> email and webchat,
> > > >>> but more alarm modes can be added in the future.
> > > >>>
> > > >>> At present，alert is a independent service. the alert service only
> > > >> consumes
> > > >>> alarm information in the database, other services produce these
> alarm
> > > >>> information.
> > > >>> If a new alarm mode is added, It is only necessary to redeploy the
> > > alert
> > > >>> service.
> > > >>>
> > > >>>
> > > >>>
> > > >>> ________________________________
> > > >>> DolphinScheduler(Incubator) PPMC
> > > >>> Gang Li 李岗
> > > >>>
> > > >>> lgcareer2019@outlook.com<ma...@outlook.com>
> > > >>>
> > > >>> 发件人： qiao zhanwei<ma...@outlook.com>
> > > >>> 发送时间： 2019-12-10 14:24
> > > >>> 收件人： dev<ma...@dolphinscheduler.apache.org>
> > > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment
> > > >>>
> > > >>> Hello All ,
> > > >>>
> > > >>> Now DolphinScheduler has so many Configuration files
> > > >>>
> > > >>> for example :
> > > >>>
> > > >>> dolphinscheduler-alert :
> > > >>> alert.properties
> > > >>>
> > > >>> dolphinscheduler-api :
> > > >>> application-api.properties
> > > >>> application-combined.properties
> > > >>>
> > > >>>
> > > >>> dolphinscheduler-common :
> > > >>> hadoop.properties
> > > >>> common.properties
> > > >>> quartz.properties
> > > >>> zookeeper.properties
> > > >>>
> > > >>> dolphinscheduler-dao :
> > > >>> application-dao.properties
> > > >>>
> > > >>> dolphinscheduler-server :
> > > >>> application-master.properties
> > > >>> application-master.properties
> > > >>> master.properties
> > > >>> worker.properties
> > > >>>
> > > >>> .dolphinscheduler_env.sh
> > > >>>
> > > >>> Can we simplify deployment ?
> > > >>>
> > > >>> Main point :
> > > >>>
> > > >>> 1 configuration file simplification and merged configuration file
> > > >>> 2 master server remove port
> > > >>> 3 support offline installation，remove kazoo dependencies in install
> > and
> > > >>> monitor
> > > >>> 4 instll.sh script simplification
> > > >>>
> > > >>>
> > > >>> —————————————
> > > >>> DolphinScheduler(Incubator) PPMC
> > > >>> Zhanwei Qiao 乔占卫
> > > >>>
> > > >>> qiaozhanwei@outlook.com
> > > >>>
> > > >>
> > >
> > >
> >
>
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
> guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
>
> > To xiaochun.
> > it's not a good way.
> > Alert must be trigger by who is scheduling the task, in DS, it's
> > MasterServer. Because, only by who is scheduling the task, it can know
> the
> > task status in time. If the task is timeout, trigger timeout alert in
> time
> > is very important for users.
> > this is why we have to move alert into server module.
> > Alert implementation will be refactor in the future. not scan db
> anymore.
> >
> > On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
> > wrote:
> >
> > > To guo jiwei:
> > >
> > > Why not put together with the api server, the alert server function is
> > > very small,
> > > and the load will not be very high. If logs are stored together in the
> > > future,
> > > we can combine alert server, log server, api server together, these can
> > be
> > > called management server.
> > >
> > > Best Regards
> > > ---------------
> > > DolphinScheduler(Incubator) Committer
> > > Xiaochun Liu 刘小春
> > > liuxiaochun@apache.org
> > > ---------------
> > >
> > >
> > >
> > > > 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
> > > >
> > > > To ligang.
> > > > it's right.
> > > > But alert server is only a small function. we define it as an
> > > > individual module and as a server. do you thing alert is expensive
> or
> > > > taking more resource ? if not , why a single module ?
> > > > And alert server trigger task event by scanning db, do you think it
> > is a
> > > > nice way ?
> > > > Moving into server module is only our first step for simplifying
> user
> > > > deployment. Extension of alert can be updated via redeploy server and
> > > it's
> > > > not a frequent operation.
> > > > As the architecture changes, alert implementation will change.
> > > >
> > > >
> > > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com>
> wrote:
> > > >
> > > >> I think from another angle,Master and Worker as key services,I think
> > not
> > > >> to redeployment during normal execution.
> > > >> If tasks are still running,redeploy master and worker may be lead
> to
> > > >> missed the timed task.
> > > >>
> > > >> ________________________________
> > > >> DolphinScheduler(Incubator) PPMC
> > > >> Gang Li 李岗
> > > >>
> > > >> lgcareer2019@outlook.com<ma...@outlook.com>
> > > >>
> > > >> From: guo jiwei<ma...@gmail.com>
> > > >> Date: 2019-12-11 18:11
> > > >> To: dev<ma...@dolphinscheduler.apache.org>
> > > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
> > > >> To ligang.
> > > >> redeploy is simple, but what about the latency of alert ?
> > > >> it's easy to redeploy master server to update alert
> > > >>
> > > >>
> > > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com>
> wrote:
> > > >>
> > > >>> I think the alert module can be retained. Currently, it only
> supports
> > > >>> email and webchat,
> > > >>> but more alarm modes can be added in the future.
> > > >>>
> > > >>> At present，alert is a independent service. the alert service only
> > > >> consumes
> > > >>> alarm information in the database, other services produce these
> alarm
> > > >>> information.
> > > >>> If a new alarm mode is added, It is only necessary to redeploy the
> > > alert
> > > >>> service.
> > > >>>
> > > >>>
> > > >>>
> > > >>> ________________________________
> > > >>> DolphinScheduler(Incubator) PPMC
> > > >>> Gang Li 李岗
> > > >>>
> > > >>> lgcareer2019@outlook.com<ma...@outlook.com>
> > > >>>
> > > >>> 发件人： qiao zhanwei<ma...@outlook.com>
> > > >>> 发送时间： 2019-12-10 14:24
> > > >>> 收件人： dev<ma...@dolphinscheduler.apache.org>
> > > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment
> > > >>>
> > > >>> Hello All ,
> > > >>>
> > > >>> Now DolphinScheduler has so many Configuration files
> > > >>>
> > > >>> for example :
> > > >>>
> > > >>> dolphinscheduler-alert :
> > > >>> alert.properties
> > > >>>
> > > >>> dolphinscheduler-api :
> > > >>> application-api.properties
> > > >>> application-combined.properties
> > > >>>
> > > >>>
> > > >>> dolphinscheduler-common :
> > > >>> hadoop.properties
> > > >>> common.properties
> > > >>> quartz.properties
> > > >>> zookeeper.properties
> > > >>>
> > > >>> dolphinscheduler-dao :
> > > >>> application-dao.properties
> > > >>>
> > > >>> dolphinscheduler-server :
> > > >>> application-master.properties
> > > >>> application-master.properties
> > > >>> master.properties
> > > >>> worker.properties
> > > >>>
> > > >>> .dolphinscheduler_env.sh
> > > >>>
> > > >>> Can we simplify deployment ?
> > > >>>
> > > >>> Main point :
> > > >>>
> > > >>> 1 configuration file simplification and merged configuration file
> > > >>> 2 master server remove port
> > > >>> 3 support offline installation，remove kazoo dependencies in install
> > and
> > > >>> monitor
> > > >>> 4 instll.sh script simplification
> > > >>>
> > > >>>
> > > >>> —————————————
> > > >>> DolphinScheduler(Incubator) PPMC
> > > >>> Zhanwei Qiao 乔占卫
> > > >>>
> > > >>> qiaozhanwei@outlook.com
> > > >>>
> > > >>
> > >
> > >
> >
>
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by leon bao <le...@apache.org>.
As an open source project, I think it's important to stay open and
extensible.
Our current alert does not have many shortcomings except for scanning the
database with one thread, at the same time, the alert service as a separate
module provides better scalability.
So I don't think it is necessary to merge alert into other modules without
much benefit.
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
> To xiaochun.
> it's not a good way.
> Alert must be trigger by who is scheduling the task, in DS, it's
> MasterServer. Because, only by who is scheduling the task, it can know the
> task status in time. If the task is timeout, trigger timeout alert in time
> is very important for users.
> this is why we have to move alert into server module.
> Alert implementation will be refactor in the future. not scan db anymore.
>
> On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
> wrote:
>
> > To guo jiwei:
> >
> > Why not put together with the api server, the alert server function is
> > very small,
> > and the load will not be very high. If logs are stored together in the
> > future,
> > we can combine alert server, log server, api server together, these can
> be
> > called management server.
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) Committer
> > Xiaochun Liu 刘小春
> > liuxiaochun@apache.org
> > ---------------
> >
> >
> >
> > > 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
> > >
> > > To ligang.
> > > it's right.
> > > But alert server is only a small function. we define it as an
> > > individual module and as a server. do you thing alert is expensive or
> > > taking more resource ? if not , why a single module ?
> > > And alert server trigger task event by scanning db, do you think it
> is a
> > > nice way ?
> > > Moving into server module is only our first step for simplifying user
> > > deployment. Extension of alert can be updated via redeploy server and
> > it's
> > > not a frequent operation.
> > > As the architecture changes, alert implementation will change.
> > >
> > >
> > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com> wrote:
> > >
> > >> I think from another angle,Master and Worker as key services,I think
> not
> > >> to redeployment during normal execution.
> > >> If tasks are still running,redeploy master and worker may be lead to
> > >> missed the timed task.
> > >>
> > >> ________________________________
> > >> DolphinScheduler(Incubator) PPMC
> > >> Gang Li 李岗
> > >>
> > >> lgcareer2019@outlook.com<ma...@outlook.com>
> > >>
> > >> From: guo jiwei<ma...@gmail.com>
> > >> Date: 2019-12-11 18:11
> > >> To: dev<ma...@dolphinscheduler.apache.org>
> > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
> > >> To ligang.
> > >> redeploy is simple, but what about the latency of alert ?
> > >> it's easy to redeploy master server to update alert
> > >>
> > >>
> > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
> > >>
> > >>> I think the alert module can be retained. Currently, it only supports
> > >>> email and webchat,
> > >>> but more alarm modes can be added in the future.
> > >>>
> > >>> At present，alert is a independent service. the alert service only
> > >> consumes
> > >>> alarm information in the database, other services produce these alarm
> > >>> information.
> > >>> If a new alarm mode is added, It is only necessary to redeploy the
> > alert
> > >>> service.
> > >>>
> > >>>
> > >>>
> > >>> ________________________________
> > >>> DolphinScheduler(Incubator) PPMC
> > >>> Gang Li 李岗
> > >>>
> > >>> lgcareer2019@outlook.com<ma...@outlook.com>
> > >>>
> > >>> 发件人： qiao zhanwei<ma...@outlook.com>
> > >>> 发送时间： 2019-12-10 14:24
> > >>> 收件人： dev<ma...@dolphinscheduler.apache.org>
> > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment
> > >>>
> > >>> Hello All ,
> > >>>
> > >>> Now DolphinScheduler has so many Configuration files
> > >>>
> > >>> for example :
> > >>>
> > >>> dolphinscheduler-alert :
> > >>> alert.properties
> > >>>
> > >>> dolphinscheduler-api :
> > >>> application-api.properties
> > >>> application-combined.properties
> > >>>
> > >>>
> > >>> dolphinscheduler-common :
> > >>> hadoop.properties
> > >>> common.properties
> > >>> quartz.properties
> > >>> zookeeper.properties
> > >>>
> > >>> dolphinscheduler-dao :
> > >>> application-dao.properties
> > >>>
> > >>> dolphinscheduler-server :
> > >>> application-master.properties
> > >>> application-master.properties
> > >>> master.properties
> > >>> worker.properties
> > >>>
> > >>> .dolphinscheduler_env.sh
> > >>>
> > >>> Can we simplify deployment ?
> > >>>
> > >>> Main point :
> > >>>
> > >>> 1 configuration file simplification and merged configuration file
> > >>> 2 master server remove port
> > >>> 3 support offline installation，remove kazoo dependencies in install
> and
> > >>> monitor
> > >>> 4 instll.sh script simplification
> > >>>
> > >>>
> > >>> —————————————
> > >>> DolphinScheduler(Incubator) PPMC
> > >>> Zhanwei Qiao 乔占卫
> > >>>
> > >>> qiaozhanwei@outlook.com
> > >>>
> > >>
> >
> >
>
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午7:33写道：
> To xiaochun.
> it's not a good way.
> Alert must be trigger by who is scheduling the task, in DS, it's
> MasterServer. Because, only by who is scheduling the task, it can know the
> task status in time. If the task is timeout, trigger timeout alert in time
> is very important for users.
> this is why we have to move alert into server module.
> Alert implementation will be refactor in the future. not scan db anymore.
>
> On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org>
> wrote:
>
> > To guo jiwei:
> >
> > Why not put together with the api server, the alert server function is
> > very small,
> > and the load will not be very high. If logs are stored together in the
> > future,
> > we can combine alert server, log server, api server together, these can
> be
> > called management server.
> >
> > Best Regards
> > ---------------
> > DolphinScheduler(Incubator) Committer
> > Xiaochun Liu 刘小春
> > liuxiaochun@apache.org
> > ---------------
> >
> >
> >
> > > 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
> > >
> > > To ligang.
> > > it's right.
> > > But alert server is only a small function. we define it as an
> > > individual module and as a server. do you thing alert is expensive or
> > > taking more resource ? if not , why a single module ?
> > > And alert server trigger task event by scanning db, do you think it
> is a
> > > nice way ?
> > > Moving into server module is only our first step for simplifying user
> > > deployment. Extension of alert can be updated via redeploy server and
> > it's
> > > not a frequent operation.
> > > As the architecture changes, alert implementation will change.
> > >
> > >
> > > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com> wrote:
> > >
> > >> I think from another angle,Master and Worker as key services,I think
> not
> > >> to redeployment during normal execution.
> > >> If tasks are still running,redeploy master and worker may be lead to
> > >> missed the timed task.
> > >>
> > >> ________________________________
> > >> DolphinScheduler(Incubator) PPMC
> > >> Gang Li 李岗
> > >>
> > >> lgcareer2019@outlook.com<ma...@outlook.com>
> > >>
> > >> From: guo jiwei<ma...@gmail.com>
> > >> Date: 2019-12-11 18:11
> > >> To: dev<ma...@dolphinscheduler.apache.org>
> > >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
> > >> To ligang.
> > >> redeploy is simple, but what about the latency of alert ?
> > >> it's easy to redeploy master server to update alert
> > >>
> > >>
> > >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
> > >>
> > >>> I think the alert module can be retained. Currently, it only supports
> > >>> email and webchat,
> > >>> but more alarm modes can be added in the future.
> > >>>
> > >>> At present，alert is a independent service. the alert service only
> > >> consumes
> > >>> alarm information in the database, other services produce these alarm
> > >>> information.
> > >>> If a new alarm mode is added, It is only necessary to redeploy the
> > alert
> > >>> service.
> > >>>
> > >>>
> > >>>
> > >>> ________________________________
> > >>> DolphinScheduler(Incubator) PPMC
> > >>> Gang Li 李岗
> > >>>
> > >>> lgcareer2019@outlook.com<ma...@outlook.com>
> > >>>
> > >>> 发件人： qiao zhanwei<ma...@outlook.com>
> > >>> 发送时间： 2019-12-10 14:24
> > >>> 收件人： dev<ma...@dolphinscheduler.apache.org>
> > >>> 主题： Aproposal for DolphinScheduler Simplified Deployment
> > >>>
> > >>> Hello All ,
> > >>>
> > >>> Now DolphinScheduler has so many Configuration files
> > >>>
> > >>> for example :
> > >>>
> > >>> dolphinscheduler-alert :
> > >>> alert.properties
> > >>>
> > >>> dolphinscheduler-api :
> > >>> application-api.properties
> > >>> application-combined.properties
> > >>>
> > >>>
> > >>> dolphinscheduler-common :
> > >>> hadoop.properties
> > >>> common.properties
> > >>> quartz.properties
> > >>> zookeeper.properties
> > >>>
> > >>> dolphinscheduler-dao :
> > >>> application-dao.properties
> > >>>
> > >>> dolphinscheduler-server :
> > >>> application-master.properties
> > >>> application-master.properties
> > >>> master.properties
> > >>> worker.properties
> > >>>
> > >>> .dolphinscheduler_env.sh
> > >>>
> > >>> Can we simplify deployment ?
> > >>>
> > >>> Main point :
> > >>>
> > >>> 1 configuration file simplification and merged configuration file
> > >>> 2 master server remove port
> > >>> 3 support offline installation，remove kazoo dependencies in install
> and
> > >>> monitor
> > >>> 4 instll.sh script simplification
> > >>>
> > >>>
> > >>> —————————————
> > >>> DolphinScheduler(Incubator) PPMC
> > >>> Zhanwei Qiao 乔占卫
> > >>>
> > >>> qiaozhanwei@outlook.com
> > >>>
> > >>
> >
> >
>
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
回复: Ambari plugin develop
Posted by qiao zhanwei <qi...@outlook.com>.
good !
―――――――――――――
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
发件人： xtdwbj<ma...@163.com>
发送时间： 2019-12-11 19:56
收件人： dev<ma...@dolphinscheduler.apache.org>
主题： Ambari plugin develop
To JUN GAO :
I will develop the ambari plugin for DS
At 2019-12-11 19:32:53, "guo jiwei" <te...@gmail.com> wrote:
>To xiaochun.
> it's not a good way.
> Alert must be trigger by who is scheduling the task, in DS, it's
>MasterServer. Because, only by who is scheduling the task, it can know the
>task status in time. If the task is timeout, trigger timeout alert in time
>is very important for users.
> this is why we have to move alert into server module.
> Alert implementation will be refactor in the future. not scan db anymore.
>
>On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org> wrote:
>
>> To guo jiwei:
>>
>> Why not put together with the api server, the alert server function is
>> very small,
>> and the load will not be very high. If logs are stored together in the
>> future,
>> we can combine alert server, log server, api server together, these can be
>> called management server.
>>
>> Best Regards
>> ---------------
>> DolphinScheduler(Incubator) Committer
>> Xiaochun Liu 刘小春
>> liuxiaochun@apache.org
>> ---------------
>>
>>
>>
>> > 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
>> >
>> > To ligang.
>> > it's right.
>> > But alert server is only a small function. we define it as an
>> > individual module and as a server. do you thing alert is expensive or
>> > taking more resource ? if not , why a single module ?
>> > And alert server trigger task event by scanning db, do you think it is a
>> > nice way ?
>> > Moving into server module is only our first step for simplifying user
>> > deployment. Extension of alert can be updated via redeploy server and
>> it's
>> > not a frequent operation.
>> > As the architecture changes, alert implementation will change.
>> >
>> >
>> > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com> wrote:
>> >
>> >> I think from another angle,Master and Worker as key services,I think not
>> >> to redeployment during normal execution.
>> >> If tasks are still running,redeploy master and worker may be lead to
>> >> missed the timed task.
>> >>
>> >> ________________________________
>> >> DolphinScheduler(Incubator) PPMC
>> >> Gang Li 李岗
>> >>
>> >> lgcareer2019@outlook.com<ma...@outlook.com>
>> >>
>> >> From: guo jiwei<ma...@gmail.com>
>> >> Date: 2019-12-11 18:11
>> >> To: dev<ma...@dolphinscheduler.apache.org>
>> >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
>> >> To ligang.
>> >> redeploy is simple, but what about the latency of alert ?
>> >> it's easy to redeploy master server to update alert
>> >>
>> >>
>> >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
>> >>
>> >>> I think the alert module can be retained. Currently, it only supports
>> >>> email and webchat,
>> >>> but more alarm modes can be added in the future.
>> >>>
>> >>> At present，alert is a independent service. the alert service only
>> >> consumes
>> >>> alarm information in the database, other services produce these alarm
>> >>> information.
>> >>> If a new alarm mode is added, It is only necessary to redeploy the
>> alert
>> >>> service.
>> >>>
>> >>>
>> >>>
>> >>> ________________________________
>> >>> DolphinScheduler(Incubator) PPMC
>> >>> Gang Li 李岗
>> >>>
>> >>> lgcareer2019@outlook.com<ma...@outlook.com>
>> >>>
>> >>> 发件人： qiao zhanwei<ma...@outlook.com>
>> >>> 发送时间： 2019-12-10 14:24
>> >>> 收件人： dev<ma...@dolphinscheduler.apache.org>
>> >>> 主题： Aproposal for DolphinScheduler Simplified Deployment
>> >>>
>> >>> Hello All ,
>> >>>
>> >>> Now DolphinScheduler has so many Configuration files
>> >>>
>> >>> for example :
>> >>>
>> >>> dolphinscheduler-alert :
>> >>> alert.properties
>> >>>
>> >>> dolphinscheduler-api :
>> >>> application-api.properties
>> >>> application-combined.properties
>> >>>
>> >>>
>> >>> dolphinscheduler-common :
>> >>> hadoop.properties
>> >>> common.properties
>> >>> quartz.properties
>> >>> zookeeper.properties
>> >>>
>> >>> dolphinscheduler-dao :
>> >>> application-dao.properties
>> >>>
>> >>> dolphinscheduler-server :
>> >>> application-master.properties
>> >>> application-master.properties
>> >>> master.properties
>> >>> worker.properties
>> >>>
>> >>> .dolphinscheduler_env.sh
>> >>>
>> >>> Can we simplify deployment ?
>> >>>
>> >>> Main point :
>> >>>
>> >>> 1 configuration file simplification and merged configuration file
>> >>> 2 master server remove port
>> >>> 3 support offline installation，remove kazoo dependencies in install and
>> >>> monitor
>> >>> 4 instll.sh script simplification
>> >>>
>> >>>
>> >>> ―――――――――――――
>> >>> DolphinScheduler(Incubator) PPMC
>> >>> Zhanwei Qiao 乔占卫
>> >>>
>> >>> qiaozhanwei@outlook.com
>> >>>
>> >>
>>
>>
Ambari plugin develop
Posted by xtdwbj <18...@163.com>.
To JUN GAO :
I will develop the ambari plugin for DS
At 2019-12-11 19:32:53, "guo jiwei" <te...@gmail.com> wrote:
>To xiaochun.
> it's not a good way.
> Alert must be trigger by who is scheduling the task, in DS, it's
>MasterServer. Because, only by who is scheduling the task, it can know the
>task status in time. If the task is timeout, trigger timeout alert in time
>is very important for users.
> this is why we have to move alert into server module.
> Alert implementation will be refactor in the future. not scan db anymore.
>
>On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org> wrote:
>
>> To guo jiwei:
>>
>> Why not put together with the api server, the alert server function is
>> very small,
>> and the load will not be very high. If logs are stored together in the
>> future,
>> we can combine alert server, log server, api server together, these can be
>> called management server.
>>
>> Best Regards
>> ---------------
>> DolphinScheduler(Incubator) Committer
>> Xiaochun Liu 刘小春
>> liuxiaochun@apache.org
>> ---------------
>>
>>
>>
>> > 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
>> >
>> > To ligang.
>> > it's right.
>> > But alert server is only a small function. we define it as an
>> > individual module and as a server. do you thing alert is expensive or
>> > taking more resource ? if not , why a single module ?
>> > And alert server trigger task event by scanning db, do you think it is a
>> > nice way ?
>> > Moving into server module is only our first step for simplifying user
>> > deployment. Extension of alert can be updated via redeploy server and
>> it's
>> > not a frequent operation.
>> > As the architecture changes, alert implementation will change.
>> >
>> >
>> > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com> wrote:
>> >
>> >> I think from another angle,Master and Worker as key services,I think not
>> >> to redeployment during normal execution.
>> >> If tasks are still running,redeploy master and worker may be lead to
>> >> missed the timed task.
>> >>
>> >> ________________________________
>> >> DolphinScheduler(Incubator) PPMC
>> >> Gang Li 李岗
>> >>
>> >> lgcareer2019@outlook.com<ma...@outlook.com>
>> >>
>> >> From: guo jiwei<ma...@gmail.com>
>> >> Date: 2019-12-11 18:11
>> >> To: dev<ma...@dolphinscheduler.apache.org>
>> >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
>> >> To ligang.
>> >> redeploy is simple, but what about the latency of alert ?
>> >> it's easy to redeploy master server to update alert
>> >>
>> >>
>> >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
>> >>
>> >>> I think the alert module can be retained. Currently, it only supports
>> >>> email and webchat,
>> >>> but more alarm modes can be added in the future.
>> >>>
>> >>> At present，alert is a independent service. the alert service only
>> >> consumes
>> >>> alarm information in the database, other services produce these alarm
>> >>> information.
>> >>> If a new alarm mode is added, It is only necessary to redeploy the
>> alert
>> >>> service.
>> >>>
>> >>>
>> >>>
>> >>> ________________________________
>> >>> DolphinScheduler(Incubator) PPMC
>> >>> Gang Li 李岗
>> >>>
>> >>> lgcareer2019@outlook.com<ma...@outlook.com>
>> >>>
>> >>> 发件人： qiao zhanwei<ma...@outlook.com>
>> >>> 发送时间： 2019-12-10 14:24
>> >>> 收件人： dev<ma...@dolphinscheduler.apache.org>
>> >>> 主题： Aproposal for DolphinScheduler Simplified Deployment
>> >>>
>> >>> Hello All ,
>> >>>
>> >>> Now DolphinScheduler has so many Configuration files
>> >>>
>> >>> for example :
>> >>>
>> >>> dolphinscheduler-alert :
>> >>> alert.properties
>> >>>
>> >>> dolphinscheduler-api :
>> >>> application-api.properties
>> >>> application-combined.properties
>> >>>
>> >>>
>> >>> dolphinscheduler-common :
>> >>> hadoop.properties
>> >>> common.properties
>> >>> quartz.properties
>> >>> zookeeper.properties
>> >>>
>> >>> dolphinscheduler-dao :
>> >>> application-dao.properties
>> >>>
>> >>> dolphinscheduler-server :
>> >>> application-master.properties
>> >>> application-master.properties
>> >>> master.properties
>> >>> worker.properties
>> >>>
>> >>> .dolphinscheduler_env.sh
>> >>>
>> >>> Can we simplify deployment ?
>> >>>
>> >>> Main point :
>> >>>
>> >>> 1 configuration file simplification and merged configuration file
>> >>> 2 master server remove port
>> >>> 3 support offline installation，remove kazoo dependencies in install and
>> >>> monitor
>> >>> 4 instll.sh script simplification
>> >>>
>> >>>
>> >>> —————————————
>> >>> DolphinScheduler(Incubator) PPMC
>> >>> Zhanwei Qiao 乔占卫
>> >>>
>> >>> qiaozhanwei@outlook.com
>> >>>
>> >>
>>
>>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by guo jiwei <te...@gmail.com>.
To xiaochun.
it's not a good way.
Alert must be trigger by who is scheduling the task, in DS, it's
MasterServer. Because, only by who is scheduling the task, it can know the
task status in time. If the task is timeout, trigger timeout alert in time
is very important for users.
this is why we have to move alert into server module.
Alert implementation will be refactor in the future. not scan db anymore.
On Wed, Dec 11, 2019 at 7:26 PM Xiaochun Liu <li...@apache.org> wrote:
> To guo jiwei:
>
> Why not put together with the api server, the alert server function is
> very small,
> and the load will not be very high. If logs are stored together in the
> future,
> we can combine alert server, log server, api server together, these can be
> called management server.
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org
> ---------------
>
>
>
> > 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
> >
> > To ligang.
> > it's right.
> > But alert server is only a small function. we define it as an
> > individual module and as a server. do you thing alert is expensive or
> > taking more resource ? if not , why a single module ?
> > And alert server trigger task event by scanning db, do you think it is a
> > nice way ?
> > Moving into server module is only our first step for simplifying user
> > deployment. Extension of alert can be updated via redeploy server and
> it's
> > not a frequent operation.
> > As the architecture changes, alert implementation will change.
> >
> >
> > On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com> wrote:
> >
> >> I think from another angle,Master and Worker as key services,I think not
> >> to redeployment during normal execution.
> >> If tasks are still running,redeploy master and worker may be lead to
> >> missed the timed task.
> >>
> >> ________________________________
> >> DolphinScheduler(Incubator) PPMC
> >> Gang Li 李岗
> >>
> >> lgcareer2019@outlook.com<ma...@outlook.com>
> >>
> >> From: guo jiwei<ma...@gmail.com>
> >> Date: 2019-12-11 18:11
> >> To: dev<ma...@dolphinscheduler.apache.org>
> >> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
> >> To ligang.
> >> redeploy is simple, but what about the latency of alert ?
> >> it's easy to redeploy master server to update alert
> >>
> >>
> >> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
> >>
> >>> I think the alert module can be retained. Currently, it only supports
> >>> email and webchat,
> >>> but more alarm modes can be added in the future.
> >>>
> >>> At present，alert is a independent service. the alert service only
> >> consumes
> >>> alarm information in the database, other services produce these alarm
> >>> information.
> >>> If a new alarm mode is added, It is only necessary to redeploy the
> alert
> >>> service.
> >>>
> >>>
> >>>
> >>> ________________________________
> >>> DolphinScheduler(Incubator) PPMC
> >>> Gang Li 李岗
> >>>
> >>> lgcareer2019@outlook.com<ma...@outlook.com>
> >>>
> >>> 发件人： qiao zhanwei<ma...@outlook.com>
> >>> 发送时间： 2019-12-10 14:24
> >>> 收件人： dev<ma...@dolphinscheduler.apache.org>
> >>> 主题： Aproposal for DolphinScheduler Simplified Deployment
> >>>
> >>> Hello All ,
> >>>
> >>> Now DolphinScheduler has so many Configuration files
> >>>
> >>> for example :
> >>>
> >>> dolphinscheduler-alert :
> >>> alert.properties
> >>>
> >>> dolphinscheduler-api :
> >>> application-api.properties
> >>> application-combined.properties
> >>>
> >>>
> >>> dolphinscheduler-common :
> >>> hadoop.properties
> >>> common.properties
> >>> quartz.properties
> >>> zookeeper.properties
> >>>
> >>> dolphinscheduler-dao :
> >>> application-dao.properties
> >>>
> >>> dolphinscheduler-server :
> >>> application-master.properties
> >>> application-master.properties
> >>> master.properties
> >>> worker.properties
> >>>
> >>> .dolphinscheduler_env.sh
> >>>
> >>> Can we simplify deployment ?
> >>>
> >>> Main point :
> >>>
> >>> 1 configuration file simplification and merged configuration file
> >>> 2 master server remove port
> >>> 3 support offline installation，remove kazoo dependencies in install and
> >>> monitor
> >>> 4 instll.sh script simplification
> >>>
> >>>
> >>> —————————————
> >>> DolphinScheduler(Incubator) PPMC
> >>> Zhanwei Qiao 乔占卫
> >>>
> >>> qiaozhanwei@outlook.com
> >>>
> >>
>
>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by Xiaochun Liu <li...@apache.org>.
To guo jiwei:
Why not put together with the api server, the alert server function is very small,
and the load will not be very high. If logs are stored together in the future,
we can combine alert server, log server, api server together, these can be called management server.
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
> 在 2019年12月11日，下午7:15，guo jiwei <te...@gmail.com> 写道：
>
> To ligang.
> it's right.
> But alert server is only a small function. we define it as an
> individual module and as a server. do you thing alert is expensive or
> taking more resource ? if not , why a single module ?
> And alert server trigger task event by scanning db, do you think it is a
> nice way ?
> Moving into server module is only our first step for simplifying user
> deployment. Extension of alert can be updated via redeploy server and it's
> not a frequent operation.
> As the architecture changes, alert implementation will change.
>
>
> On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com> wrote:
>
>> I think from another angle,Master and Worker as key services,I think not
>> to redeployment during normal execution.
>> If tasks are still running,redeploy master and worker may be lead to
>> missed the timed task.
>>
>> ________________________________
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com<ma...@outlook.com>
>>
>> From: guo jiwei<ma...@gmail.com>
>> Date: 2019-12-11 18:11
>> To: dev<ma...@dolphinscheduler.apache.org>
>> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
>> To ligang.
>> redeploy is simple, but what about the latency of alert ?
>> it's easy to redeploy master server to update alert
>>
>>
>> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
>>
>>> I think the alert module can be retained. Currently, it only supports
>>> email and webchat,
>>> but more alarm modes can be added in the future.
>>>
>>> At present，alert is a independent service. the alert service only
>> consumes
>>> alarm information in the database, other services produce these alarm
>>> information.
>>> If a new alarm mode is added, It is only necessary to redeploy the alert
>>> service.
>>>
>>>
>>>
>>> ________________________________
>>> DolphinScheduler(Incubator) PPMC
>>> Gang Li 李岗
>>>
>>> lgcareer2019@outlook.com<ma...@outlook.com>
>>>
>>> 发件人： qiao zhanwei<ma...@outlook.com>
>>> 发送时间： 2019-12-10 14:24
>>> 收件人： dev<ma...@dolphinscheduler.apache.org>
>>> 主题： Aproposal for DolphinScheduler Simplified Deployment
>>>
>>> Hello All ,
>>>
>>> Now DolphinScheduler has so many Configuration files
>>>
>>> for example :
>>>
>>> dolphinscheduler-alert :
>>> alert.properties
>>>
>>> dolphinscheduler-api :
>>> application-api.properties
>>> application-combined.properties
>>>
>>>
>>> dolphinscheduler-common :
>>> hadoop.properties
>>> common.properties
>>> quartz.properties
>>> zookeeper.properties
>>>
>>> dolphinscheduler-dao :
>>> application-dao.properties
>>>
>>> dolphinscheduler-server :
>>> application-master.properties
>>> application-master.properties
>>> master.properties
>>> worker.properties
>>>
>>> .dolphinscheduler_env.sh
>>>
>>> Can we simplify deployment ?
>>>
>>> Main point :
>>>
>>> 1 configuration file simplification and merged configuration file
>>> 2 master server remove port
>>> 3 support offline installation，remove kazoo dependencies in install and
>>> monitor
>>> 4 instll.sh script simplification
>>>
>>>
>>> —————————————
>>> DolphinScheduler(Incubator) PPMC
>>> Zhanwei Qiao 乔占卫
>>>
>>> qiaozhanwei@outlook.com
>>>
>>
Re: Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by guo jiwei <te...@gmail.com>.
To ligang.
it's right.
But alert server is only a small function. we define it as an
individual module and as a server. do you thing alert is expensive or
taking more resource ? if not , why a single module ?
And alert server trigger task event by scanning db, do you think it is a
nice way ?
Moving into server module is only our first step for simplifying user
deployment. Extension of alert can be updated via redeploy server and it's
not a frequent operation.
As the architecture changes, alert implementation will change.
On Wed, Dec 11, 2019 at 6:23 PM 李 岗 <lg...@outlook.com> wrote:
> I think from another angle,Master and Worker as key services,I think not
> to redeployment during normal execution.
> If tasks are still running,redeploy master and worker may be lead to
> missed the timed task.
>
> ________________________________
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> From: guo jiwei<ma...@gmail.com>
> Date: 2019-12-11 18:11
> To: dev<ma...@dolphinscheduler.apache.org>
> Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
> To ligang.
> redeploy is simple, but what about the latency of alert ?
> it's easy to redeploy master server to update alert
>
>
> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
>
> > I think the alert module can be retained. Currently, it only supports
> > email and webchat,
> > but more alarm modes can be added in the future.
> >
> > At present，alert is a independent service. the alert service only
> consumes
> > alarm information in the database, other services produce these alarm
> > information.
> > If a new alarm mode is added, It is only necessary to redeploy the alert
> > service.
> >
> >
> >
> > ________________________________
> > DolphinScheduler(Incubator) PPMC
> > Gang Li 李岗
> >
> > lgcareer2019@outlook.com<ma...@outlook.com>
> >
> > 发件人： qiao zhanwei<ma...@outlook.com>
> > 发送时间： 2019-12-10 14:24
> > 收件人： dev<ma...@dolphinscheduler.apache.org>
> > 主题： Aproposal for DolphinScheduler Simplified Deployment
> >
> > Hello All ,
> >
> > Now DolphinScheduler has so many Configuration files
> >
> > for example :
> >
> > dolphinscheduler-alert :
> > alert.properties
> >
> > dolphinscheduler-api :
> > application-api.properties
> > application-combined.properties
> >
> >
> > dolphinscheduler-common :
> > hadoop.properties
> > common.properties
> > quartz.properties
> > zookeeper.properties
> >
> > dolphinscheduler-dao :
> > application-dao.properties
> >
> > dolphinscheduler-server :
> > application-master.properties
> > application-master.properties
> > master.properties
> > worker.properties
> >
> > .dolphinscheduler_env.sh
> >
> > Can we simplify deployment ?
> >
> > Main point :
> >
> > 1 configuration file simplification and merged configuration file
> > 2 master server remove port
> > 3 support offline installation，remove kazoo dependencies in install and
> > monitor
> > 4 instll.sh script simplification
> >
> >
> > —————————————
> > DolphinScheduler(Incubator) PPMC
> > Zhanwei Qiao 乔占卫
> >
> > qiaozhanwei@outlook.com
> >
>
Re: Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by 李 岗 <lg...@outlook.com>.
I think from another angle,Master and Worker as key services,I think not to redeployment during normal execution.
If tasks are still running,redeploy master and worker may be lead to missed the timed task.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
From: guo jiwei<ma...@gmail.com>
Date: 2019-12-11 18:11
To: dev<ma...@dolphinscheduler.apache.org>
Subject: Re: Aproposal for DolphinScheduler Simplified Deployment
To ligang.
redeploy is simple, but what about the latency of alert ?
it's easy to redeploy master server to update alert
On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
> I think the alert module can be retained. Currently, it only supports
> email and webchat,
> but more alarm modes can be added in the future.
>
> At present，alert is a independent service. the alert service only consumes
> alarm information in the database, other services produce these alarm
> information.
> If a new alarm mode is added, It is only necessary to redeploy the alert
> service.
>
>
>
> ________________________________
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> 发件人： qiao zhanwei<ma...@outlook.com>
> 发送时间： 2019-12-10 14:24
> 收件人： dev<ma...@dolphinscheduler.apache.org>
> 主题： Aproposal for DolphinScheduler Simplified Deployment
>
> Hello All ,
>
> Now DolphinScheduler has so many Configuration files
>
> for example :
>
> dolphinscheduler-alert :
> alert.properties
>
> dolphinscheduler-api :
> application-api.properties
> application-combined.properties
>
>
> dolphinscheduler-common :
> hadoop.properties
> common.properties
> quartz.properties
> zookeeper.properties
>
> dolphinscheduler-dao :
> application-dao.properties
>
> dolphinscheduler-server :
> application-master.properties
> application-master.properties
> master.properties
> worker.properties
>
> .dolphinscheduler_env.sh
>
> Can we simplify deployment ?
>
> Main point :
>
> 1 configuration file simplification and merged configuration file
> 2 master server remove port
> 3 support offline installation，remove kazoo dependencies in install and
> monitor
> 4 instll.sh script simplification
>
>
> —————————————
> DolphinScheduler(Incubator) PPMC
> Zhanwei Qiao 乔占卫
>
> qiaozhanwei@outlook.com
>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by nauu <bo...@qq.com>.
From a design perspective, each service module should be separated. This makes it easier for the developer to contribute.
It's like forward-looking design as Wu Sheng said.
For development environment ,We can design a combine mode ,to make it easy to debug.
boness@qq.com
zhukai/nauu ygsoft
> On Dec 11, 2019, at 6:11 PM, guo jiwei <te...@gmail.com> wrote:
>
> To ligang.
> redeploy is simple, but what about the latency of alert ?
> it's easy to redeploy master server to update alert
>
>
> On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
>
>> I think the alert module can be retained. Currently, it only supports
>> email and webchat,
>> but more alarm modes can be added in the future.
>>
>> At present，alert is a independent service. the alert service only consumes
>> alarm information in the database, other services produce these alarm
>> information.
>> If a new alarm mode is added, It is only necessary to redeploy the alert
>> service.
>>
>>
>>
>> ________________________________
>> DolphinScheduler(Incubator) PPMC
>> Gang Li 李岗
>>
>> lgcareer2019@outlook.com<ma...@outlook.com>
>>
>> 发件人： qiao zhanwei<ma...@outlook.com>
>> 发送时间： 2019-12-10 14:24
>> 收件人： dev<ma...@dolphinscheduler.apache.org>
>> 主题： Aproposal for DolphinScheduler Simplified Deployment
>>
>> Hello All ,
>>
>> Now DolphinScheduler has so many Configuration files
>>
>> for example :
>>
>> dolphinscheduler-alert :
>> alert.properties
>>
>> dolphinscheduler-api :
>> application-api.properties
>> application-combined.properties
>>
>>
>> dolphinscheduler-common :
>> hadoop.properties
>> common.properties
>> quartz.properties
>> zookeeper.properties
>>
>> dolphinscheduler-dao :
>> application-dao.properties
>>
>> dolphinscheduler-server :
>> application-master.properties
>> application-master.properties
>> master.properties
>> worker.properties
>>
>> .dolphinscheduler_env.sh
>>
>> Can we simplify deployment ?
>>
>> Main point :
>>
>> 1 configuration file simplification and merged configuration file
>> 2 master server remove port
>> 3 support offline installation，remove kazoo dependencies in install and
>> monitor
>> 4 instll.sh script simplification
>>
>>
>> —————————————
>> DolphinScheduler(Incubator) PPMC
>> Zhanwei Qiao 乔占卫
>>
>> qiaozhanwei@outlook.com
>>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by guo jiwei <te...@gmail.com>.
To ligang.
redeploy is simple, but what about the latency of alert ?
it's easy to redeploy master server to update alert
On Wed, Dec 11, 2019 at 6:03 PM 李 岗 <lg...@outlook.com> wrote:
> I think the alert module can be retained. Currently, it only supports
> email and webchat,
> but more alarm modes can be added in the future.
>
> At present，alert is a independent service. the alert service only consumes
> alarm information in the database, other services produce these alarm
> information.
> If a new alarm mode is added, It is only necessary to redeploy the alert
> service.
>
>
>
> ________________________________
> DolphinScheduler(Incubator) PPMC
> Gang Li 李岗
>
> lgcareer2019@outlook.com<ma...@outlook.com>
>
> 发件人： qiao zhanwei<ma...@outlook.com>
> 发送时间： 2019-12-10 14:24
> 收件人： dev<ma...@dolphinscheduler.apache.org>
> 主题： Aproposal for DolphinScheduler Simplified Deployment
>
> Hello All ,
>
> Now DolphinScheduler has so many Configuration files
>
> for example :
>
> dolphinscheduler-alert :
> alert.properties
>
> dolphinscheduler-api :
> application-api.properties
> application-combined.properties
>
>
> dolphinscheduler-common :
> hadoop.properties
> common.properties
> quartz.properties
> zookeeper.properties
>
> dolphinscheduler-dao :
> application-dao.properties
>
> dolphinscheduler-server :
> application-master.properties
> application-master.properties
> master.properties
> worker.properties
>
> .dolphinscheduler_env.sh
>
> Can we simplify deployment ?
>
> Main point :
>
> 1 configuration file simplification and merged configuration file
> 2 master server remove port
> 3 support offline installation，remove kazoo dependencies in install and
> monitor
> 4 instll.sh script simplification
>
>
> —————————————
> DolphinScheduler(Incubator) PPMC
> Zhanwei Qiao 乔占卫
>
> qiaozhanwei@outlook.com
>
回复: Aproposal for DolphinScheduler Simplified Deployment
Posted by 李 岗 <lg...@outlook.com>.
I think the alert module can be retained. Currently, it only supports email and webchat,
but more alarm modes can be added in the future.
At present，alert is a independent service. the alert service only consumes alarm information in the database, other services produce these alarm information.
If a new alarm mode is added, It is only necessary to redeploy the alert service.
________________________________
DolphinScheduler(Incubator) PPMC
Gang Li 李岗
lgcareer2019@outlook.com<ma...@outlook.com>
发件人： qiao zhanwei<ma...@outlook.com>
发送时间： 2019-12-10 14:24
收件人： dev<ma...@dolphinscheduler.apache.org>
主题： Aproposal for DolphinScheduler Simplified Deployment
Hello All ,
Now DolphinScheduler has so many Configuration files
for example :
dolphinscheduler-alert :
alert.properties
dolphinscheduler-api :
application-api.properties
application-combined.properties
dolphinscheduler-common :
hadoop.properties
common.properties
quartz.properties
zookeeper.properties
dolphinscheduler-dao :
application-dao.properties
dolphinscheduler-server :
application-master.properties
application-master.properties
master.properties
worker.properties
.dolphinscheduler_env.sh
Can we simplify deployment ?
Main point :
1 configuration file simplification and merged configuration file
2 master server remove port
3 support offline installation，remove kazoo dependencies in install and monitor
4 instll.sh script simplification
―――――――――――――
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by guo jiwei <te...@gmail.com>.
To mentor,
yes, we will take care of these.
On Wed, Dec 11, 2019 at 5:54 PM Sheng Wu <wu...@gmail.com> wrote:
> I am saying from an observability system developer perspective.
> Master node as a trigger, is just the source of observability data.
> How and when to alert is totally another story. If you embed the alert
> inside it, you need to take care of the distributed coordination, again.
> In my mind, the alert function should an internal, scheduled by the master
> with the highest priority. It could read data from all master nodes, and
> calculate or do anything you need to do.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午5:36写道：
>
> > Hi handsome mentor.
> > Master node is a common role in distribute system, but in DS, it's not
> very
> > meet. Master node is changing due to zk lock and is now responsible for
> > triggering task, spitting task and sending task command to worker .
> > so we put alert module into server module and triggered by master server
> is
> > suitable.
> > the important problem is the alert implementation. we will change it as
> > the architecture change.
> >
> > PS
> > Alert module is design to trigger task event change like timeout, fail or
> > success etc. and has an individual module.
> >
> >
> > On Wed, Dec 11, 2019 at 5:18 PM Sheng Wu <wu...@gmail.com>
> > wrote:
> >
> > > Ware a normal contributor hat,
> > >
> > > Merging alert into master seems a very strange design. I could get your
> > > point, but I haven't seen any distributed system asking the master
> nodes
> > to
> > > do business logic monitoring.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午5:00写道：
> > >
> > > > Hello everyone .
> > > > Simplify deployment is very nice to users. Currently, we have
> > discussed
> > > > combining or decreasing configuration files . How about deleting
> alert
> > > > module ?
> > > > We can move alert module into server module . MasterServer can take
> > the
> > > > responsibility for the task alert .
> > > >
> > > > General Implementation:
> > > > 1. Combine alert module into server module. Create a new package
> > for
> > > > alert function.
> > > > 2. MasterServer is responsible for alert. When starts
> MasterServer,
> > > > startup alert server.
> > > > 3. We can use zk lock to round-robin doing the alert around the
> > > > MasterServer. or just leader do(for it uses little resource)
> > > >
> > > >
> > > > If this proposal is taken into consideration , users do not to deploy
> > > alert
> > > > module anymore .
> > > >
> > > > On Tue, Dec 10, 2019 at 2:24 PM qiao zhanwei <
> qiaozhanwei@outlook.com>
> > > > wrote:
> > > >
> > > > >
> > > > > Hello All ,
> > > > >
> > > > > Now DolphinScheduler has so many Configuration files
> > > > >
> > > > > for example :
> > > > >
> > > > > dolphinscheduler-alert :
> > > > > alert.properties
> > > > >
> > > > > dolphinscheduler-api :
> > > > > application-api.properties
> > > > > application-combined.properties
> > > > >
> > > > >
> > > > > dolphinscheduler-common :
> > > > > hadoop.properties
> > > > > common.properties
> > > > > quartz.properties
> > > > > zookeeper.properties
> > > > >
> > > > > dolphinscheduler-dao :
> > > > > application-dao.properties
> > > > >
> > > > > dolphinscheduler-server :
> > > > > application-master.properties
> > > > > application-master.properties
> > > > > master.properties
> > > > > worker.properties
> > > > >
> > > > > .dolphinscheduler_env.sh
> > > > >
> > > > > Can we simplify deployment ?
> > > > >
> > > > > Main point :
> > > > >
> > > > > 1 configuration file simplification and merged configuration file
> > > > > 2 master server remove port
> > > > > 3 support offline installation，remove kazoo dependencies in install
> > and
> > > > > monitor
> > > > > 4 instll.sh script simplification
> > > > >
> > > > >
> > > > > —————————————
> > > > > DolphinScheduler(Incubator) PPMC
> > > > > Zhanwei Qiao 乔占卫
> > > > >
> > > > > qiaozhanwei@outlook.com
> > > > >
> > > >
> > >
> >
>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by Sheng Wu <wu...@gmail.com>.
I am saying from an observability system developer perspective.
Master node as a trigger, is just the source of observability data.
How and when to alert is totally another story. If you embed the alert
inside it, you need to take care of the distributed coordination, again.
In my mind, the alert function should an internal, scheduled by the master
with the highest priority. It could read data from all master nodes, and
calculate or do anything you need to do.
Sheng Wu 吴晟
Twitter, wusheng1108
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午5:36写道：
> Hi handsome mentor.
> Master node is a common role in distribute system, but in DS, it's not very
> meet. Master node is changing due to zk lock and is now responsible for
> triggering task, spitting task and sending task command to worker .
> so we put alert module into server module and triggered by master server is
> suitable.
> the important problem is the alert implementation. we will change it as
> the architecture change.
>
> PS
> Alert module is design to trigger task event change like timeout, fail or
> success etc. and has an individual module.
>
>
> On Wed, Dec 11, 2019 at 5:18 PM Sheng Wu <wu...@gmail.com>
> wrote:
>
> > Ware a normal contributor hat,
> >
> > Merging alert into master seems a very strange design. I could get your
> > point, but I haven't seen any distributed system asking the master nodes
> to
> > do business logic monitoring.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午5:00写道：
> >
> > > Hello everyone .
> > > Simplify deployment is very nice to users. Currently, we have
> discussed
> > > combining or decreasing configuration files . How about deleting alert
> > > module ?
> > > We can move alert module into server module . MasterServer can take
> the
> > > responsibility for the task alert .
> > >
> > > General Implementation:
> > > 1. Combine alert module into server module. Create a new package
> for
> > > alert function.
> > > 2. MasterServer is responsible for alert. When starts MasterServer,
> > > startup alert server.
> > > 3. We can use zk lock to round-robin doing the alert around the
> > > MasterServer. or just leader do(for it uses little resource)
> > >
> > >
> > > If this proposal is taken into consideration , users do not to deploy
> > alert
> > > module anymore .
> > >
> > > On Tue, Dec 10, 2019 at 2:24 PM qiao zhanwei <qi...@outlook.com>
> > > wrote:
> > >
> > > >
> > > > Hello All ,
> > > >
> > > > Now DolphinScheduler has so many Configuration files
> > > >
> > > > for example :
> > > >
> > > > dolphinscheduler-alert :
> > > > alert.properties
> > > >
> > > > dolphinscheduler-api :
> > > > application-api.properties
> > > > application-combined.properties
> > > >
> > > >
> > > > dolphinscheduler-common :
> > > > hadoop.properties
> > > > common.properties
> > > > quartz.properties
> > > > zookeeper.properties
> > > >
> > > > dolphinscheduler-dao :
> > > > application-dao.properties
> > > >
> > > > dolphinscheduler-server :
> > > > application-master.properties
> > > > application-master.properties
> > > > master.properties
> > > > worker.properties
> > > >
> > > > .dolphinscheduler_env.sh
> > > >
> > > > Can we simplify deployment ?
> > > >
> > > > Main point :
> > > >
> > > > 1 configuration file simplification and merged configuration file
> > > > 2 master server remove port
> > > > 3 support offline installation，remove kazoo dependencies in install
> and
> > > > monitor
> > > > 4 instll.sh script simplification
> > > >
> > > >
> > > > —————————————
> > > > DolphinScheduler(Incubator) PPMC
> > > > Zhanwei Qiao 乔占卫
> > > >
> > > > qiaozhanwei@outlook.com
> > > >
> > >
> >
>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by guo jiwei <te...@gmail.com>.
Hi handsome mentor.
Master node is a common role in distribute system, but in DS, it's not very
meet. Master node is changing due to zk lock and is now responsible for
triggering task, spitting task and sending task command to worker .
so we put alert module into server module and triggered by master server is
suitable.
the important problem is the alert implementation. we will change it as
the architecture change.
PS
Alert module is design to trigger task event change like timeout, fail or
success etc. and has an individual module.
On Wed, Dec 11, 2019 at 5:18 PM Sheng Wu <wu...@gmail.com> wrote:
> Ware a normal contributor hat,
>
> Merging alert into master seems a very strange design. I could get your
> point, but I haven't seen any distributed system asking the master nodes to
> do business logic monitoring.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午5:00写道：
>
> > Hello everyone .
> > Simplify deployment is very nice to users. Currently, we have discussed
> > combining or decreasing configuration files . How about deleting alert
> > module ?
> > We can move alert module into server module . MasterServer can take the
> > responsibility for the task alert .
> >
> > General Implementation:
> > 1. Combine alert module into server module. Create a new package for
> > alert function.
> > 2. MasterServer is responsible for alert. When starts MasterServer,
> > startup alert server.
> > 3. We can use zk lock to round-robin doing the alert around the
> > MasterServer. or just leader do(for it uses little resource)
> >
> >
> > If this proposal is taken into consideration , users do not to deploy
> alert
> > module anymore .
> >
> > On Tue, Dec 10, 2019 at 2:24 PM qiao zhanwei <qi...@outlook.com>
> > wrote:
> >
> > >
> > > Hello All ,
> > >
> > > Now DolphinScheduler has so many Configuration files
> > >
> > > for example :
> > >
> > > dolphinscheduler-alert :
> > > alert.properties
> > >
> > > dolphinscheduler-api :
> > > application-api.properties
> > > application-combined.properties
> > >
> > >
> > > dolphinscheduler-common :
> > > hadoop.properties
> > > common.properties
> > > quartz.properties
> > > zookeeper.properties
> > >
> > > dolphinscheduler-dao :
> > > application-dao.properties
> > >
> > > dolphinscheduler-server :
> > > application-master.properties
> > > application-master.properties
> > > master.properties
> > > worker.properties
> > >
> > > .dolphinscheduler_env.sh
> > >
> > > Can we simplify deployment ?
> > >
> > > Main point :
> > >
> > > 1 configuration file simplification and merged configuration file
> > > 2 master server remove port
> > > 3 support offline installation，remove kazoo dependencies in install and
> > > monitor
> > > 4 instll.sh script simplification
> > >
> > >
> > > —————————————
> > > DolphinScheduler(Incubator) PPMC
> > > Zhanwei Qiao 乔占卫
> > >
> > > qiaozhanwei@outlook.com
> > >
> >
>
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by Sheng Wu <wu...@gmail.com>.
Ware a normal contributor hat,
Merging alert into master seems a very strange design. I could get your
point, but I haven't seen any distributed system asking the master nodes to
do business logic monitoring.
Sheng Wu 吴晟
Twitter, wusheng1108
guo jiwei <te...@gmail.com> 于2019年12月11日周三 下午5:00写道：
> Hello everyone .
> Simplify deployment is very nice to users. Currently, we have discussed
> combining or decreasing configuration files . How about deleting alert
> module ?
> We can move alert module into server module . MasterServer can take the
> responsibility for the task alert .
>
> General Implementation:
> 1. Combine alert module into server module. Create a new package for
> alert function.
> 2. MasterServer is responsible for alert. When starts MasterServer,
> startup alert server.
> 3. We can use zk lock to round-robin doing the alert around the
> MasterServer. or just leader do(for it uses little resource)
>
>
> If this proposal is taken into consideration , users do not to deploy alert
> module anymore .
>
> On Tue, Dec 10, 2019 at 2:24 PM qiao zhanwei <qi...@outlook.com>
> wrote:
>
> >
> > Hello All ,
> >
> > Now DolphinScheduler has so many Configuration files
> >
> > for example :
> >
> > dolphinscheduler-alert :
> > alert.properties
> >
> > dolphinscheduler-api :
> > application-api.properties
> > application-combined.properties
> >
> >
> > dolphinscheduler-common :
> > hadoop.properties
> > common.properties
> > quartz.properties
> > zookeeper.properties
> >
> > dolphinscheduler-dao :
> > application-dao.properties
> >
> > dolphinscheduler-server :
> > application-master.properties
> > application-master.properties
> > master.properties
> > worker.properties
> >
> > .dolphinscheduler_env.sh
> >
> > Can we simplify deployment ?
> >
> > Main point :
> >
> > 1 configuration file simplification and merged configuration file
> > 2 master server remove port
> > 3 support offline installation，remove kazoo dependencies in install and
> > monitor
> > 4 instll.sh script simplification
> >
> >
> > —————————————
> > DolphinScheduler(Incubator) PPMC
> > Zhanwei Qiao 乔占卫
> >
> > qiaozhanwei@outlook.com
> >
>
回复: Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by qiao zhanwei <qi...@outlook.com>.
Is it more appropriate to put it in common module ?
reason :
1，Now server module email alert, but it may not be ruled out that in the future, for example, the api server module will also use the alert.
2，Another problem is that it is better to store the warning information. That is, it is stored in the database before sending, and then the database is updated after sending.
Can keep some warning information. Of course, except the case where the SQL task sends email directly
―――――――――――――
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
发件人： Xiaochun Liu<ma...@apache.org>
发送时间： 2019-12-11 17:02
收件人： dev<ma...@dolphinscheduler.apache.org>
主题： Re: Aproposal for DolphinScheduler Simplified Deployment
+ 1
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
> 在 2019年12月11日，下午5:00，guo jiwei <te...@gmail.com> 写道：
>
>
> General Implementation:
> 1. Combine alert module into server module. Create a new package for
> alert function.
> 2. MasterServer is responsible for alert. When starts MasterServer,
> startup alert server.
> 3. We can use zk lock to round-robin doing the alert around the
> MasterServer. or just leader do(for it uses little resource)
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by Xiaochun Liu <li...@apache.org>.
+ 1
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
> 在 2019年12月11日，下午5:00，guo jiwei <te...@gmail.com> 写道：
>
>
> General Implementation:
> 1. Combine alert module into server module. Create a new package for
> alert function.
> 2. MasterServer is responsible for alert. When starts MasterServer,
> startup alert server.
> 3. We can use zk lock to round-robin doing the alert around the
> MasterServer. or just leader do(for it uses little resource)
Re: Aproposal for DolphinScheduler Simplified Deployment
Posted by guo jiwei <te...@gmail.com>.
Hello everyone .
Simplify deployment is very nice to users. Currently, we have discussed
combining or decreasing configuration files . How about deleting alert
module ?
We can move alert module into server module . MasterServer can take the
responsibility for the task alert .
General Implementation:
1. Combine alert module into server module. Create a new package for
alert function.
2. MasterServer is responsible for alert. When starts MasterServer,
startup alert server.
3. We can use zk lock to round-robin doing the alert around the
MasterServer. or just leader do(for it uses little resource)
If this proposal is taken into consideration , users do not to deploy alert
module anymore .
On Tue, Dec 10, 2019 at 2:24 PM qiao zhanwei <qi...@outlook.com>
wrote:
>
> Hello All ,
>
> Now DolphinScheduler has so many Configuration files
>
> for example :
>
> dolphinscheduler-alert :
> alert.properties
>
> dolphinscheduler-api :
> application-api.properties
> application-combined.properties
>
>
> dolphinscheduler-common :
> hadoop.properties
> common.properties
> quartz.properties
> zookeeper.properties
>
> dolphinscheduler-dao :
> application-dao.properties
>
> dolphinscheduler-server :
> application-master.properties
> application-master.properties
> master.properties
> worker.properties
>
> .dolphinscheduler_env.sh
>
> Can we simplify deployment ?
>
> Main point :
>
> 1 configuration file simplification and merged configuration file
> 2 master server remove port
> 3 support offline installation，remove kazoo dependencies in install and
> monitor
> 4 instll.sh script simplification
>
>
> —————————————
> DolphinScheduler(Incubator) PPMC
> Zhanwei Qiao 乔占卫
>
> qiaozhanwei@outlook.com
>