You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by leon bao <le...@apache.org> on 2020/11/03 03:47:03 UTC
[DISCUSS]Which part of DolphinScheduler can be more scalable?
Hello Everyone:
DS has good horizontal scalability with its non central design
architecture, which attracts many developers. With more and more users, the
demand for scheduling is becoming more and more complex.
At the same time, the functional design of DS is required to be more
scalable，for example: the plug-in function of alarm mode.
So we can discuss what parts of plug-ins DS can do at present. We can
reconstruct DolphinScheduler according to the results of this discussion.
At present, there are several parts of demand:
- alert model:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- task plugin:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- register center:
refer to:
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- log model:
The current log is recorded by writing local files of the server.
Can we make this plug-in type, which can facilitate users to extend the
log reading and writing types, such as writing to the database or other
third-party systems.
- global task queue
At present, tasks are stored in the memory queue of the master, which
results in the priority of a task can only work within the scope of a
master.
In order to make the priority of a task effective globally, we need a
global queue to make the global priority work.
(in version 1.2, we used zookeeper as the global queue, which was removed
because of the delay of ZK operation)
Implementation details can be discussed within each topic. Here, we only
discuss the requirements.
Very appreciate you can put forward more opinions.
==================================================================================================================================================================
DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
目前已经有需求的几个部分：
- 告警插件(running)
相关讨论：
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- 任务插件
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- 注册中心
相关讨论：
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- 日志插件
目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
- 全局队列插件
目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by sun zhaohe <su...@gmail.com>.
File resource plug-in, the current DS file resource storage supports HDFS, S3 and NONE, making plug-in allows users to customize storage methods to meet various storage needs
====================================================================
文件资源插件化，目前DS的文件资源存储是支持HDFS，S3 和 NONE，做成插件化可以使得用户可自定义存储方式，满足用户个性化存储需求
On 2020/11/03 10:40:30, CalvinKirs <ac...@163.com> wrote:
>
>
> Great planning!
> But I have a little question, what are the specific requirements of the log SPI? I am not very clear at present, are we only implementing SPI for data storage? If this is the case, is it necessary? I think this user can use logagent (or other technologies) for related implementations. Different users have different needs. Some users may also involve aggregation, calculation, and even different magnitudes, and may use additional components. Therefore, if we store the original data in this piece, a lot of redundant data may be generated.
>
>
> 非常棒的规划！
> 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术） 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
>
>
> Best wishes！
> CalvinKirs
>
>
> On 11/3/2020 11:47，leon bao<le...@apache.org> wrote：
> Hello Everyone:
>
> DS has good horizontal scalability with its non central design
> architecture, which attracts many developers. With more and more users, the
> demand for scheduling is becoming more and more complex.
> At the same time, the functional design of DS is required to be more
> scalable，for example: the plug-in function of alarm mode.
> So we can discuss what parts of plug-ins DS can do at present. We can
> reconstruct DolphinScheduler according to the results of this discussion.
> At present, there are several parts of demand:
>
> - alert model:
> refer to:
> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>
> - task plugin:
> refer to:
> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>
> - register center:
> refer to:
>
> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>
> - log model:
> The current log is recorded by writing local files of the server.
> Can we make this plug-in type, which can facilitate users to extend the
> log reading and writing types, such as writing to the database or other
> third-party systems.
>
> - global task queue
> At present, tasks are stored in the memory queue of the master, which
> results in the priority of a task can only work within the scope of a
> master.
> In order to make the priority of a task effective globally, we need a
> global queue to make the global priority work.
> (in version 1.2, we used zookeeper as the global queue, which was removed
> because of the delay of ZK operation)
>
> Implementation details can be discussed within each topic. Here, we only
> discuss the requirements.
> Very appreciate you can put forward more opinions.
>
> ==================================================================================================================================================================
>
> DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
> 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
> 目前已经有需求的几个部分：
>
> - 告警插件(running)
> 相关讨论：
> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>
> - 任务插件
> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>
> - 注册中心
> 相关讨论：
>
> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>
> - 日志插件
> 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
>
> - 全局队列插件
>
> 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
>
> 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
回复：[DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by CalvinKirs <ac...@163.com>.
+1
In fact, before improving spi, I have seen some issue and pr about this part of the content. Contributors want to integrate other storage methods, but the related license may inconformity，Therefore, I support your point of view...
| |
Kirs
|
|
邮箱：acm_master@163.com
|
Signature is customized by Netease Mail Master
在2020年11月10日 17:36，zixi0825 写道：
File resource SPI , the current DS file resource storage supports HDFS, S3 and NONE, making plug-in allows users to customize storage methods to meet various storage needs
And the task spi should be the highest priority
===============================================================================================================
文件资源插件化，目前DS的文件资源存储是支持HDFS，S3 and NONE，做成插件化可以使得用户可自定义存储方式，满足多种存储需求
Task SPI 应该是最高优先级的需求
------------------ 原始邮件 ------------------
发件人: "dev" <18910529250@163.com>;
发送时间: 2020年11月4日(星期三) 上午10:20
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re:Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
log SPI +1
在 2020-11-04 10:07:58，"JUN GAO" <gaojun2048@apache.org> 写道：
>log SPI +1
>
>But I think this is a low priority task, we should give priority to
>complete the registration center SPI and task SPI.
>
>
>
>CalvinKirs <acm_master@163.com> 于2020年11月3日周二 下午11:04写道：
>
>> Thank you very much for your answer, I misunderstood what you meant.
>>
>>
>> log SPI+1
>>
>>
>> Best wishes！
>> CalvinKirs
>>
>>
>> On 11/3/2020 22:55，leon bao<leonbao@apache.org> wrote：
>> Maybe it didn't express my meaning well.
>> Yes, DolphinScheduler doesn't need to do this storage. We just need to do
>> the log plug-in function to support users to implement their own log
>> storage mode, just like we do alert plug-ins.
>>
>>
>> 可能是没有表达好的我的意思。
>> 是的，DolphinScheduler没必要去做这个存储，我们只需要做日志插件功能，支持用户实现自己的日志存储方式，就像我们做告警插件一样。
>>
>>
>> boyi <zhangboyi_mx@163.com> 于2020年11月3日周二 下午10:37写道：
>>
>> hi:
>>
>>
>> Agree with Calvin KIRs
>>
>>
>>
>>
>>
>>
>> Generally, there will be a unified log collection for such files, such as
>> ELK. Especially when deployed in the docker environment, we can consider
>> focusing on more important things
>>
>>
>> --------------------------------------
>> BoYi ZhangE-mail : zhangboyi_mx@163.com
>> On 11/3/2020 22:23，CalvinKirs<acm_master@163.com> wrote：
>> I understand what you mean, but what I want to express is whether it would
>> be better for users to do this part of the work. Generally, companies have
>> their own log collection system. If we were to do this part of the storage,
>> the workload would be great, and the benefits would not be great.
>>
>>
>>
>>
>> 我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。
>>
>>
>> Best wishes！
>> CalvinKirs
>>
>>
>> On 11/3/2020 22:09，leon bao<leonbao@apache.org> wrote：
>> @CalvinKirs
>>
>> about log spi , now have a requirement in scalable
>> services(master/worker), this kind of application scenario requires that
>> the task log cannot be stored in the worker / master, but needs to be
>> stored in a third-party place, which maybe database or other
>> storage. Therefore, if the dolphin scheduler can provide this plug-in
>> function, users can read and write logs according to their own needs.
>>
>>
>> 关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上，
>>
>>
>> 而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写.
>>
>> CalvinKirs <acm_master@163.com> 于2020年11月3日周二 下午6:40写道：
>>
>>
>>
>> Great planning!
>> But I have a little question, what are the specific requirements of the
>> log SPI? I am not very clear at present, are we only implementing SPI for
>> data storage? If this is the case, is it necessary? I think this user can
>> use logagent (or other technologies) for related implementations. Different
>> users have different needs. Some users may also involve aggregation,
>> calculation, and even different magnitudes, and may use additional
>> components. Therefore, if we store the original data in this piece, a lot
>> of redundant data may be generated.
>>
>>
>> 非常棒的规划！
>>
>>
>> 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术）
>>
>>
>> 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
>>
>>
>> Best wishes！
>> CalvinKirs
>>
>>
>> On 11/3/2020 11:47，leon bao<leonbao@apache.org> wrote：
>> Hello Everyone:
>>
>> DS has good horizontal scalability with its non central design
>> architecture, which attracts many developers. With more and more users, the
>> demand for scheduling is becoming more and more complex.
>> At the same time, the functional design of DS is required to be more
>> scalable，for example: the plug-in function of alarm mode.
>> So we can discuss what parts of plug-ins DS can do at present. We can
>> reconstruct DolphinScheduler according to the results of this discussion.
>> At present, there are several parts of demand:
>>
>> - alert model:
>> refer to:
>> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>>
>> - task plugin:
>> refer to:
>> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>>
>> - register center:
>> refer to:
>>
>>
>>
>>
>> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
>> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>>
>> - log model:
>> The current log is recorded by writing local files of the server.
>> Can we make this plug-in type, which can facilitate users to extend the
>> log reading and writing types, such as writing to the database or other
>> third-party systems.
>>
>> - global task queue
>> At present, tasks are stored in the memory queue of the master, which
>> results in the priority of a task can only work within the scope of a
>> master.
>> In order to make the priority of a task effective globally, we need a
>> global queue to make the global priority work.
>> (in version 1.2, we used zookeeper as the global queue, which was removed
>> because of the delay of ZK operation)
>>
>> Implementation details can be discussed within each topic. Here, we only
>> discuss the requirements.
>> Very appreciate you can put forward more opinions.
>>
>>
>>
>>
>> ==================================================================================================================================================================
>>
>>
>>
>>
>> DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
>> 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
>> 目前已经有需求的几个部分：
>>
>> - 告警插件(running)
>> 相关讨论：
>> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>>
>> - 任务插件
>> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>>
>> - 注册中心
>> 相关讨论：
>>
>>
>>
>>
>> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
>> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>>
>> - 日志插件
>> 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
>>
>> - 全局队列插件
>>
>>
>>
>>
>> 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
>>
>> 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
>>
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
>>
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
>
>
>--
>
>DolphinScheduler(Incubator) PPMC
>Jun Gao 高俊
>gaojun2048@gmail.com
回复：Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by zixi0825 <64...@qq.com>.
File resource SPI , the current DS file resource storage supports HDFS, S3 and NONE, making plug-in allows users to customize storage methods to meet various storage needs
And the task spi should be the highest priority
===============================================================================================================
文件资源插件化，目前DS的文件资源存储是支持HDFS，S3 and NONE，做成插件化可以使得用户可自定义存储方式，满足多种存储需求
Task SPI 应该是最高优先级的需求
------------------ 原始邮件 ------------------
发件人: "dev" <18910529250@163.com>;
发送时间: 2020年11月4日(星期三) 上午10:20
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re:Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
log SPI +1
在 2020-11-04 10:07:58，"JUN GAO" <gaojun2048@apache.org> 写道：
>log SPI +1
>
>But I think this is a low priority task, we should give priority to
>complete the registration center SPI and task SPI.
>
>
>
>CalvinKirs <acm_master@163.com> 于2020年11月3日周二 下午11:04写道：
>
>> Thank you very much for your answer, I misunderstood what you meant.
>>
>>
>> log SPI+1
>>
>>
>> Best wishes！
>> CalvinKirs
>>
>>
>> On 11/3/2020 22:55，leon bao<leonbao@apache.org> wrote：
>> Maybe it didn't express my meaning well.
>> Yes, DolphinScheduler doesn't need to do this storage. We just need to do
>> the log plug-in function to support users to implement their own log
>> storage mode, just like we do alert plug-ins.
>>
>>
>> 可能是没有表达好的我的意思。
>> 是的，DolphinScheduler没必要去做这个存储，我们只需要做日志插件功能，支持用户实现自己的日志存储方式，就像我们做告警插件一样。
>>
>>
>> boyi <zhangboyi_mx@163.com> 于2020年11月3日周二 下午10:37写道：
>>
>> hi:
>>
>>
>> Agree with Calvin KIRs
>>
>>
>>
>>
>>
>>
>> Generally, there will be a unified log collection for such files, such as
>> ELK. Especially when deployed in the docker environment, we can consider
>> focusing on more important things
>>
>>
>> --------------------------------------
>> BoYi ZhangE-mail : zhangboyi_mx@163.com
>> On 11/3/2020 22:23，CalvinKirs<acm_master@163.com> wrote：
>> I understand what you mean, but what I want to express is whether it would
>> be better for users to do this part of the work. Generally, companies have
>> their own log collection system. If we were to do this part of the storage,
>> the workload would be great, and the benefits would not be great.
>>
>>
>>
>>
>> 我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。
>>
>>
>> Best wishes！
>> CalvinKirs
>>
>>
>> On 11/3/2020 22:09，leon bao<leonbao@apache.org> wrote：
>> @CalvinKirs
>>
>> about log spi , now have a requirement in scalable
>> services(master/worker), this kind of application scenario requires that
>> the task log cannot be stored in the worker / master, but needs to be
>> stored in a third-party place, which maybe database or other
>> storage. Therefore, if the dolphin scheduler can provide this plug-in
>> function, users can read and write logs according to their own needs.
>>
>>
>> 关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上，
>>
>>
>> 而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写.
>>
>> CalvinKirs <acm_master@163.com> 于2020年11月3日周二 下午6:40写道：
>>
>>
>>
>> Great planning!
>> But I have a little question, what are the specific requirements of the
>> log SPI? I am not very clear at present, are we only implementing SPI for
>> data storage? If this is the case, is it necessary? I think this user can
>> use logagent (or other technologies) for related implementations. Different
>> users have different needs. Some users may also involve aggregation,
>> calculation, and even different magnitudes, and may use additional
>> components. Therefore, if we store the original data in this piece, a lot
>> of redundant data may be generated.
>>
>>
>> 非常棒的规划！
>>
>>
>> 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术）
>>
>>
>> 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
>>
>>
>> Best wishes！
>> CalvinKirs
>>
>>
>> On 11/3/2020 11:47，leon bao<leonbao@apache.org> wrote：
>> Hello Everyone:
>>
>> DS has good horizontal scalability with its non central design
>> architecture, which attracts many developers. With more and more users, the
>> demand for scheduling is becoming more and more complex.
>> At the same time, the functional design of DS is required to be more
>> scalable，for example: the plug-in function of alarm mode.
>> So we can discuss what parts of plug-ins DS can do at present. We can
>> reconstruct DolphinScheduler according to the results of this discussion.
>> At present, there are several parts of demand:
>>
>> - alert model:
>> refer to:
>> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>>
>> - task plugin:
>> refer to:
>> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>>
>> - register center:
>> refer to:
>>
>>
>>
>>
>> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
>> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>>
>> - log model:
>> The current log is recorded by writing local files of the server.
>> Can we make this plug-in type, which can facilitate users to extend the
>> log reading and writing types, such as writing to the database or other
>> third-party systems.
>>
>> - global task queue
>> At present, tasks are stored in the memory queue of the master, which
>> results in the priority of a task can only work within the scope of a
>> master.
>> In order to make the priority of a task effective globally, we need a
>> global queue to make the global priority work.
>> (in version 1.2, we used zookeeper as the global queue, which was removed
>> because of the delay of ZK operation)
>>
>> Implementation details can be discussed within each topic. Here, we only
>> discuss the requirements.
>> Very appreciate you can put forward more opinions.
>>
>>
>>
>>
>> ==================================================================================================================================================================
>>
>>
>>
>>
>> DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
>> 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
>> 目前已经有需求的几个部分：
>>
>> - 告警插件(running)
>> 相关讨论：
>> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>>
>> - 任务插件
>> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>>
>> - 注册中心
>> 相关讨论：
>>
>>
>>
>>
>> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
>> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>>
>> - 日志插件
>> 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
>>
>> - 全局队列插件
>>
>>
>>
>>
>> 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
>>
>> 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
>>
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
>>
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
>
>
>--
>
>DolphinScheduler(Incubator) PPMC
>Jun Gao 高俊
>gaojun2048@gmail.com
Re:Re: [DISCUSS]Which part of DolphinScheduler can be more
scalable?
Posted by zhangchunyang <18...@163.com>.
log SPI +1
在 2020-11-04 10:07:58，"JUN GAO" <ga...@apache.org> 写道：
>log SPI +1
>
>But I think this is a low priority task, we should give priority to
>complete the registration center SPI and task SPI.
>
>
>
>CalvinKirs <ac...@163.com> 于2020年11月3日周二 下午11:04写道：
>
>> Thank you very much for your answer, I misunderstood what you meant.
>>
>>
>> log SPI+1
>>
>>
>> Best wishes！
>> CalvinKirs
>>
>>
>> On 11/3/2020 22:55，leon bao<le...@apache.org> wrote：
>> Maybe it didn't express my meaning well.
>> Yes, DolphinScheduler doesn't need to do this storage. We just need to do
>> the log plug-in function to support users to implement their own log
>> storage mode, just like we do alert plug-ins.
>>
>>
>> 可能是没有表达好的我的意思。
>> 是的，DolphinScheduler没必要去做这个存储，我们只需要做日志插件功能，支持用户实现自己的日志存储方式，就像我们做告警插件一样。
>>
>>
>> boyi <zh...@163.com> 于2020年11月3日周二 下午10:37写道：
>>
>> hi:
>>
>>
>> Agree with Calvin KIRs
>>
>>
>>
>>
>>
>>
>> Generally, there will be a unified log collection for such files, such as
>> ELK. Especially when deployed in the docker environment, we can consider
>> focusing on more important things
>>
>>
>> --------------------------------------
>> BoYi ZhangE-mail : zhangboyi_mx@163.com
>> On 11/3/2020 22:23，CalvinKirs<ac...@163.com> wrote：
>> I understand what you mean, but what I want to express is whether it would
>> be better for users to do this part of the work. Generally, companies have
>> their own log collection system. If we were to do this part of the storage,
>> the workload would be great, and the benefits would not be great.
>>
>>
>>
>>
>> 我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。
>>
>>
>> Best wishes！
>> CalvinKirs
>>
>>
>> On 11/3/2020 22:09，leon bao<le...@apache.org> wrote：
>> @CalvinKirs
>>
>> about log spi , now have a requirement in scalable
>> services(master/worker), this kind of application scenario requires that
>> the task log cannot be stored in the worker / master, but needs to be
>> stored in a third-party place, which maybe database or other
>> storage. Therefore, if the dolphin scheduler can provide this plug-in
>> function, users can read and write logs according to their own needs.
>>
>>
>> 关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上，
>>
>>
>> 而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写.
>>
>> CalvinKirs <ac...@163.com> 于2020年11月3日周二 下午6:40写道：
>>
>>
>>
>> Great planning!
>> But I have a little question, what are the specific requirements of the
>> log SPI? I am not very clear at present, are we only implementing SPI for
>> data storage? If this is the case, is it necessary? I think this user can
>> use logagent (or other technologies) for related implementations. Different
>> users have different needs. Some users may also involve aggregation,
>> calculation, and even different magnitudes, and may use additional
>> components. Therefore, if we store the original data in this piece, a lot
>> of redundant data may be generated.
>>
>>
>> 非常棒的规划！
>>
>>
>> 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术）
>>
>>
>> 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
>>
>>
>> Best wishes！
>> CalvinKirs
>>
>>
>> On 11/3/2020 11:47，leon bao<le...@apache.org> wrote：
>> Hello Everyone:
>>
>> DS has good horizontal scalability with its non central design
>> architecture, which attracts many developers. With more and more users, the
>> demand for scheduling is becoming more and more complex.
>> At the same time, the functional design of DS is required to be more
>> scalable，for example: the plug-in function of alarm mode.
>> So we can discuss what parts of plug-ins DS can do at present. We can
>> reconstruct DolphinScheduler according to the results of this discussion.
>> At present, there are several parts of demand:
>>
>> - alert model:
>> refer to:
>> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>>
>> - task plugin:
>> refer to:
>> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>>
>> - register center:
>> refer to:
>>
>>
>>
>>
>> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
>> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>>
>> - log model:
>> The current log is recorded by writing local files of the server.
>> Can we make this plug-in type, which can facilitate users to extend the
>> log reading and writing types, such as writing to the database or other
>> third-party systems.
>>
>> - global task queue
>> At present, tasks are stored in the memory queue of the master, which
>> results in the priority of a task can only work within the scope of a
>> master.
>> In order to make the priority of a task effective globally, we need a
>> global queue to make the global priority work.
>> (in version 1.2, we used zookeeper as the global queue, which was removed
>> because of the delay of ZK operation)
>>
>> Implementation details can be discussed within each topic. Here, we only
>> discuss the requirements.
>> Very appreciate you can put forward more opinions.
>>
>>
>>
>>
>> ==================================================================================================================================================================
>>
>>
>>
>>
>> DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
>> 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
>> 目前已经有需求的几个部分：
>>
>> - 告警插件(running)
>> 相关讨论：
>> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>>
>> - 任务插件
>> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>>
>> - 注册中心
>> 相关讨论：
>>
>>
>>
>>
>> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
>> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>>
>> - 日志插件
>> 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
>>
>> - 全局队列插件
>>
>>
>>
>>
>> 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
>>
>> 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
>>
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
>>
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang 鲍亮
>> leonbao@apache.org
>>
>
>
>--
>
>DolphinScheduler(Incubator) PPMC
>Jun Gao 高俊
>gaojun2048@gmail.com
Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by JUN GAO <ga...@apache.org>.
log SPI +1
But I think this is a low priority task, we should give priority to
complete the registration center SPI and task SPI.
CalvinKirs <ac...@163.com> 于2020年11月3日周二 下午11:04写道：
> Thank you very much for your answer, I misunderstood what you meant.
>
>
> log SPI+1
>
>
> Best wishes！
> CalvinKirs
>
>
> On 11/3/2020 22:55，leon bao<le...@apache.org> wrote：
> Maybe it didn't express my meaning well.
> Yes, DolphinScheduler doesn't need to do this storage. We just need to do
> the log plug-in function to support users to implement their own log
> storage mode, just like we do alert plug-ins.
>
>
> 可能是没有表达好的我的意思。
> 是的，DolphinScheduler没必要去做这个存储，我们只需要做日志插件功能，支持用户实现自己的日志存储方式，就像我们做告警插件一样。
>
>
> boyi <zh...@163.com> 于2020年11月3日周二 下午10:37写道：
>
> hi:
>
>
> Agree with Calvin KIRs
>
>
>
>
>
>
> Generally, there will be a unified log collection for such files, such as
> ELK. Especially when deployed in the docker environment, we can consider
> focusing on more important things
>
>
> --------------------------------------
> BoYi ZhangE-mail : zhangboyi_mx@163.com
> On 11/3/2020 22:23，CalvinKirs<ac...@163.com> wrote：
> I understand what you mean, but what I want to express is whether it would
> be better for users to do this part of the work. Generally, companies have
> their own log collection system. If we were to do this part of the storage,
> the workload would be great, and the benefits would not be great.
>
>
>
>
> 我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。
>
>
> Best wishes！
> CalvinKirs
>
>
> On 11/3/2020 22:09，leon bao<le...@apache.org> wrote：
> @CalvinKirs
>
> about log spi , now have a requirement in scalable
> services(master/worker), this kind of application scenario requires that
> the task log cannot be stored in the worker / master, but needs to be
> stored in a third-party place, which maybe database or other
> storage. Therefore, if the dolphin scheduler can provide this plug-in
> function, users can read and write logs according to their own needs.
>
>
> 关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上，
>
>
> 而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写.
>
> CalvinKirs <ac...@163.com> 于2020年11月3日周二 下午6:40写道：
>
>
>
> Great planning!
> But I have a little question, what are the specific requirements of the
> log SPI? I am not very clear at present, are we only implementing SPI for
> data storage? If this is the case, is it necessary? I think this user can
> use logagent (or other technologies) for related implementations. Different
> users have different needs. Some users may also involve aggregation,
> calculation, and even different magnitudes, and may use additional
> components. Therefore, if we store the original data in this piece, a lot
> of redundant data may be generated.
>
>
> 非常棒的规划！
>
>
> 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术）
>
>
> 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
>
>
> Best wishes！
> CalvinKirs
>
>
> On 11/3/2020 11:47，leon bao<le...@apache.org> wrote：
> Hello Everyone:
>
> DS has good horizontal scalability with its non central design
> architecture, which attracts many developers. With more and more users, the
> demand for scheduling is becoming more and more complex.
> At the same time, the functional design of DS is required to be more
> scalable，for example: the plug-in function of alarm mode.
> So we can discuss what parts of plug-ins DS can do at present. We can
> reconstruct DolphinScheduler according to the results of this discussion.
> At present, there are several parts of demand:
>
> - alert model:
> refer to:
> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>
> - task plugin:
> refer to:
> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>
> - register center:
> refer to:
>
>
>
>
> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>
> - log model:
> The current log is recorded by writing local files of the server.
> Can we make this plug-in type, which can facilitate users to extend the
> log reading and writing types, such as writing to the database or other
> third-party systems.
>
> - global task queue
> At present, tasks are stored in the memory queue of the master, which
> results in the priority of a task can only work within the scope of a
> master.
> In order to make the priority of a task effective globally, we need a
> global queue to make the global priority work.
> (in version 1.2, we used zookeeper as the global queue, which was removed
> because of the delay of ZK operation)
>
> Implementation details can be discussed within each topic. Here, we only
> discuss the requirements.
> Very appreciate you can put forward more opinions.
>
>
>
>
> ==================================================================================================================================================================
>
>
>
>
> DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
> 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
> 目前已经有需求的几个部分：
>
> - 告警插件(running)
> 相关讨论：
> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>
> - 任务插件
> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>
> - 注册中心
> 相关讨论：
>
>
>
>
> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>
> - 日志插件
> 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
>
> - 全局队列插件
>
>
>
>
> 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
>
> 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
>
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
>
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
--
DolphinScheduler(Incubator) PPMC
Jun Gao 高俊
gaojun2048@gmail.com
Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by CalvinKirs <ac...@163.com>.
Thank you very much for your answer, I misunderstood what you meant.
log SPI+1
Best wishes！
CalvinKirs
On 11/3/2020 22:55，leon bao<le...@apache.org> wrote：
Maybe it didn't express my meaning well.
Yes, DolphinScheduler doesn't need to do this storage. We just need to do
the log plug-in function to support users to implement their own log
storage mode, just like we do alert plug-ins.
可能是没有表达好的我的意思。
是的，DolphinScheduler没必要去做这个存储，我们只需要做日志插件功能，支持用户实现自己的日志存储方式，就像我们做告警插件一样。
boyi <zh...@163.com> 于2020年11月3日周二 下午10:37写道：
hi:
Agree with Calvin KIRs
Generally, there will be a unified log collection for such files, such as
ELK. Especially when deployed in the docker environment, we can consider
focusing on more important things
--------------------------------------
BoYi ZhangE-mail : zhangboyi_mx@163.com
On 11/3/2020 22:23，CalvinKirs<ac...@163.com> wrote：
I understand what you mean, but what I want to express is whether it would
be better for users to do this part of the work. Generally, companies have
their own log collection system. If we were to do this part of the storage,
the workload would be great, and the benefits would not be great.
我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。
Best wishes！
CalvinKirs
On 11/3/2020 22:09，leon bao<le...@apache.org> wrote：
@CalvinKirs
about log spi , now have a requirement in scalable
services(master/worker), this kind of application scenario requires that
the task log cannot be stored in the worker / master, but needs to be
stored in a third-party place, which maybe database or other
storage. Therefore, if the dolphin scheduler can provide this plug-in
function, users can read and write logs according to their own needs.
关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上，
而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写.
CalvinKirs <ac...@163.com> 于2020年11月3日周二 下午6:40写道：
Great planning!
But I have a little question, what are the specific requirements of the
log SPI? I am not very clear at present, are we only implementing SPI for
data storage? If this is the case, is it necessary? I think this user can
use logagent (or other technologies) for related implementations. Different
users have different needs. Some users may also involve aggregation,
calculation, and even different magnitudes, and may use additional
components. Therefore, if we store the original data in this piece, a lot
of redundant data may be generated.
非常棒的规划！
但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术）
来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
Best wishes！
CalvinKirs
On 11/3/2020 11:47，leon bao<le...@apache.org> wrote：
Hello Everyone:
DS has good horizontal scalability with its non central design
architecture, which attracts many developers. With more and more users, the
demand for scheduling is becoming more and more complex.
At the same time, the functional design of DS is required to be more
scalable，for example: the plug-in function of alarm mode.
So we can discuss what parts of plug-ins DS can do at present. We can
reconstruct DolphinScheduler according to the results of this discussion.
At present, there are several parts of demand:
- alert model:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- task plugin:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- register center:
refer to:
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- log model:
The current log is recorded by writing local files of the server.
Can we make this plug-in type, which can facilitate users to extend the
log reading and writing types, such as writing to the database or other
third-party systems.
- global task queue
At present, tasks are stored in the memory queue of the master, which
results in the priority of a task can only work within the scope of a
master.
In order to make the priority of a task effective globally, we need a
global queue to make the global priority work.
(in version 1.2, we used zookeeper as the global queue, which was removed
because of the delay of ZK operation)
Implementation details can be discussed within each topic. Here, we only
discuss the requirements.
Very appreciate you can put forward more opinions.
==================================================================================================================================================================
DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
目前已经有需求的几个部分：
- 告警插件(running)
相关讨论：
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- 任务插件
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- 注册中心
相关讨论：
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- 日志插件
目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
- 全局队列插件
目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by leon bao <le...@apache.org>.
Maybe it didn't express my meaning well.
Yes, DolphinScheduler doesn't need to do this storage. We just need to do
the log plug-in function to support users to implement their own log
storage mode, just like we do alert plug-ins.
可能是没有表达好的我的意思。
是的，DolphinScheduler没必要去做这个存储，我们只需要做日志插件功能，支持用户实现自己的日志存储方式，就像我们做告警插件一样。
boyi <zh...@163.com> 于2020年11月3日周二 下午10:37写道：
> hi:
>
>
> Agree with Calvin KIRs
>
>
>
>
>
>
> Generally, there will be a unified log collection for such files, such as
> ELK. Especially when deployed in the docker environment, we can consider
> focusing on more important things
>
>
> --------------------------------------
> BoYi ZhangE-mail : zhangboyi_mx@163.com
> On 11/3/2020 22:23，CalvinKirs<ac...@163.com> wrote：
> I understand what you mean, but what I want to express is whether it would
> be better for users to do this part of the work. Generally, companies have
> their own log collection system. If we were to do this part of the storage,
> the workload would be great, and the benefits would not be great.
>
>
>
> 我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。
>
>
> Best wishes！
> CalvinKirs
>
>
> On 11/3/2020 22:09，leon bao<le...@apache.org> wrote：
> @CalvinKirs
>
> about log spi , now have a requirement in scalable
> services(master/worker), this kind of application scenario requires that
> the task log cannot be stored in the worker / master, but needs to be
> stored in a third-party place, which maybe database or other
> storage. Therefore, if the dolphin scheduler can provide this plug-in
> function, users can read and write logs according to their own needs.
>
>
> 关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上，
>
> 而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写.
>
> CalvinKirs <ac...@163.com> 于2020年11月3日周二 下午6:40写道：
>
>
>
> Great planning!
> But I have a little question, what are the specific requirements of the
> log SPI? I am not very clear at present, are we only implementing SPI for
> data storage? If this is the case, is it necessary? I think this user can
> use logagent (or other technologies) for related implementations. Different
> users have different needs. Some users may also involve aggregation,
> calculation, and even different magnitudes, and may use additional
> components. Therefore, if we store the original data in this piece, a lot
> of redundant data may be generated.
>
>
> 非常棒的规划！
>
> 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术）
>
> 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
>
>
> Best wishes！
> CalvinKirs
>
>
> On 11/3/2020 11:47，leon bao<le...@apache.org> wrote：
> Hello Everyone:
>
> DS has good horizontal scalability with its non central design
> architecture, which attracts many developers. With more and more users, the
> demand for scheduling is becoming more and more complex.
> At the same time, the functional design of DS is required to be more
> scalable，for example: the plug-in function of alarm mode.
> So we can discuss what parts of plug-ins DS can do at present. We can
> reconstruct DolphinScheduler according to the results of this discussion.
> At present, there are several parts of demand:
>
> - alert model:
> refer to:
> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>
> - task plugin:
> refer to:
> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>
> - register center:
> refer to:
>
>
>
> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>
> - log model:
> The current log is recorded by writing local files of the server.
> Can we make this plug-in type, which can facilitate users to extend the
> log reading and writing types, such as writing to the database or other
> third-party systems.
>
> - global task queue
> At present, tasks are stored in the memory queue of the master, which
> results in the priority of a task can only work within the scope of a
> master.
> In order to make the priority of a task effective globally, we need a
> global queue to make the global priority work.
> (in version 1.2, we used zookeeper as the global queue, which was removed
> because of the delay of ZK operation)
>
> Implementation details can be discussed within each topic. Here, we only
> discuss the requirements.
> Very appreciate you can put forward more opinions.
>
>
>
> ==================================================================================================================================================================
>
>
>
> DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
> 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
> 目前已经有需求的几个部分：
>
> - 告警插件(running)
> 相关讨论：
> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>
> - 任务插件
> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>
> - 注册中心
> 相关讨论：
>
>
>
> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>
> - 日志插件
> 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
>
> - 全局队列插件
>
>
>
> 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
>
> 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
>
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by boyi <zh...@163.com>.
hi:
Agree with Calvin KIRs
Generally, there will be a unified log collection for such files, such as ELK. Especially when deployed in the docker environment, we can consider focusing on more important things
--------------------------------------
BoYi ZhangE-mail : zhangboyi_mx@163.com
On 11/3/2020 22:23，CalvinKirs<ac...@163.com> wrote：
I understand what you mean, but what I want to express is whether it would be better for users to do this part of the work. Generally, companies have their own log collection system. If we were to do this part of the storage, the workload would be great, and the benefits would not be great.
我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。
Best wishes！
CalvinKirs
On 11/3/2020 22:09，leon bao<le...@apache.org> wrote：
@CalvinKirs
about log spi , now have a requirement in scalable
services(master/worker), this kind of application scenario requires that
the task log cannot be stored in the worker / master, but needs to be
stored in a third-party place, which maybe database or other
storage. Therefore, if the dolphin scheduler can provide this plug-in
function, users can read and write logs according to their own needs.
关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上，
而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写.
CalvinKirs <ac...@163.com> 于2020年11月3日周二 下午6:40写道：
Great planning!
But I have a little question, what are the specific requirements of the
log SPI? I am not very clear at present, are we only implementing SPI for
data storage? If this is the case, is it necessary? I think this user can
use logagent (or other technologies) for related implementations. Different
users have different needs. Some users may also involve aggregation,
calculation, and even different magnitudes, and may use additional
components. Therefore, if we store the original data in this piece, a lot
of redundant data may be generated.
非常棒的规划！
但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术）
来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
Best wishes！
CalvinKirs
On 11/3/2020 11:47，leon bao<le...@apache.org> wrote：
Hello Everyone:
DS has good horizontal scalability with its non central design
architecture, which attracts many developers. With more and more users, the
demand for scheduling is becoming more and more complex.
At the same time, the functional design of DS is required to be more
scalable，for example: the plug-in function of alarm mode.
So we can discuss what parts of plug-ins DS can do at present. We can
reconstruct DolphinScheduler according to the results of this discussion.
At present, there are several parts of demand:
- alert model:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- task plugin:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- register center:
refer to:
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- log model:
The current log is recorded by writing local files of the server.
Can we make this plug-in type, which can facilitate users to extend the
log reading and writing types, such as writing to the database or other
third-party systems.
- global task queue
At present, tasks are stored in the memory queue of the master, which
results in the priority of a task can only work within the scope of a
master.
In order to make the priority of a task effective globally, we need a
global queue to make the global priority work.
(in version 1.2, we used zookeeper as the global queue, which was removed
because of the delay of ZK operation)
Implementation details can be discussed within each topic. Here, we only
discuss the requirements.
Very appreciate you can put forward more opinions.
==================================================================================================================================================================
DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
目前已经有需求的几个部分：
- 告警插件(running)
相关讨论：
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- 任务插件
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- 注册中心
相关讨论：
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- 日志插件
目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
- 全局队列插件
目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by CalvinKirs <ac...@163.com>.
I understand what you mean, but what I want to express is whether it would be better for users to do this part of the work. Generally, companies have their own log collection system. If we were to do this part of the storage, the workload would be great, and the benefits would not be great.
我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。
Best wishes！
CalvinKirs
On 11/3/2020 22:09，leon bao<le...@apache.org> wrote：
@CalvinKirs
about log spi , now have a requirement in scalable
services(master/worker), this kind of application scenario requires that
the task log cannot be stored in the worker / master, but needs to be
stored in a third-party place, which maybe database or other
storage. Therefore, if the dolphin scheduler can provide this plug-in
function, users can read and write logs according to their own needs.
关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上，
而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写.
CalvinKirs <ac...@163.com> 于2020年11月3日周二 下午6:40写道：
Great planning!
But I have a little question, what are the specific requirements of the
log SPI? I am not very clear at present, are we only implementing SPI for
data storage? If this is the case, is it necessary? I think this user can
use logagent (or other technologies) for related implementations. Different
users have different needs. Some users may also involve aggregation,
calculation, and even different magnitudes, and may use additional
components. Therefore, if we store the original data in this piece, a lot
of redundant data may be generated.
非常棒的规划！
但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术）
来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
Best wishes！
CalvinKirs
On 11/3/2020 11:47，leon bao<le...@apache.org> wrote：
Hello Everyone:
DS has good horizontal scalability with its non central design
architecture, which attracts many developers. With more and more users, the
demand for scheduling is becoming more and more complex.
At the same time, the functional design of DS is required to be more
scalable，for example: the plug-in function of alarm mode.
So we can discuss what parts of plug-ins DS can do at present. We can
reconstruct DolphinScheduler according to the results of this discussion.
At present, there are several parts of demand:
- alert model:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- task plugin:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- register center:
refer to:
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- log model:
The current log is recorded by writing local files of the server.
Can we make this plug-in type, which can facilitate users to extend the
log reading and writing types, such as writing to the database or other
third-party systems.
- global task queue
At present, tasks are stored in the memory queue of the master, which
results in the priority of a task can only work within the scope of a
master.
In order to make the priority of a task effective globally, we need a
global queue to make the global priority work.
(in version 1.2, we used zookeeper as the global queue, which was removed
because of the delay of ZK operation)
Implementation details can be discussed within each topic. Here, we only
discuss the requirements.
Very appreciate you can put forward more opinions.
==================================================================================================================================================================
DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
目前已经有需求的几个部分：
- 告警插件(running)
相关讨论：
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- 任务插件
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- 注册中心
相关讨论：
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- 日志插件
目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
- 全局队列插件
目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by leon bao <le...@apache.org>.
@CalvinKirs
about log spi , now have a requirement in scalable
services(master/worker), this kind of application scenario requires that
the task log cannot be stored in the worker / master, but needs to be
stored in a third-party place, which maybe database or other
storage. Therefore, if the dolphin scheduler can provide this plug-in
function, users can read and write logs according to their own needs.
关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上，
而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写.
CalvinKirs <ac...@163.com> 于2020年11月3日周二 下午6:40写道：
>
>
> Great planning!
> But I have a little question, what are the specific requirements of the
> log SPI? I am not very clear at present, are we only implementing SPI for
> data storage? If this is the case, is it necessary? I think this user can
> use logagent (or other technologies) for related implementations. Different
> users have different needs. Some users may also involve aggregation,
> calculation, and even different magnitudes, and may use additional
> components. Therefore, if we store the original data in this piece, a lot
> of redundant data may be generated.
>
>
> 非常棒的规划！
> 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术）
> 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
>
>
> Best wishes！
> CalvinKirs
>
>
> On 11/3/2020 11:47，leon bao<le...@apache.org> wrote：
> Hello Everyone:
>
> DS has good horizontal scalability with its non central design
> architecture, which attracts many developers. With more and more users, the
> demand for scheduling is becoming more and more complex.
> At the same time, the functional design of DS is required to be more
> scalable，for example: the plug-in function of alarm mode.
> So we can discuss what parts of plug-ins DS can do at present. We can
> reconstruct DolphinScheduler according to the results of this discussion.
> At present, there are several parts of demand:
>
> - alert model:
> refer to:
> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>
> - task plugin:
> refer to:
> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>
> - register center:
> refer to:
>
>
> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>
> - log model:
> The current log is recorded by writing local files of the server.
> Can we make this plug-in type, which can facilitate users to extend the
> log reading and writing types, such as writing to the database or other
> third-party systems.
>
> - global task queue
> At present, tasks are stored in the memory queue of the master, which
> results in the priority of a task can only work within the scope of a
> master.
> In order to make the priority of a task effective globally, we need a
> global queue to make the global priority work.
> (in version 1.2, we used zookeeper as the global queue, which was removed
> because of the delay of ZK operation)
>
> Implementation details can be discussed within each topic. Here, we only
> discuss the requirements.
> Very appreciate you can put forward more opinions.
>
>
> ==================================================================================================================================================================
>
>
> DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
> 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
> 目前已经有需求的几个部分：
>
> - 告警插件(running)
> 相关讨论：
> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>
> - 任务插件
> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>
> - 注册中心
> 相关讨论：
>
>
> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>
> - 日志插件
> 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
>
> - 全局队列插件
>
>
> 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
>
> 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
>
> --
> DolphinScheduler(Incubator) PPMC
> BaoLiang 鲍亮
> leonbao@apache.org
>
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org
Re:[DISCUSS]Which part of DolphinScheduler can be more scalable?
Posted by CalvinKirs <ac...@163.com>.
Great planning!
But I have a little question, what are the specific requirements of the log SPI? I am not very clear at present, are we only implementing SPI for data storage? If this is the case, is it necessary? I think this user can use logagent (or other technologies) for related implementations. Different users have different needs. Some users may also involve aggregation, calculation, and even different magnitudes, and may use additional components. Therefore, if we store the original data in this piece, a lot of redundant data may be generated.
非常棒的规划！
但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术） 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。
Best wishes！
CalvinKirs
On 11/3/2020 11:47，leon bao<le...@apache.org> wrote：
Hello Everyone:
DS has good horizontal scalability with its non central design
architecture, which attracts many developers. With more and more users, the
demand for scheduling is becoming more and more complex.
At the same time, the functional design of DS is required to be more
scalable，for example: the plug-in function of alarm mode.
So we can discuss what parts of plug-ins DS can do at present. We can
reconstruct DolphinScheduler according to the results of this discussion.
At present, there are several parts of demand:
- alert model:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- task plugin:
refer to:
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- register center:
refer to:
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- log model:
The current log is recorded by writing local files of the server.
Can we make this plug-in type, which can facilitate users to extend the
log reading and writing types, such as writing to the database or other
third-party systems.
- global task queue
At present, tasks are stored in the memory queue of the master, which
results in the priority of a task can only work within the scope of a
master.
In order to make the priority of a task effective globally, we need a
global queue to make the global priority work.
(in version 1.2, we used zookeeper as the global queue, which was removed
because of the delay of ZK operation)
Implementation details can be discussed within each topic. Here, we only
discuss the requirements.
Very appreciate you can put forward more opinions.
==================================================================================================================================================================
DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性
比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。
目前已经有需求的几个部分：
- 告警插件(running)
相关讨论：
https://github.com/apache/incubator-dolphinscheduler/issues/3049
- 任务插件
https://github.com/apache/incubator-dolphinscheduler/issues/2869
- 注册中心
相关讨论：
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
https://github.com/apache/incubator-dolphinscheduler/issues/3961
- 日志插件
目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。
- 全局队列插件
目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。
实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。
--
DolphinScheduler(Incubator) PPMC
BaoLiang 鲍亮
leonbao@apache.org