You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by "xingchun.chen" <xi...@qq.com> on 2020/09/02 11:14:39 UTC
回复： [VOTE]next version what to do
Hi, all:
I have sorted out the unresolved bugs in the git issue, as follows, welcome to claim them：
1.[Bug][master] After batch deleting the executing process instances, the master cannot get the worker feedback results. [https://github.com/apache/incubator-dolphinscheduler/issues/3621]
2.[Bug][worker] Run tens of thousands of workflows, too many open files. [https://github.com/apache/incubator-dolphinscheduler/issues/3618]
3.[Bug][master] After subtask fault tolerance, 2 task instances are generated，The process instance status always is executing. [https://github.com/apache/incubator-dolphinscheduler/issues/3617]
4.[Bug][master] When the master receives the result from the worker, the master just stops. After the master restarts, the workflow status is always executing. [https://github.com/apache/incubator-dolphinscheduler/issues/3616]
5.[Bug][master] After the task is executed successfully, but the next task has not been submitted, stop the master，the workflow will fail. [https://github.com/apache/incubator-dolphinscheduler/issues/3615]
6.[Bug][api] potential horizontal unauthorized access. [https://github.com/apache/incubator-dolphinscheduler/issues/3573]
7.[Bug][api] worker group manage ui could not display createtime and updatetime. [https://github.com/apache/incubator-dolphinscheduler/issues/3493]
8.[Bug][api] ClassCastException: AppClassLoader cannot be cast to class java.net.URLClassLoader. [https://github.com/apache/incubator-dolphinscheduler/issues/3233]
9.[Bug][server] if resultset has null value of some column, fastxml.jackson may ocurr exception. [https://github.com/apache/incubator-dolphinscheduler/issues/3436]
10.[Bug][api] Execute Hive UDF to report errors with tez. [https://github.com/apache/incubator-dolphinscheduler/issues/3455]
11.[Bug][Master] there exists some problems in checking task dependency. [https://github.com/apache/incubator-dolphinscheduler/issues/3324]
12.[Bug][Server] when the worker and master are configured with different data.basedir.path, it will cause the task to fail. [https://github.com/apache/incubator-dolphinscheduler/issues/3257]
13.[Bug][Resource Center] Can not create folder in docker with standalone mode. [https://github.com/apache/incubator-dolphinscheduler/issues/3238]
14.[BUG][api] The shellTask scheduler reruns the date expression in the script to the current system time. [https://github.com/apache/incubator-dolphinscheduler/issues/3177]
15.[BUG][worker] python task problem. [https://github.com/apache/incubator-dolphinscheduler/issues/2769]
16.[BUG][master] MasterExecThread frequently create/destroy thread pool. [https://github.com/apache/incubator-dolphinscheduler/issues/2711]
17.[Bug][API] OOM occurs when uploading large files. [https://github.com/apache/incubator-dolphinscheduler/issues/3643]
18.[bug][worker] Implementation Flink problem. [https://github.com/apache/incubator-dolphinscheduler/issues/3457]
19.[Bug][k8s] when recreate or upgrade helm release, the host ip changed. [https://github.com/apache/incubator-dolphinscheduler/issues/3298]
20.[Bug][docker][k8s] when use externalDatabase ,helm run error. [https://github.com/apache/incubator-dolphinscheduler/issues/3239]
21.[Bug][docker] the file requirements.yaml in helm mode has error flag. [https://github.com/apache/incubator-dolphinscheduler/issues/3237]
22.[BUG][docker] a error when build a docker image. [https://github.com/apache/incubator-dolphinscheduler/issues/3124]
23.[BUG][docker] DS1.3 in docker, ApiApplicationServer can't listen to 12345 , but 9090. [https://github.com/apache/incubator-dolphinscheduler/issues/3023]
Best Regards!
Xingchun Chen
------------------ 原始邮件 ------------------
发件人: "dev" <1048262223@qq.com>;
发送时间: 2020年9月2日(星期三) 中午12:54
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re: [VOTE]next version what to do
Hi,
state machine processor of worker and master optimize
+1
At present, there is a lot of bug type issues about the basic scheduling,&nbsp;
List some Issue example from github, there is a lot issue question about task instance running status. Why there will appear two same task instance. Why the process instance status is still runing when it fail. Why master kill itself......
In my opinion, we can first focus on the optimize about the basic scheduling to avoid those bugs, after that, we can investigate&nbsp;ecology.
Best,
Yichao Yang
------------------ Original ------------------
From: zixi0825 <649790970@qq.com&gt;
Date: Wed,Sep 2,2020 11:34 AM
To: dev <dev@dolphinscheduler.apache.org&gt;
Subject: Re: [VOTE]next version what to do
3:SQL task support multiple sqls and parameter passing optimization(we should add a new task type, such as SqlScript @gabrywu @qiaozhanwei @yangyichao_mango)[need discuss]
8:SPI requirements are collected and developed. We need to have a discussion to find out the modules that currently need SPI most(@gaojun_2048 @lenboo @CalvinKirs)
+1
------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
发件人:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "dev"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <acm_master@163.com&amp;gt;;
发送时间:&amp;nbsp;2020年9月2日(星期三) 凌晨1:04
收件人:&amp;nbsp;"dev"<dev@dolphinscheduler.apache.org&amp;gt;;
主题:&amp;nbsp;[VOTE]next version what to do
Hi, everybody. In the email about what we need to do in the next version, we collected the following work. We hope that everyone can discuss the priority of related issues and the specific implementation of the plan. Each job is accompanied by related proposers and advisers. If you are interested, you can openly communicate with them. (You can communicate via email for a certain sub-item, and the email is only for voting)
1:Api Server communicate for Master instand of scan command table(@qiaozhanwei)
2:Alert module support service for other service invoke(@qiaozhanwei @gaojun_2048)
3:SQL task support multiple sqls and parameter passing optimization(we should add a new task type, such as SqlScript @gabrywu @qiaozhanwei @yangyichao_mango)[need discuss]
4:Suport passing parameters between tasks (@qiaozhaowei)
5:Sort out and simplify the license process
6:Focus on fixing bugs base on the branch some new features can be released in 1.4.0_release base on branch 'dev'.(@Gabrywu)
7:Expect dolphinScheduler to provide SSO to facilitate seamless integration with existing platforms.((@yinyong))
8:SPI requirements are collected and developed. We need to have a discussion to find out the modules that currently need SPI most(@gaojun_2048 @lenboo @CalvinKirs)
9:Worker Poll slot(@yangyichao_mango)
10:state machine processor of worker and master optimize(@yangyichao_mango)
11:How to do project migration.
Including workflow definitions, resource files, and data sources.
The development environment and the production environment are two sets of environments.(@BoYiZhang)
12:Adding a batch stop on the process instance page(@xingchun_chen)
13:Many new functions have been added in the dev branch and some issues have been solved. It is recommended to merge them into the next version(@all)
You can list the digital subscripts of the functions you expect, and then reply directly to the email. We will determine the main work content for the next issue based on your opinions. We look forward to your suggestions. thanks
Best&amp;nbsp; wishes！
CalvinKirs