You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by zh...@apache.org on 2022/12/28 06:21:45 UTC
[dolphinscheduler] branch 3.0.4-prepare updated (bab80aaba2 -> e028e640af)
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a change to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
from bab80aaba2 [maven-release-plugin] prepare for next development iteration
new 1acfe78185 [Improvement][doc] correct error in CONTRIBUTING.md (#13012)
new 82ea78aa83 [Bug] wrong logic when master-server check data-quality result (#13065)
new d79dc31e31 [Improvement-13084][UI] Make download log button disabled when host is null (#13090)
new 750cf62641 Fix datasource jdbc url param doesn't work (#13098)
new c6b9afe878 [fix-13041] fix example checkFormula (#13110)
new e970ead651 Throws the correct exception for the updateResource interface (#13120)
new 19771e506f [Fix] Replaced dead links in the docs. (#13170)
new bc1cf25f4d Solve the deadlock problem caused by queuing (#13191)
new 31f8d6d517 [Fix][Doc]Update pseudo-cluster.md (#13207)
new f409749cb9 [Fix][Doc] Replace the links. (#13276)
new e028e640af [fix][dml] Also query workflow without tenant by given name (#13075)
The 11 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.dlc.json | 13 ++-
CONTRIBUTING.md | 102 +--------------------
README.md | 4 +-
README_zh_CN.md | 8 +-
deploy/README.md | 4 +-
docs/docs/en/contribute/architecture-design.md | 4 +-
.../en/contribute/development-environment-setup.md | 4 +-
docs/docs/en/contribute/join/review.md | 4 +-
docs/docs/en/contribute/release/release.md | 4 +-
docs/docs/en/faq.md | 21 +++--
docs/docs/en/guide/data-quality.md | 4 +-
docs/docs/en/guide/installation/kubernetes.md | 2 +-
docs/docs/en/guide/installation/pseudo-cluster.md | 28 +++---
docs/docs/en/guide/installation/standalone.md | 4 +-
docs/docs/en/guide/start/docker.md | 6 +-
docs/docs/en/guide/upgrade.md | 6 +-
docs/docs/zh/contribute/architecture-design.md | 2 +-
docs/docs/zh/contribute/join/review.md | 6 +-
docs/docs/zh/faq.md | 24 +++--
docs/docs/zh/guide/data-quality.md | 12 +--
docs/docs/zh/guide/installation/kubernetes.md | 2 +-
docs/docs/zh/guide/installation/pseudo-cluster.md | 18 ++--
docs/docs/zh/guide/installation/standalone.md | 16 ++--
docs/docs/zh/guide/start/docker.md | 2 +-
docs/docs/zh/guide/upgrade.md | 30 +++---
.../api/service/impl/ResourcesServiceImpl.java | 3 +-
.../api/service/DataSourceServiceTest.java | 39 +++++---
.../dao/mapper/ProcessDefinitionMapper.xml | 4 +-
.../datasource/AbstractDataSourceProcessor.java | 14 +++
.../clickhouse/ClickHouseDataSourceProcessor.java | 29 ++----
.../api/datasource/db2/Db2DataSourceProcessor.java | 30 ++----
.../datasource/hive/HiveDataSourceProcessor.java | 59 ++----------
.../datasource/mysql/MySQLDataSourceProcessor.java | 29 ++----
.../oracle/OracleDataSourceProcessor.java | 28 ++----
.../postgresql/PostgreSQLDataSourceProcessor.java | 34 +++----
.../presto/PrestoDataSourceProcessor.java | 22 +----
.../redshift/RedshiftDataSourceProcessor.java | 25 ++---
.../datasource/spark/SparkDataSourceProcessor.java | 22 +----
.../sqlserver/SQLServerDataSourceProcessor.java | 21 +----
.../api/provider/JDBCDataSourceProvider.java | 9 +-
.../AbstractDataSourceProcessorTest.java | 72 +++++++++++++++
.../datasource/db2/Db2DataSourceProcessorTest.java | 7 +-
.../oracle/OracleDataSourceProcessorTest.java | 9 +-
.../PostgreSQLDataSourceProcessorTest.java | 9 +-
.../presto/PrestoDataSourceProcessorTest.java | 3 +-
.../redshift/RedshiftDataSourceProcessorTest.java | 12 ++-
.../spark/SparkDataSourceProcessorTest.java | 3 +-
...Exception.java => StateEventHandleFailure.java} | 9 +-
.../server/master/event/StateEventHandler.java | 5 +-
.../event/TaskWaitTaskGroupStateHandler.java | 15 ++-
.../master/runner/WorkflowExecuteRunnable.java | 8 ++
.../server/utils/DataQualityResultOperator.java | 16 ++--
.../spi/datasource/BaseConnectionParam.java | 87 +-----------------
.../plugin/task/dq/DataQualityTaskTest.java | 10 +-
.../src/views/projects/task/instance/use-table.ts | 1 +
55 files changed, 396 insertions(+), 568 deletions(-)
copy dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/{StateEventHandleException.java => StateEventHandleFailure.java} (79%)
[dolphinscheduler] 09/11: [Fix][Doc]Update pseudo-cluster.md (#13207)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 31f8d6d5171e20ea054507a76b00678d74075811
Author: lifeng <53...@users.noreply.github.com>
AuthorDate: Fri Dec 16 19:58:43 2022 +0800
[Fix][Doc]Update pseudo-cluster.md (#13207)
* Update pseudo-cluster.md
* Update pseudo-cluster.md
(cherry picked from commit 84496c5f04f5db2ecef9ac51b4515b27b80a869c)
---
docs/docs/zh/guide/installation/pseudo-cluster.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/docs/zh/guide/installation/pseudo-cluster.md b/docs/docs/zh/guide/installation/pseudo-cluster.md
index 426ca72663..5d1012d762 100644
--- a/docs/docs/zh/guide/installation/pseudo-cluster.md
+++ b/docs/docs/zh/guide/installation/pseudo-cluster.md
@@ -99,7 +99,7 @@ deployUser="dolphinscheduler"
文件 `./bin/env/dolphinscheduler_env.sh` 描述了下列配置：
-* DolphinScheduler 的数据库配置，详细配置方法见[初始化数据库](#初始化数据库)
+* DolphinScheduler 的数据库配置，详细配置方法见[初始化数据库](初始化数据库)
* 一些任务类型外部依赖路径或库文件，如 `JAVA_HOME` 和 `SPARK_HOME`都是在这里定义的
* 注册中心`zookeeper`
* 服务端相关配置，比如缓存，时区设置等
@@ -141,7 +141,7 @@ export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:
## 初始化数据库
-请参考 [数据源配置](../howto/datasource-setting.md) `伪分布式/分布式安装初始化数据库` 创建并初始化数据库
+请参考 [数据源配置] `伪分布式/分布式安装初始化数据库` 创建并初始化数据库
## 启动 DolphinScheduler
[dolphinscheduler] 05/11: [fix-13041] fix example checkFormula (#13110)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit c6b9afe878a0546540f441329fa0e487557ef7a2
Author: longtb <67...@users.noreply.github.com>
AuthorDate: Sat Dec 10 12:23:44 2022 +0800
[fix-13041] fix example checkFormula (#13110)
Co-authored-by: zhangshunmin <zh...@kezaihui.com>
(cherry picked from commit 917df4f6ca9513f0d535f5c3b8ceb372a0ad9966)
---
docs/docs/en/guide/data-quality.md | 4 ++--
docs/docs/zh/guide/data-quality.md | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/docs/docs/en/guide/data-quality.md b/docs/docs/en/guide/data-quality.md
index ec308df3bb..0df3bbdcb7 100644
--- a/docs/docs/en/guide/data-quality.md
+++ b/docs/docs/en/guide/data-quality.md
@@ -33,8 +33,8 @@ data-quality.jar.name=dolphinscheduler-data-quality-dev-SNAPSHOT.jar
| CheckFormula | <ul><li>Expected-Actual</li><li>Actual-Expected</li><li>(Actual/Expected)x100%</li><li>(Expected-Actual)/Expected x100%</li></ul> |
| Operator | =, >, >=, <, <=, != |
| ExpectedValue | <ul><li>FixValue</li><li>DailyAvg</li><li>WeeklyAvg</li><li>MonthlyAvg</li><li>Last7DayAvg</li><li>Last30DayAvg</li><li>SrcTableTotalRows</li><li>TargetTableTotalRows</li></ul> |
-| Example |<ul><li>CheckFormula：Expected-Actual</li><li>Operator：></li><li>Threshold：0</li><li>ExpectedValue：FixValue=9</li></ul>
-
+| Example | <ul><li>CheckFormula：Actual-Expected</li><li>Operator：></li><li>Threshold：0</li><li>ExpectedValue：FixValue=9</li></ul> |
+
In the example, assuming that the actual value is 10, the operator is >, and the expected value is 9, then the result 10 -9 > 0 is true, which means that the row data in the empty column has exceeded the threshold, and the task is judged to fail.
# Task Operation Guide
diff --git a/docs/docs/zh/guide/data-quality.md b/docs/docs/zh/guide/data-quality.md
index 50e2c92f14..718efc2352 100644
--- a/docs/docs/zh/guide/data-quality.md
+++ b/docs/docs/zh/guide/data-quality.md
@@ -44,12 +44,12 @@ data-quality.jar.name=dolphinscheduler-data-quality-dev-SNAPSHOT.jar
- 目标表总行数
- 例子
- - 校验方式为：[Expected-Actual][期望值-实际值]
- - [操作符]：>
- - [阈值]：0
- - 期望值类型：固定值=9。
-
- 假设实际值为10，操作符为 >, 期望值为9，那么结果 10 -9 > 0 为真，那就意味列为空的行数据已经超过阈值，任务被判定为失败
+ - 校验方式为：[Actual-Expected][实际值-期望值]
+ - [操作符]：>
+ - [阈值]：0
+ - 期望值类型：固定值=9。
+
+ 假设实际值为10，操作符为 >, 期望值为9，那么结果 10 -9 > 0 为真，那就意味列为空的行数据已经超过阈值，任务被判定为失败
# 任务操作指南
## 单表检查之空值检查
[dolphinscheduler] 06/11: Throws the correct exception for the updateResource interface (#13120)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit e970ead651962b99fba4e1e4025d39135fd1856c
Author: J·Y <ji...@gmail.com>
AuthorDate: Fri Dec 9 16:17:20 2022 +0800
Throws the correct exception for the updateResource interface (#13120)
Co-authored-by: rqyin <rq...@easipass.com>
(cherry picked from commit f9cf9331a6d9399d512302dead39880fcd59d3e2)
---
.../apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
index 5cdac59d37..fb00f2595b 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
@@ -64,7 +64,6 @@ import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.multipart.MultipartFile;
import java.io.IOException;
-import java.rmi.ServerException;
import java.text.MessageFormat;
import java.util.ArrayList;
import java.util.Arrays;
@@ -1326,7 +1325,7 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
return org.apache.dolphinscheduler.api.utils.FileUtils.file2Resource(localFileName);
} catch (IOException e) {
logger.error("download resource error, the path is {}, and local filename is {}, the error message is {}", fileName, localFileName, e.getMessage());
- throw new ServerException("download the resource file failed ,it may be related to your storage");
+ throw new ServiceException("download the resource file failed ,it may be related to your storage");
}
[dolphinscheduler] 01/11: [Improvement][doc] correct error in CONTRIBUTING.md (#13012)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 1acfe781858282843f89744d6fd258f841f830af
Author: David <da...@gmail.com>
AuthorDate: Sat Dec 17 12:04:00 2022 +0800
[Improvement][doc] correct error in CONTRIBUTING.md (#13012)
(cherry picked from commit f233023b7422159a319bf021d14658b80aa52d2c)
---
CONTRIBUTING.md | 102 +-------------------------------------------------------
1 file changed, 1 insertion(+), 101 deletions(-)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 1ea7469b38..cbbf95c524 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,101 +1 @@
-# How To Contribute
-
-Start by forking the dolphinscheduler GitHub repository, make changes in a branch and then send a pull request.
-
-## Set up your dolphinscheduler GitHub Repository
-
-There are three branches in the remote repository currently:
-
- - `master` : normal delivery branch. After the stable version is released, the code for the stable version branch is merged into the master branch.
-
- - `dev` : daily development branch. The daily development branch, the newly submitted code can pull requests to this branch.
-
- - `x.x.x-release` : the stable release version.
-
-So, you should fork the `dev` branch.
-
-After forking the [dolphinscheduler upstream source repository](https://github.com/apache/dolphinscheduler/fork) to your personal repository, you can set your personal development environment.
-
-```sh
-cd <your work direcotry>
-git clone <your personal forked dolphinscheduler repo>
-cd dolphinscheduler
-```
-
-## Set git remote as `upstream`
-
-Add remote repository address, named upstream
-
-```sh
-git remote add upstream https://github.com/apache/dolphinscheduler.git
-```
-
-View repository:
-
-```sh
-git remote -v
-```
-
-There will be two repositories at this time: origin (your own warehouse) and upstream (remote repository)
-
-Get/update remote repository code (already the latest code, skip it).
-
-
-```sh
-git fetch upstream
-```
-
-Synchronize remote repository code to local repository
-
-```sh
-git checkout origin/dev
-git merge --no-ff upstream/dev
-```
-
-If remote branch has a new branch `dev-1.0`, you need to synchronize this branch to the local repository, then push to your own repository.
-
-```sh
-git checkout -b dev-1.0 upstream/dev-1.0
-git push --set-upstream origin dev-1.0
-```
-
-## Create your feature branch
-
-Before making code changes, make sure you create a separate branch for them.
-
-```sh
-git checkout -b <your-feature-branch> dev
-```
-
-## Commit changes
-
-After modifying the code locally, submit it to your own repository:
-
-```sh
-git commit -m 'information about your feature'
-```
-
-## Push to the branch
-
-Push your locally committed changes to the remote origin (your fork).
-
-```sh
-git push origin <your-feature-branch>
-```
-
-## Create a pull request
-
-After submitting changes to your remote repository, you should click on the new pull request On the following github page.
-
-<p align = "center">
-<img src = "http://geek.analysys.cn/static/upload/221/2019-04-02/90f3abbf-70ef-4334-b8d6-9014c9cf4c7f.png" width ="60%"/>
-</p>
-
-
-Select the modified local branch and the branch to merge past to create a pull request.
-
-<p align = "center">
-<img src = "http://geek.analysys.cn/static/upload/221/2019-04-02/fe7eecfe-2720-4736-951b-b3387cf1ae41.png" width ="60%"/>
-</p>
-
-Next, the administrator is responsible for **merging** to complete the pull request.
+Please refer to the contribution document [How to contribute](https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/contribute/join/contribute.html)
[dolphinscheduler] 11/11: [fix][dml] Also query workflow without tenant by given name (#13075)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit e028e640af7481cb22b5c33159acaf0b33d374ec
Author: Jay Chung <zh...@gmail.com>
AuthorDate: Wed Dec 28 14:14:29 2022 +0800
[fix][dml] Also query workflow without tenant by given name (#13075)
before this, once remove user tenant, all users workflow
can not be found anymore, this patch fix this problem, make
this entrypoint can query workflow without tenant to its
users
(cherry picked from commit 6627c38be572a4b0621d6f9b4109d6143b8df7e3)
---
.../apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml
index 7ac596ea3c..4bad9cae9f 100644
--- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml
+++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml
@@ -62,8 +62,8 @@
from t_ds_process_definition pd
JOIN t_ds_user u ON pd.user_id = u.id
JOIN t_ds_project p ON pd.project_code = p.code
- JOIN t_ds_tenant t ON t.id = u.tenant_id
- JOIN t_ds_queue q ON t.queue_id = q.id
+ LEFT JOIN t_ds_tenant t ON t.id = u.tenant_id
+ LEFT JOIN t_ds_queue q ON t.queue_id = q.id
WHERE p.code = #{projectCode}
and pd.name = #{processDefinitionName}
</select>
[dolphinscheduler] 07/11: [Fix] Replaced dead links in the docs. (#13170)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 19771e506ff8fbbb7706c7f2f640df2d3bf49256
Author: Amy0104 <am...@163.com>
AuthorDate: Tue Dec 13 14:36:53 2022 +0800
[Fix] Replaced dead links in the docs. (#13170)
* [Fix] Replaced dead links.
* [Improvement] Format markdowns.
(cherry picked from commit dcf02eab45055052452a9d7f7a2303278ac00058)
---
README_zh_CN.md | 8 +++---
deploy/README.md | 4 +--
docs/docs/en/contribute/join/review.md | 4 +--
docs/docs/en/faq.md | 19 ++++++++------
docs/docs/en/guide/installation/pseudo-cluster.md | 28 ++++++++++-----------
docs/docs/en/guide/installation/standalone.md | 4 +--
docs/docs/en/guide/start/docker.md | 6 ++---
docs/docs/en/guide/upgrade.md | 6 ++---
docs/docs/zh/contribute/join/review.md | 6 ++---
docs/docs/zh/faq.md | 20 +++++++++------
docs/docs/zh/guide/installation/pseudo-cluster.md | 2 +-
docs/docs/zh/guide/installation/standalone.md | 16 ++++++------
docs/docs/zh/guide/start/docker.md | 2 +-
docs/docs/zh/guide/upgrade.md | 30 +++++++++++------------
14 files changed, 83 insertions(+), 72 deletions(-)
diff --git a/README_zh_CN.md b/README_zh_CN.md
index b300f2b31e..ddf66202f4 100644
--- a/README_zh_CN.md
+++ b/README_zh_CN.md
@@ -53,11 +53,11 @@ DolphinScheduler的工作计划：<a href="https://github.com/apache/dolphinsche
## 快速试用 Docker
-请参考官方文档: [快速试用 Docker 部署](https://dolphinscheduler.apache.org/en-us/docs/3.0.2/user_doc/guide/start/docker.html)
+请参考官方文档: [快速试用 Docker 部署](https://dolphinscheduler.apache.org/#/zh-cn/docs/3.0.3/guide/start/docker)
## 快速试用 Kubernetes
-请参考官方文档: [快速试用 Kubernetes 部署](https://dolphinscheduler.apache.org/en-us/docs/3.0.2/user_doc/guide/installation/kubernetes.html)
+请参考官方文档: [快速试用 Kubernetes 部署](http://dolphinscheduler.apache.org/#/zh-cn/docs/3.0.3/guide/installation/kubernetes)
## 如何构建
@@ -80,11 +80,11 @@ Dolphin Scheduler使用了很多优秀的开源项目，比如google的guava、g
## 获得帮助
1. 提交 [issue](https://github.com/apache/dolphinscheduler/issues/new/choose)
-2. [加入slack群](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) 并在频道 `#troubleshooting` 中提问
+2. [加入slack群](https://s.apache.org/dolphinscheduler-slack) 并在频道 `#troubleshooting` 中提问
## 社区
-1. 通过[该申请链接](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw)加入slack channel
+1. 通过[该申请链接](https://s.apache.org/dolphinscheduler-slack)加入slack channel
2. 关注[Apache Dolphin Scheduler的Twitter账号](https://twitter.com/dolphinschedule)获取实时动态
## 版权
diff --git a/deploy/README.md b/deploy/README.md
index ad679e8921..80d8123ba9 100644
--- a/deploy/README.md
+++ b/deploy/README.md
@@ -1,4 +1,4 @@
# DolphinScheduler for Docker and Kubernetes
-* [Start Up DolphinScheduler with Docker](https://dolphinscheduler.apache.org/en-us/docs/3.0.2/user_doc/guide/start/docker.html)
-* [Start Up DolphinScheduler with Kubernetes](https://dolphinscheduler.apache.org/en-us/docs/3.0.2/user_doc/guide/installation/kubernetes.html)
+- [Start Up DolphinScheduler with Docker](https://dolphinscheduler.apache.org/#/en-us/docs/3.0.3/guide/start/docker)
+- [Start Up DolphinScheduler with Kubernetes](https://dolphinscheduler.apache.org/#/en-us/docs/3.0.3/guide/installation/kubernetes)
diff --git a/docs/docs/en/contribute/join/review.md b/docs/docs/en/contribute/join/review.md
index 40c8a23a7a..29a8b8465a 100644
--- a/docs/docs/en/contribute/join/review.md
+++ b/docs/docs/en/contribute/join/review.md
@@ -1,6 +1,6 @@
# Community Review
-Beside submit Issues and pull requests to the GitHub repository mentioned in [team](/us-en/community/community.html), another important way to
+Beside submit Issues and pull requests to the GitHub repository mentioned in [team](https://dolphinscheduler.apache.org/#/en-us/community), another important way to
contribute to DolphinScheduler is reviewing GitHub Issues or Pull Requests. You can not only know the latest new and
direction of the community, but also understand the good design in others during you reviewing. At the same time, you can
increase your exposure in the community and accumulate your honor.
@@ -12,7 +12,7 @@ in [GitHub Discussion][discussion-result-review-wanted].
> Note: It is only users mentioned in the [GitHub Discussion][discussion-result-review-wanted] can review Issues or Pull
> Requests, Community advocates **Anyone is encouraged to review Issues and Pull Requests**. Users in
> [GitHub Discussion][discussion-result-review-wanted] show their willing to review when we collect in the mail thread.
-> The advantage of this list is when the community has discussion, in addition to the mention Members in [team](/us-en/community/community.html),
+> The advantage of this list is when the community has discussion, in addition to the mention Members in [team](https://dolphinscheduler.apache.org/#/en-us/community),
> you can also find some help in [GitHub Discussion][discussion-result-review-wanted] people. If you want to join the
> [GitHub Discussion][discussion-result-review-wanted], please comment in that discussion and leave a module you are interested
> in, and the maintainer will add you to the list.
diff --git a/docs/docs/en/faq.md b/docs/docs/en/faq.md
index bbb612c4b5..3bb5b16efb 100644
--- a/docs/docs/en/faq.md
+++ b/docs/docs/en/faq.md
@@ -301,7 +301,8 @@ change into
---
## Q : how to add a worker server
-A: 1, Create deployment user and hosts mapping, please refer 1.3 part of [cluster deployment](https://dolphinscheduler.apache.org/en-us/docs/laster/user_doc/installation/cluster.html)
+
+A: 1, Create deployment user and hosts mapping, please refer 1.3 part of [cluster deployment](https://dolphinscheduler.apache.org/#/en-us/docs/3.0.3/user_doc/installation/cluster)
2, Configure hosts mapping and ssh access and modify directory permissions. please refer 1.4 part of [cluster deployment](https://dolphinscheduler.apache.org/en-us/docs/laster/user_doc/installation/cluster.html)
@@ -446,12 +447,16 @@ A: 1, cd dolphinscheduler-ui and delete node_modules directory
```
sudo rm -rf node_modules
```
- 2, install node-sass through npm.taobao.org
- ```
- sudo npm uninstall node-sass
- sudo npm i node-sass --sass_binary_site=https://npm.taobao.org/mirrors/node-sass/
- ```
- 3, if the 2nd step failure, please, [referer url](https://dolphinscheduler.apache.org/en-us/development/frontend-development.html)
+
+ 2, install node-sass through npm.taobao.org
+
+```
+sudo npm uninstall node-sass
+sudo npm i node-sass --sass_binary_site=https://npm.taobao.org/mirrors/node-sass/
+```
+
+3, if the 2nd step failure, please, [referer url](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/frontend-development.md)
+
```
sudo npm rebuild node-sass
```
diff --git a/docs/docs/en/guide/installation/pseudo-cluster.md b/docs/docs/en/guide/installation/pseudo-cluster.md
index 7efa52af1c..603c0ca62b 100644
--- a/docs/docs/en/guide/installation/pseudo-cluster.md
+++ b/docs/docs/en/guide/installation/pseudo-cluster.md
@@ -8,13 +8,13 @@ If you are a new hand and want to experience DolphinScheduler functions, we reco
Pseudo-cluster deployment of DolphinScheduler requires external software support:
-* JDK：download [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment.
-* Binary package: Download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download/download.html)
-* Database: [PostgreSQL](https://www.postgresql.org/download/) (8.2.15+) or [MySQL](https://dev.mysql.com/downloads/mysql/) (5.7+), you can choose one of the two, such as MySQL requires JDBC Driver 8.0.16
-* Registry Center: [ZooKeeper](https://zookeeper.apache.org/releases.html) (3.8.0+)，[download link][zookeeper]
-* Process tree analysis
- * `pstree` for macOS
- * `psmisc` for Fedora/Red/Hat/CentOS/Ubuntu/Debian
+- JDK：download [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment.
+- Binary package: Download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/#/en-us/download)
+- Database: [PostgreSQL](https://www.postgresql.org/download/) (8.2.15+) or [MySQL](https://dev.mysql.com/downloads/mysql/) (5.7+), you can choose one of the two, such as MySQL requires JDBC Driver 8.0.16
+- Registry Center: [ZooKeeper](https://zookeeper.apache.org/releases.html) (3.8.0+)，[download link][zookeeper]
+- Process tree analysis
+ - `pstree` for macOS
+ - `psmisc` for Fedora/Red/Hat/CentOS/Ubuntu/Debian
> **_Note:_** DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but if you need to run tasks that depend on them, you need to have the corresponding environment support.
@@ -42,8 +42,8 @@ chmod -R 755 apache-dolphinscheduler-*-bin
> **_NOTICE:_**
>
-> * Due to DolphinScheduler's multi-tenant task switch user using command `sudo -u {linux-user}`, the deployment user needs to have `sudo` privileges and be password-free. If novice learners don’t understand, you can ignore this point for now.
-> * If you find the line "Defaults requirett" in the `/etc/sudoers` file, please comment the content.
+> - Due to DolphinScheduler's multi-tenant task switch user using command `sudo -u {linux-user}`, the deployment user needs to have `sudo` privileges and be password-free. If novice learners don’t understand, you can ignore this point for now.
+> - If you find the line "Defaults requirett" in the `/etc/sudoers` file, please comment the content.
### Configure Machine SSH Password-Free Login
@@ -99,12 +99,12 @@ deployUser="dolphinscheduler"
### Modify `dolphinscheduler_env.sh`
-File `./bin/env/dolphinscheduler_env.sh` describes the following configurations:
+File `./bin/env/dolphinscheduler_env.sh` describes the following configurations:
-* Database configuration of DolphinScheduler, see [Initialize the Database](#initialize-the-database) for detailed instructions.
-* Some tasks which need external dependencies or libraries such as `JAVA_HOME` and `SPARK_HOME`.
-* Registry center `zookeeper`.
-* Server related configuration, such as cache type, timezone, etc.
+- Database configuration of DolphinScheduler, see [Initialize the Database](#initialize-the-database) for detailed instructions.
+- Some tasks which need external dependencies or libraries such as `JAVA_HOME` and `SPARK_HOME`.
+- Registry center `zookeeper`.
+- Server related configuration, such as cache type, timezone, etc.
You could ignore the task external dependencies if you do not use those tasks, but you have to change `JAVA_HOME`, registry center and database
related configurations based on your environment.
diff --git a/docs/docs/en/guide/installation/standalone.md b/docs/docs/en/guide/installation/standalone.md
index 27f6a32f5b..e5201c6438 100644
--- a/docs/docs/en/guide/installation/standalone.md
+++ b/docs/docs/en/guide/installation/standalone.md
@@ -11,8 +11,8 @@ If you want to deploy DolphinScheduler in production, we recommend you follow [c
## Preparation
-* JDK：download [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment.
-* Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download/download.html).
+- JDK：download [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment.
+- Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/#/en-us/download).
## Start DolphinScheduler Standalone Server
diff --git a/docs/docs/en/guide/start/docker.md b/docs/docs/en/guide/start/docker.md
index 7b5b7887eb..8172af5d8f 100644
--- a/docs/docs/en/guide/start/docker.md
+++ b/docs/docs/en/guide/start/docker.md
@@ -45,14 +45,14 @@ modify docker-compose's free memory up to 4 GB.
- Hyper-V mode: Click `Docker Desktop -> Settings -> Resources -> Memory` modified it
- WSL 2 mode: see [WSL 2 utility VM](https://docs.microsoft.com/zh-cn/windows/wsl/wsl-config#configure-global-options-with-wslconfig) for more detail.
-After complete the configuration, we can get the `docker-compose.yaml` file from [download page](/en-us/download/download.html)
+After complete the configuration, we can get the `docker-compose.yaml` file from [download page](https://dolphinscheduler.apache.org/#/en-us/download)
form its source package, and make sure you get the right version. After download the package, you can run the commands as below.
```shell
$ DOLPHINSCHEDULER_VERSION=3.0.3
$ tar -zxf apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src.tar.gz
# Going to docker-compose's location
-# For Mac or Linux users
+# For Mac or Linux users
$ cd apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src/deploy/docker
# For Windows users, you should run command `cd apache-dolphinscheduler-"${DOLPHINSCHEDULER_VERSION}"-src\deploy\docker`
@@ -81,7 +81,7 @@ $ docker run -d --name dolphinscheduler-tools \
-e SPRING_DATASOURCE_USERNAME="<USER>" \
-e SPRING_DATASOURCE_PASSWORD="<PASSWORD>" \
--net host \
- apache/dolphinscheduler-tools:"${DOLPHINSCHEDULER_VERSION}" tools/bin/upgrade-schema.sh
+ apache/dolphinscheduler-tools:"${DOLPHINSCHEDULER_VERSION}" tools/bin/upgrade-schema.sh
# Starting DolphinScheduler service
$ docker run -d --name dolphinscheduler-master \
-e DATABASE="postgresql" \
diff --git a/docs/docs/en/guide/upgrade.md b/docs/docs/en/guide/upgrade.md
index af44a7483c..3af991c13b 100644
--- a/docs/docs/en/guide/upgrade.md
+++ b/docs/docs/en/guide/upgrade.md
@@ -8,7 +8,7 @@ To prevent data loss by some miss-operation, it is recommended to back up data b
### Download the Latest Version Installation Package
-Download the latest binary distribute package from [download](/en-us/download/download.html) and then put it in the different
+Download the latest binary distribute package from [download](https://dolphinscheduler.apache.org/#/en-us/download) and then put it in the different
directory where current service running. And all below command is running in this directory.
## Upgrade
@@ -54,14 +54,14 @@ The architecture of worker group is different between version before version 1.3
#### How Can I Do When I Upgrade from 1.3.1 to version before 2.0.0
-* Check the backup database, search records in table `t_ds_worker_group` table and mainly focus on three columns: `id, name and IP`.
+- Check the backup database, search records in table `t_ds_worker_group` table and mainly focus on three columns: `id, name and IP`.
| id | name | ip_list |
| :--- | :---: | ---: |
| 1 | service1 | 192.168.xx.10 |
| 2 | service2 | 192.168.xx.11,192.168.xx.12 |
-* Modify worker related configuration in `bin/env/install_config.conf`.
+- Modify worker related configuration in `bin/env/install_config.conf`.
Assume bellow are the machine worker service to be deployed:
diff --git a/docs/docs/zh/contribute/join/review.md b/docs/docs/zh/contribute/join/review.md
index 6f86ff0937..d48ae791e4 100644
--- a/docs/docs/zh/contribute/join/review.md
+++ b/docs/docs/zh/contribute/join/review.md
@@ -1,6 +1,6 @@
# 参与社区 review
-贡献 DolphinScheduler 的方式，除了向 [团队](/us-en/community/community.html) 中提到的 GitHub 仓库提交 Issues 和 pull requests 外，另一非常重要的方式是
+贡献 DolphinScheduler 的方式，除了向 [团队](https://dolphinscheduler.apache.org/#/zh-cn/community) 中提到的 GitHub 仓库提交 Issues 和 pull requests 外，另一非常重要的方式是
review 社区的 Issues 或者 Pull Requests。通过别人 Issues 和 Pull Requests，你不仅能知道社区的最新进展和发展方向，还能了解别人代码的设
计思想，同时可以增加自己在社区的曝光、积累自己在社区的荣誉值。
@@ -8,8 +8,8 @@ review 社区的 Issues 或者 Pull Requests。通过别人 Issues 和 Pull Requ
以及 Pull Requests，详见 [邮件][mail-review-wanted]，并将其结果放到了 [GitHub Discussion][discussion-result-review-wanted] 中。
> 注意: 这里并不是说只有 [GitHub Discussion][discussion-result-review-wanted] 中提及的用户才可以协助 review Issue 或者 Pull Requests，
-> 请记住社区的主张是 **任何人都被鼓励去 review 社区的 Issues 和 Pull Requests**。只是那部分用户在邮件列表意见征集的时候，表达了愿意付
-> 出更多的时间，参与社区的 review。另一个好处是，当社区有不确定的问题的时，除了可以找 [团队](/us-en/community/community.html) 中对应的 Members 外，还可以找
+> 请记住社区的主张是 **任何人都被鼓励去 review 社区的 Issues 和 Pull Requests**。只是那部分用户在邮件列表意见征集的时候，表达了愿意付
+> 出更多的时间，参与社区的 review。另一个好处是，当社区有不确定的问题的时，除了可以找 [团队](https://dolphinscheduler.apache.org/#/zh-cn/community) 中对应的 Members 外，还可以找
> [GitHub Discussion][discussion-result-review-wanted] 中提及的人解答对应的问题。如果你要想要加入到 [GitHub Discussion][discussion-result-review-wanted]
> 中，请在该 discussion 中评论并留下你感兴趣的模块，维护者会将你加入到对应的名单中。
diff --git a/docs/docs/zh/faq.md b/docs/docs/zh/faq.md
index 1a8dfdb093..67b45302cd 100644
--- a/docs/docs/zh/faq.md
+++ b/docs/docs/zh/faq.md
@@ -284,7 +284,8 @@ A： 将 hive pom
---
## Q：如何增加一台工作服务器
-A： 1，参考官网[部署文档](https://dolphinscheduler.apache.org/zh-cn/docs/laster/user_doc/installation/cluster.html) 1.3 小节，创建部署用户和 hosts 映射
+
+A： 1，参考官网[部署文档](https://dolphinscheduler.apache.org/#/zh-cn/docs/3.0.3/installation/cluster) 1.3 小节，创建部署用户和 hosts 映射
2，参考官网[部署文档](https://dolphinscheduler.apache.org/zh-cn/docs/laster/user_doc/installation/cluster.html) 1.4 小节，配置 hosts 映射和 ssh 打通及修改目录权限.
1.4 小节的最后一步是在当前新增机器上执行的，即需要给部署目录部署用户的权限
@@ -427,12 +428,16 @@ A：1，cd dolphinscheduler-ui 然后删除 node_modules 目录
```
sudo rm -rf node_modules
```
- 2，通过 npm.taobao.org 下载 node-sass
- ```
- sudo npm uninstall node-sass
- sudo npm i node-sass --sass_binary_site=https://npm.taobao.org/mirrors/node-sass/
- ```
- 3，如果步骤 2 报错，请重新构建 node-saas [参考链接](https://dolphinscheduler.apache.org/en-us/development/frontend-development.html)
+
+ 2，通过 npm.taobao.org 下载 node-sass
+
+```
+sudo npm uninstall node-sass
+sudo npm i node-sass --sass_binary_site=https://npm.taobao.org/mirrors/node-sass/
+```
+
+3，如果步骤 2 报错，请重新构建 node-saas [参考链接](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/zh/contribute/frontend-development.md)
+
```
sudo npm rebuild node-sass
```
@@ -582,6 +587,7 @@ sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
---
## Q：Yarn多集群支持
+
A：将Worker节点分别部署至多个Yarn集群，步骤如下（例如AWS EMR）：
1. 将 Worker 节点部署至 EMR 集群的 Master 节点
diff --git a/docs/docs/zh/guide/installation/pseudo-cluster.md b/docs/docs/zh/guide/installation/pseudo-cluster.md
index 5cd4123542..426ca72663 100644
--- a/docs/docs/zh/guide/installation/pseudo-cluster.md
+++ b/docs/docs/zh/guide/installation/pseudo-cluster.md
@@ -9,7 +9,7 @@
伪分布式部署 DolphinScheduler 需要有外部软件的支持
* JDK：下载[JDK][jdk] (1.8+)，安装并配置 `JAVA_HOME` 环境变量，并将其下的 `bin` 目录追加到 `PATH` 环境变量中。如果你的环境中已存在，可以跳过这步。
-* 二进制包：在[下载页面](https://dolphinscheduler.apache.org/zh-cn/download/download.html)下载 DolphinScheduler 二进制包
+* 二进制包：在[下载页面](https://dolphinscheduler.apache.org/#/zh-cn/download)下载 DolphinScheduler 二进制包
* 数据库：[PostgreSQL](https://www.postgresql.org/download/) (8.2.15+) 或者 [MySQL](https://dev.mysql.com/downloads/mysql/) (5.7+)，两者任选其一即可，如 MySQL 则需要 JDBC Driver 8.0.16
* 注册中心：[ZooKeeper](https://zookeeper.apache.org/releases.html) (3.8.0+)，[下载地址][zookeeper]
* 进程树分析
diff --git a/docs/docs/zh/guide/installation/standalone.md b/docs/docs/zh/guide/installation/standalone.md
index 82af53e2ab..e334293682 100644
--- a/docs/docs/zh/guide/installation/standalone.md
+++ b/docs/docs/zh/guide/installation/standalone.md
@@ -1,22 +1,22 @@
-# Standalone极速体验版
+# Standalone 极速体验版
Standalone 仅适用于 DolphinScheduler 的快速体验.
-如果你是新手，想要体验 DolphinScheduler 的功能，推荐使用Standalone方式体检。如果你想体验更完整的功能，或者更大的任务量，推荐使用[伪集群部署](pseudo-cluster.md)。如果你是在生产中使用，推荐使用[集群部署](cluster.md)或者[kubernetes](kubernetes.md)
+如果你是新手，想要体验 DolphinScheduler 的功能，推荐使用 Standalone 方式体检。如果你想体验更完整的功能，或者更大的任务量，推荐使用[伪集群部署](pseudo-cluster.md)。如果你是在生产中使用，推荐使用[集群部署](cluster.md)或者[kubernetes](kubernetes.md)
-> **_注意:_** Standalone仅建议20个以下工作流使用，因为其采用内存式的H2 Database, Zookeeper Testing Server，任务过多可能导致不稳定，并且如果重启或者停止standalone-server会导致内存中数据库里的数据清空。
-> 如果您要连接外部数据库，比如mysql或者postgresql，请看[配置数据库](#配置数据库)
+> **_注意:_** Standalone 仅建议 20 个以下工作流使用，因为其采用内存式的 H2 Database, Zookeeper Testing Server，任务过多可能导致不稳定，并且如果重启或者停止 standalone-server 会导致内存中数据库里的数据清空。
+> 如果您要连接外部数据库，比如 mysql 或者 postgresql，请看[配置数据库](#配置数据库)
## 前置准备工作
-* JDK：下载[JDK][jdk] (1.8+)，安装并配置 `JAVA_HOME` 环境变量，并将其下的 `bin` 目录追加到 `PATH` 环境变量中。如果你的环境中已存在，可以跳过这步。
-* 二进制包：在[下载页面](https://dolphinscheduler.apache.org/zh-cn/download/download.html)下载 DolphinScheduler 二进制包
+- JDK：下载[JDK][jdk] (1.8+)，安装并配置 `JAVA_HOME` 环境变量，并将其下的 `bin` 目录追加到 `PATH` 环境变量中。如果你的环境中已存在，可以跳过这步。
+- 二进制包：在[下载页面](https://dolphinscheduler.apache.org/#/zh-cn/download)下载 DolphinScheduler 二进制包
## 启动 DolphinScheduler Standalone Server
### 解压并启动 DolphinScheduler
-二进制压缩包中有 standalone 启动的脚本，解压后即可快速启动。切换到有sudo权限的用户，运行脚本
+二进制压缩包中有 standalone 启动的脚本，解压后即可快速启动。切换到有 sudo 权限的用户，运行脚本
```shell
# 解压并运行 Standalone Server
@@ -28,7 +28,7 @@ bash ./bin/dolphinscheduler-daemon.sh start standalone-server
### 登录 DolphinScheduler
-浏览器访问地址 http://localhost:12345/dolphinscheduler/ui 即可登录系统UI。默认的用户名和密码是 **admin/dolphinscheduler123**
+浏览器访问地址 http://localhost:12345/dolphinscheduler/ui 即可登录系统 UI。默认的用户名和密码是 **admin/dolphinscheduler123**
## 启停服务
diff --git a/docs/docs/zh/guide/start/docker.md b/docs/docs/zh/guide/start/docker.md
index 7d9097f7ce..df3020d2f7 100644
--- a/docs/docs/zh/guide/start/docker.md
+++ b/docs/docs/zh/guide/start/docker.md
@@ -39,7 +39,7 @@ $ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:2
- Hyper-V 模式：点击 `Docker Desktop -> Settings -> Resources -> Memory` 调整内存大小
- WSL 2 模式 模式：参考 [WSL 2 utility VM](https://docs.microsoft.com/zh-cn/windows/wsl/wsl-config#configure-global-options-with-wslconfig) 调整内存大小
-配置完成后我们需要获取 `docker-compose.yaml` 文件，通过[下载页面](/zh-cn/download/download.html)下载对应版本源码包可能是最快的方法，
+配置完成后我们需要获取 `docker-compose.yaml` 文件，通过[下载页面](https://dolphinscheduler.apache.org/#/zh-cn/download)下载对应版本源码包可能是最快的方法，
源码包对应的值为 "Total Source Code"。当下载完源码后就可以运行命令进行部署了。
```shell
diff --git a/docs/docs/zh/guide/upgrade.md b/docs/docs/zh/guide/upgrade.md
index a8483cce69..917fa8f3f9 100644
--- a/docs/docs/zh/guide/upgrade.md
+++ b/docs/docs/zh/guide/upgrade.md
@@ -8,7 +8,7 @@
### 下载新版本的安装包
-在[下载](/zh-cn/download/download.html)页面下载最新版本的二进制安装包，并将二进制包放到与当前 dolphinscheduler 服务不一样的路径中，以下升级操作都需要在新版本的目录进行。
+在[下载](https://dolphinscheduler.apache.org/#/zh-cn/download)页面下载最新版本的二进制安装包，并将二进制包放到与当前 dolphinscheduler 服务不一样的路径中，以下升级操作都需要在新版本的目录进行。
## 升级步骤
@@ -46,41 +46,41 @@ jar 包 并添加到 `./tools/libs` 目录下，修改 `./bin/env/dolphinschedul
### worker 分组的区别（以 1.3.1 版本为界）
-创建worker分组在1.3.1版本之前，与 1.3.1之后到 2.0.0 之间的版本有不同的设计：
+创建 worker 分组在 1.3.1 版本之前，与 1.3.1 之后到 2.0.0 之间的版本有不同的设计：
-- worker分组在1.3.1版本之前是通过UI界面创建
-- worker分组在1.3.1 到 2.0.0之前的版本是修改 worker 配置指定
+- worker 分组在 1.3.1 版本之前是通过 UI 界面创建
+- worker 分组在 1.3.1 到 2.0.0 之前的版本是修改 worker 配置指定
#### 面对这种区别我应该怎么升级
-1.3.1之前的版本升级1.3.2时如何设置worker分组与之前一致
+1.3.1 之前的版本升级 1.3.2 时如何设置 worker 分组与之前一致
-* 查询已备份的数据库，查看 `t_ds_worker_group` 表记录，重点看下id、name和ip_list三个字段
+- 查询已备份的数据库，查看 `t_ds_worker_group` 表记录，重点看下 id、name 和 ip_list 三个字段
| id | name | ip_list |
| :--- | :---: | ---: |
| 1 | service1 | 192.168.xx.10 |
| 2 | service2 | 192.168.xx.11,192.168.xx.12 |
-* 修改 `bin/env/install_config.conf` 中的 workers 参数
+- 修改 `bin/env/install_config.conf` 中的 workers 参数
-假设以下为要部署的worker主机名和ip的对应关系
+假设以下为要部署的 worker 主机名和 ip 的对应关系
| 主机名 | ip |
-| :--- | :---: |
-| ds1 | 192.168.xx.10 |
-| ds2 | 192.168.xx.11 |
-| ds3 | 192.168.xx.12 |
+| :--- | :---: |
+| ds1 | 192.168.xx.10 |
+| ds2 | 192.168.xx.11 |
+| ds3 | 192.168.xx.12 |
-那么为了保持与之前版本worker分组一致，则需要把workers参数改为如下
+那么为了保持与之前版本 worker 分组一致，则需要把 workers 参数改为如下
```sh
# worker服务部署在哪台机器上,并指定此worker属于哪一个worker组
workers="ds1:service1,ds2:service2,ds3:service2"
```
-#### 1.3.2及以后的版本对 worker 分组功能进行增强
+#### 1.3.2 及以后的版本对 worker 分组功能进行增强
-1.3.1 以及之前的版本worker不能同时属于多个worker分组，1.3.2及之后，2.0.0之前的版本是可以支持的，所以可以使用如下配置对一台worker配置多个分组
+1.3.1 以及之前的版本 worker 不能同时属于多个 worker 分组，1.3.2 及之后，2.0.0 之前的版本是可以支持的，所以可以使用如下配置对一台 worker 配置多个分组
```sh
workers="ds1:service1,ds1:service2"
[dolphinscheduler] 02/11: [Bug] wrong logic when master-server check data-quality result (#13065)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 82ea78aa83be958268b734e3f61ddf1c5db83d09
Author: Kevin.Shin <ma...@163.com>
AuthorDate: Thu Dec 1 13:58:50 2022 +0800
[Bug] wrong logic when master-server check data-quality result (#13065)
Co-authored-by: shenk-b <sh...@glodon.com>
(cherry picked from commit 12a6138d3343a489069ddf98c48f61957801b167)
---
.../server/utils/DataQualityResultOperator.java | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/utils/DataQualityResultOperator.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/utils/DataQualityResultOperator.java
index d172228b36..e94d6f1422 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/utils/DataQualityResultOperator.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/utils/DataQualityResultOperator.java
@@ -95,7 +95,7 @@ public class DataQualityResultOperator {
private void checkDqExecuteResult(TaskEvent taskResponseEvent,
DqExecuteResult dqExecuteResult,
ProcessInstance processInstance) {
- if (isFailure(dqExecuteResult)) {
+ if (isFailed(dqExecuteResult)) {
DqFailureStrategy dqFailureStrategy = DqFailureStrategy.of(dqExecuteResult.getFailureStrategy());
if (dqFailureStrategy != null) {
dqExecuteResult.setState(DqTaskState.FAILURE.getCode());
@@ -124,7 +124,7 @@ public class DataQualityResultOperator {
* @param dqExecuteResult
* @return
*/
- private boolean isFailure(DqExecuteResult dqExecuteResult) {
+ private boolean isFailed(DqExecuteResult dqExecuteResult) {
CheckType checkType = CheckType.of(dqExecuteResult.getCheckType());
double statisticsValue = dqExecuteResult.getStatisticsValue();
@@ -133,36 +133,36 @@ public class DataQualityResultOperator {
OperatorType operatorType = OperatorType.of(dqExecuteResult.getOperator());
- boolean isFailure = false;
+ boolean isFailed = false;
if (operatorType != null) {
double srcValue = 0;
switch (checkType) {
case COMPARISON_MINUS_STATISTICS:
srcValue = comparisonValue - statisticsValue;
- isFailure = getCompareResult(operatorType,srcValue,threshold);
+ isFailed = !getCompareResult(operatorType, srcValue, threshold);
break;
case STATISTICS_MINUS_COMPARISON:
srcValue = statisticsValue - comparisonValue;
- isFailure = getCompareResult(operatorType,srcValue,threshold);
+ isFailed = !getCompareResult(operatorType, srcValue, threshold);
break;
case STATISTICS_COMPARISON_PERCENTAGE:
if (comparisonValue > 0) {
srcValue = statisticsValue / comparisonValue * 100;
}
- isFailure = getCompareResult(operatorType,srcValue,threshold);
+ isFailed = !getCompareResult(operatorType, srcValue, threshold);
break;
case STATISTICS_COMPARISON_DIFFERENCE_COMPARISON_PERCENTAGE:
if (comparisonValue > 0) {
srcValue = Math.abs(comparisonValue - statisticsValue) / comparisonValue * 100;
}
- isFailure = getCompareResult(operatorType,srcValue,threshold);
+ isFailed = !getCompareResult(operatorType, srcValue, threshold);
break;
default:
break;
}
}
- return isFailure;
+ return isFailed;
}
private void sendDqTaskResultAlert(DqExecuteResult dqExecuteResult, ProcessInstance processInstance) {
[dolphinscheduler] 08/11: Solve the deadlock problem caused by queuing (#13191)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit bc1cf25f4d705efd5fe500a9b27691b1b83fb771
Author: sssqhai <35...@users.noreply.github.com>
AuthorDate: Fri Dec 16 19:55:02 2022 +0800
Solve the deadlock problem caused by queuing (#13191)
* Solve the deadlock problem caused by queuing
* Solve the deadlock problem caused by queuing
* Solve the deadlock problem caused by queuing
* Solve the deadlock problem caused by queuing,move the event to the tail by throwing a exception
Co-authored-by: wfs <wa...@cdqcp.cpm>
(cherry picked from commit 7a0a2c2a46a0224d0c9fb1f649c1f901d11e814b)
---
...ateHandler.java => StateEventHandleFailure.java} | 21 +++++++++------------
.../server/master/event/StateEventHandler.java | 5 +++--
.../master/event/TaskWaitTaskGroupStateHandler.java | 15 +++++++++++++--
.../master/runner/WorkflowExecuteRunnable.java | 8 ++++++++
4 files changed, 33 insertions(+), 16 deletions(-)
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/TaskWaitTaskGroupStateHandler.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/StateEventHandleFailure.java
similarity index 59%
copy from dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/TaskWaitTaskGroupStateHandler.java
copy to dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/StateEventHandleFailure.java
index 9a3c59a949..5e757c7858 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/TaskWaitTaskGroupStateHandler.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/StateEventHandleFailure.java
@@ -17,20 +17,17 @@
package org.apache.dolphinscheduler.server.master.event;
-import org.apache.dolphinscheduler.common.enums.StateEventType;
-import org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable;
-
-import com.google.auto.service.AutoService;
+/**
+ * This exception represent the exception can be recovered, when we get this exception,
+ * we will move the event to the fail of the queue.
+ */
+public class StateEventHandleFailure extends Exception {
-@AutoService(StateEventHandler.class)
-public class TaskWaitTaskGroupStateHandler implements StateEventHandler {
- @Override
- public boolean handleStateEvent(WorkflowExecuteRunnable workflowExecuteRunnable, StateEvent stateEvent) {
- return workflowExecuteRunnable.checkForceStartAndWakeUp(stateEvent);
+ public StateEventHandleFailure(String message) {
+ super(message);
}
- @Override
- public StateEventType getEventType() {
- return StateEventType.WAIT_TASK_GROUP;
+ public StateEventHandleFailure(String message, Throwable throwable) {
+ super(message, throwable);
}
}
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/StateEventHandler.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/StateEventHandler.java
index 00808b2e29..377ea71f62 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/StateEventHandler.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/StateEventHandler.java
@@ -28,9 +28,10 @@ public interface StateEventHandler {
* @param stateEvent given state event.
* @throws StateEventHandleException this exception means it can be recovered.
* @throws StateEventHandleError this exception means it cannot be recovered, so the event need to drop.
+ * @throws StateEventHandleException this means it can be recovered.
*/
- boolean handleStateEvent(WorkflowExecuteRunnable workflowExecuteRunnable, StateEvent stateEvent)
- throws StateEventHandleException, StateEventHandleError;
+ boolean handleStateEvent(WorkflowExecuteRunnable workflowExecuteRunnable,
+ StateEvent stateEvent) throws StateEventHandleException, StateEventHandleError, StateEventHandleFailure;
StateEventType getEventType();
}
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/TaskWaitTaskGroupStateHandler.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/TaskWaitTaskGroupStateHandler.java
index 9a3c59a949..bb0c7b8b70 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/TaskWaitTaskGroupStateHandler.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/event/TaskWaitTaskGroupStateHandler.java
@@ -20,13 +20,24 @@ package org.apache.dolphinscheduler.server.master.event;
import org.apache.dolphinscheduler.common.enums.StateEventType;
import org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
import com.google.auto.service.AutoService;
@AutoService(StateEventHandler.class)
public class TaskWaitTaskGroupStateHandler implements StateEventHandler {
+
+ private static final Logger logger = LoggerFactory.getLogger(TaskWaitTaskGroupStateHandler.class);
+
@Override
- public boolean handleStateEvent(WorkflowExecuteRunnable workflowExecuteRunnable, StateEvent stateEvent) {
- return workflowExecuteRunnable.checkForceStartAndWakeUp(stateEvent);
+ public boolean handleStateEvent(WorkflowExecuteRunnable workflowExecuteRunnable,
+ StateEvent stateEvent) throws StateEventHandleFailure {
+ logger.info("Handle task instance wait task group event, taskInstanceId: {}", stateEvent.getTaskInstanceId());
+ if (!workflowExecuteRunnable.checkForceStartAndWakeUp(stateEvent)) {
+ throw new StateEventHandleFailure("Task state event handle failed due to robing taskGroup resource failed");
+ }
+ return true;
}
@Override
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java
index 5c0fec018d..fda109e1f4 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java
@@ -70,6 +70,7 @@ import org.apache.dolphinscheduler.server.master.dispatch.executor.NettyExecutor
import org.apache.dolphinscheduler.server.master.event.StateEvent;
import org.apache.dolphinscheduler.server.master.event.StateEventHandleError;
import org.apache.dolphinscheduler.server.master.event.StateEventHandleException;
+import org.apache.dolphinscheduler.server.master.event.StateEventHandleFailure;
import org.apache.dolphinscheduler.server.master.event.StateEventHandler;
import org.apache.dolphinscheduler.server.master.event.StateEventHandlerManager;
import org.apache.dolphinscheduler.server.master.metrics.TaskMetrics;
@@ -279,6 +280,13 @@ public class WorkflowExecuteRunnable implements Callable<WorkflowSubmitStatue> {
stateEvent,
stateEventHandleException);
ThreadUtils.sleep(Constants.SLEEP_TIME_MILLIS);
+ } catch (StateEventHandleFailure stateEventHandleFailure) {
+ logger.error("State event handle failed, will move event to the tail: {}",
+ stateEvent,
+ stateEventHandleFailure);
+ this.stateEvents.remove(stateEvent);
+ this.stateEvents.offer(stateEvent);
+ ThreadUtils.sleep(Constants.SLEEP_TIME_MILLIS);
} catch (Exception e) {
// we catch the exception here, since if the state event handle failed, the state event will still keep in the stateEvents queue.
logger.error("State event handle error, get a unknown exception, will retry this event: {}",
[dolphinscheduler] 04/11: Fix datasource jdbc url param doesn't work (#13098)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 750cf62641483c28d388c7052370e9670abe3927
Author: Wenjun Ruan <we...@apache.org>
AuthorDate: Mon Dec 12 11:46:40 2022 +0800
Fix datasource jdbc url param doesn't work (#13098)
(cherry picked from commit ab96a3d0)
---
.../api/service/DataSourceServiceTest.java | 39 ++++++----
.../datasource/AbstractDataSourceProcessor.java | 14 ++++
.../clickhouse/ClickHouseDataSourceProcessor.java | 29 +++-----
.../api/datasource/db2/Db2DataSourceProcessor.java | 30 +++-----
.../datasource/hive/HiveDataSourceProcessor.java | 59 +++------------
.../datasource/mysql/MySQLDataSourceProcessor.java | 29 +++-----
.../oracle/OracleDataSourceProcessor.java | 28 ++-----
.../postgresql/PostgreSQLDataSourceProcessor.java | 34 ++++-----
.../presto/PrestoDataSourceProcessor.java | 22 ++----
.../redshift/RedshiftDataSourceProcessor.java | 25 ++-----
.../datasource/spark/SparkDataSourceProcessor.java | 22 ++----
.../sqlserver/SQLServerDataSourceProcessor.java | 21 ++----
.../api/provider/JDBCDataSourceProvider.java | 9 ++-
.../AbstractDataSourceProcessorTest.java | 72 ++++++++++++++++++
.../datasource/db2/Db2DataSourceProcessorTest.java | 7 +-
.../oracle/OracleDataSourceProcessorTest.java | 9 ++-
.../PostgreSQLDataSourceProcessorTest.java | 9 ++-
.../presto/PrestoDataSourceProcessorTest.java | 3 +-
.../redshift/RedshiftDataSourceProcessorTest.java | 12 ++-
.../spark/SparkDataSourceProcessorTest.java | 3 +-
.../spi/datasource/BaseConnectionParam.java | 87 +---------------------
.../plugin/task/dq/DataQualityTaskTest.java | 10 +--
22 files changed, 236 insertions(+), 337 deletions(-)
diff --git a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/DataSourceServiceTest.java b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/DataSourceServiceTest.java
index a3b07fa2d0..4f3e00f36a 100644
--- a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/DataSourceServiceTest.java
+++ b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/DataSourceServiceTest.java
@@ -381,21 +381,30 @@ public class DataSourceServiceTest {
@Test
public void buildParameterWithDecodePassword() {
- PropertyUtils.setValue(Constants.DATASOURCE_ENCRYPTION_ENABLE, "true");
- Map<String, String> other = new HashMap<>();
- other.put("autoDeserialize", "yes");
- other.put("allowUrlInLocalInfile", "true");
- MySQLDataSourceParamDTO mysqlDatasourceParamDTO = new MySQLDataSourceParamDTO();
- mysqlDatasourceParamDTO.setHost("192.168.9.1");
- mysqlDatasourceParamDTO.setPort(1521);
- mysqlDatasourceParamDTO.setDatabase("im");
- mysqlDatasourceParamDTO.setUserName("test");
- mysqlDatasourceParamDTO.setPassword("123456");
- mysqlDatasourceParamDTO.setOther(other);
- ConnectionParam connectionParam = DataSourceUtils.buildConnectionParams(mysqlDatasourceParamDTO);
- String expected = "{\"user\":\"test\",\"password\":\"IUAjJCVeJipNVEl6TkRVMg==\",\"address\":\"jdbc:mysql://192.168.9.1:1521\",\"database\":\"im\",\"jdbcUrl\":\"jdbc:mysql://192.168.9.1:1521/"
- + "im\",\"driverClassName\":\"com.mysql.cj.jdbc.Driver\",\"validationQuery\":\"select 1\",\"props\":{\"autoDeserialize\":\"yes\",\"allowUrlInLocalInfile\":\"true\"}}";
- Assert.assertEquals(expected, JSONUtils.toJsonString(connectionParam));
+ try (MockedStatic<PropertyUtils> mockedStaticPropertyUtils = Mockito.mockStatic(PropertyUtils.class)) {
+ mockedStaticPropertyUtils
+ .when(() -> PropertyUtils.getBoolean(DataSourceConstants.DATASOURCE_ENCRYPTION_ENABLE, false))
+ .thenReturn(true);
+ Map<String, String> other = new HashMap<>();
+ other.put("autoDeserialize", "yes");
+ other.put("allowUrlInLocalInfile", "true");
+ other.put("useSSL", "true");
+ MySQLDataSourceParamDTO mysqlDatasourceParamDTO = new MySQLDataSourceParamDTO();
+ mysqlDatasourceParamDTO.setHost("192.168.9.1");
+ mysqlDatasourceParamDTO.setPort(1521);
+ mysqlDatasourceParamDTO.setDatabase("im");
+ mysqlDatasourceParamDTO.setUserName("test");
+ mysqlDatasourceParamDTO.setPassword("123456");
+ mysqlDatasourceParamDTO.setOther(other);
+ ConnectionParam connectionParam = DataSourceUtils.buildConnectionParams(mysqlDatasourceParamDTO);
+ String expected =
+ "{\"user\":\"test\",\"password\":\"bnVsbE1USXpORFUy\",\"address\":\"jdbc:mysql://192.168.9.1:1521\","
+ +
+ "\"database\":\"im\",\"jdbcUrl\":\"jdbc:mysql://192.168.9.1:1521/im\",\"driverClassName\":\"com.mysql.cj.jdbc.Driver\","
+ +
+ "\"validationQuery\":\"select 1\",\"other\":{\"autoDeserialize\":\"yes\",\"allowUrlInLocalInfile\":\"true\",\"useSSL\":\"true\"}}";
+ Assertions.assertEquals(expected, JSONUtils.toJsonString(connectionParam));
+ }
PropertyUtils.setValue(Constants.DATASOURCE_ENCRYPTION_ENABLE, "false");
mysqlDatasourceParamDTO = new MySQLDataSourceParamDTO();
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/AbstractDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/AbstractDataSourceProcessor.java
index 04c619d955..2227062bda 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/AbstractDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/AbstractDataSourceProcessor.java
@@ -17,17 +17,23 @@
package org.apache.dolphinscheduler.plugin.datasource.api.datasource;
+import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.plugin.datasource.api.utils.PasswordUtils;
import org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam;
import org.apache.dolphinscheduler.spi.datasource.ConnectionParam;
import org.apache.dolphinscheduler.spi.enums.DbType;
import org.apache.commons.collections4.MapUtils;
+import org.apache.commons.lang3.StringUtils;
import java.text.MessageFormat;
+import java.util.Collections;
import java.util.Map;
import java.util.regex.Pattern;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.google.common.collect.Sets;
+
public abstract class AbstractDataSourceProcessor implements DataSourceProcessor {
private static final Pattern IPV4_PATTERN = Pattern.compile("^[a-zA-Z0-9\\_\\-\\.\\,]+$");
@@ -82,6 +88,14 @@ public abstract class AbstractDataSourceProcessor implements DataSourceProcessor
}
}
+ protected Map<String, String> transformOtherParamToMap(String other) {
+ if (StringUtils.isBlank(other)) {
+ return Collections.emptyMap();
+ }
+ return JSONUtils.parseObject(other, new TypeReference<Map<String, String>>() {
+ });
+ }
+
@Override
public String getDatasourceUniqueId(ConnectionParam connectionParam, DbType dbType) {
BaseConnectionParam baseConnectionParam = (BaseConnectionParam) connectionParam;
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/clickhouse/ClickHouseDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/clickhouse/ClickHouseDataSourceProcessor.java
index ea71d25a8d..44fed51109 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/clickhouse/ClickHouseDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/clickhouse/ClickHouseDataSourceProcessor.java
@@ -31,7 +31,8 @@ import org.apache.commons.collections4.MapUtils;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.util.LinkedHashMap;
+import java.util.ArrayList;
+import java.util.List;
import java.util.Map;
public class ClickHouseDataSourceProcessor extends AbstractDataSourceProcessor {
@@ -43,7 +44,7 @@ public class ClickHouseDataSourceProcessor extends AbstractDataSourceProcessor {
ClickHouseDataSourceParamDTO clickHouseDatasourceParamDTO = new ClickHouseDataSourceParamDTO();
clickHouseDatasourceParamDTO.setDatabase(connectionParams.getDatabase());
clickHouseDatasourceParamDTO.setUserName(connectionParams.getUser());
- clickHouseDatasourceParamDTO.setOther(parseOther(connectionParams.getOther()));
+ clickHouseDatasourceParamDTO.setOther(connectionParams.getOther());
String[] hostSeperator = connectionParams.getAddress().split(Constants.DOUBLE_SLASH);
String[] hostPortArray = hostSeperator[hostSeperator.length - 1].split(Constants.COMMA);
@@ -67,8 +68,7 @@ public class ClickHouseDataSourceProcessor extends AbstractDataSourceProcessor {
clickhouseConnectionParam.setPassword(PasswordUtils.encodePassword(clickHouseParam.getPassword()));
clickhouseConnectionParam.setDriverClassName(getDatasourceDriver());
clickhouseConnectionParam.setValidationQuery(getValidationQuery());
- clickhouseConnectionParam.setOther(transformOther(clickHouseParam.getOther()));
- clickhouseConnectionParam.setProps(clickHouseParam.getOther());
+ clickhouseConnectionParam.setOther(clickHouseParam.getOther());
return clickhouseConnectionParam;
}
@@ -91,8 +91,8 @@ public class ClickHouseDataSourceProcessor extends AbstractDataSourceProcessor {
public String getJdbcUrl(ConnectionParam connectionParam) {
ClickHouseConnectionParam clickhouseConnectionParam = (ClickHouseConnectionParam) connectionParam;
String jdbcUrl = clickhouseConnectionParam.getJdbcUrl();
- if (!StringUtils.isEmpty(clickhouseConnectionParam.getOther())) {
- jdbcUrl = String.format("%s?%s", jdbcUrl, clickhouseConnectionParam.getOther());
+ if (MapUtils.isNotEmpty(clickhouseConnectionParam.getOther())) {
+ jdbcUrl = String.format("%s?%s", jdbcUrl, transformOther(clickhouseConnectionParam.getOther()));
}
return jdbcUrl;
}
@@ -114,20 +114,9 @@ public class ClickHouseDataSourceProcessor extends AbstractDataSourceProcessor {
if (MapUtils.isEmpty(otherMap)) {
return null;
}
- StringBuilder stringBuilder = new StringBuilder();
- otherMap.forEach((key, value) -> stringBuilder.append(String.format("%s=%s%s", key, value, "&")));
- return stringBuilder.toString();
+ List<String> otherList = new ArrayList<>();
+ otherMap.forEach((key, value) -> otherList.add(String.format("%s=%s", key, value)));
+ return String.join("&", otherList);
}
- private Map<String, String> parseOther(String other) {
- if (other == null) {
- return null;
- }
- Map<String, String> otherMap = new LinkedHashMap<>();
- String[] configs = other.split("&");
- for (String config : configs) {
- otherMap.put(config.split("=")[0], config.split("=")[1]);
- }
- return otherMap;
- }
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/db2/Db2DataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/db2/Db2DataSourceProcessor.java
index 7e0548c71f..94356dcd56 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/db2/Db2DataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/db2/Db2DataSourceProcessor.java
@@ -32,7 +32,8 @@ import org.apache.commons.collections4.MapUtils;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.util.LinkedHashMap;
+import java.util.ArrayList;
+import java.util.List;
import java.util.Map;
public class Db2DataSourceProcessor extends AbstractDataSourceProcessor {
@@ -43,7 +44,7 @@ public class Db2DataSourceProcessor extends AbstractDataSourceProcessor {
Db2DataSourceParamDTO db2DatasourceParamDTO = new Db2DataSourceParamDTO();
db2DatasourceParamDTO.setDatabase(connectionParams.getDatabase());
- db2DatasourceParamDTO.setOther(parseOther(connectionParams.getOther()));
+ db2DatasourceParamDTO.setOther(connectionParams.getOther());
db2DatasourceParamDTO.setUserName(db2DatasourceParamDTO.getUserName());
String[] hostSeperator = connectionParams.getAddress().split(Constants.DOUBLE_SLASH);
@@ -68,8 +69,7 @@ public class Db2DataSourceProcessor extends AbstractDataSourceProcessor {
db2ConnectionParam.setPassword(PasswordUtils.encodePassword(db2Param.getPassword()));
db2ConnectionParam.setDriverClassName(getDatasourceDriver());
db2ConnectionParam.setValidationQuery(getValidationQuery());
- db2ConnectionParam.setOther(transformOther(db2Param.getOther()));
- db2ConnectionParam.setProps(db2Param.getOther());
+ db2ConnectionParam.setOther(db2Param.getOther());
return db2ConnectionParam;
}
@@ -87,8 +87,9 @@ public class Db2DataSourceProcessor extends AbstractDataSourceProcessor {
@Override
public String getJdbcUrl(ConnectionParam connectionParam) {
Db2ConnectionParam db2ConnectionParam = (Db2ConnectionParam) connectionParam;
- if (!StringUtils.isEmpty(db2ConnectionParam.getOther())) {
- return String.format("%s;%s", db2ConnectionParam.getJdbcUrl(), db2ConnectionParam.getOther());
+ if (MapUtils.isNotEmpty(db2ConnectionParam.getOther())) {
+ return String.format("%s;%s", db2ConnectionParam.getJdbcUrl(),
+ transformOther(db2ConnectionParam.getOther()));
}
return db2ConnectionParam.getJdbcUrl();
}
@@ -115,20 +116,9 @@ public class Db2DataSourceProcessor extends AbstractDataSourceProcessor {
if (MapUtils.isEmpty(otherMap)) {
return null;
}
- StringBuilder stringBuilder = new StringBuilder();
- otherMap.forEach((key, value) -> stringBuilder.append(String.format("%s=%s%s", key, value, ";")));
- stringBuilder.deleteCharAt(stringBuilder.length() - 1);
- return stringBuilder.toString();
+ List<String> otherList = new ArrayList<>();
+ otherMap.forEach((key, value) -> otherList.add(String.format("%s=%s", key, value)));
+ return String.join(";", otherList);
}
- private Map<String, String> parseOther(String other) {
- if (other == null) {
- return null;
- }
- Map<String, String> otherMap = new LinkedHashMap<>();
- for (String config : other.split("&")) {
- otherMap.put(config.split("=")[0], config.split("=")[1]);
- }
- return otherMap;
- }
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/hive/HiveDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/hive/HiveDataSourceProcessor.java
index 898ceb39aa..9362f89c23 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/hive/HiveDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/hive/HiveDataSourceProcessor.java
@@ -34,7 +34,8 @@ import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.util.LinkedHashMap;
+import java.util.ArrayList;
+import java.util.List;
import java.util.Map;
public class HiveDataSourceProcessor extends AbstractDataSourceProcessor {
@@ -46,7 +47,7 @@ public class HiveDataSourceProcessor extends AbstractDataSourceProcessor {
hiveDataSourceParamDTO.setDatabase(hiveConnectionParam.getDatabase());
hiveDataSourceParamDTO.setUserName(hiveConnectionParam.getUser());
- hiveDataSourceParamDTO.setOther(parseOther(hiveConnectionParam.getOther()));
+ hiveDataSourceParamDTO.setOther(hiveConnectionParam.getOther());
hiveDataSourceParamDTO.setLoginUserKeytabUsername(hiveConnectionParam.getLoginUserKeytabUsername());
hiveDataSourceParamDTO.setLoginUserKeytabPath(hiveConnectionParam.getLoginUserKeytabPath());
hiveDataSourceParamDTO.setJavaSecurityKrb5Conf(hiveConnectionParam.getJavaSecurityKrb5Conf());
@@ -90,8 +91,7 @@ public class HiveDataSourceProcessor extends AbstractDataSourceProcessor {
hiveConnectionParam.setLoginUserKeytabPath(hiveParam.getLoginUserKeytabPath());
hiveConnectionParam.setLoginUserKeytabUsername(hiveParam.getLoginUserKeytabUsername());
}
- hiveConnectionParam.setOther(transformOther(hiveParam.getOther()));
- hiveConnectionParam.setProps(hiveParam.getOther());
+ hiveConnectionParam.setOther(hiveParam.getOther());
return hiveConnectionParam;
}
@@ -114,11 +114,10 @@ public class HiveDataSourceProcessor extends AbstractDataSourceProcessor {
public String getJdbcUrl(ConnectionParam connectionParam) {
HiveConnectionParam hiveConnectionParam = (HiveConnectionParam) connectionParam;
String jdbcUrl = hiveConnectionParam.getJdbcUrl();
- String otherParams = filterOther(hiveConnectionParam.getOther());
- if (StringUtils.isNotEmpty(otherParams) && !"?".equals(otherParams.substring(0, 1))) {
- jdbcUrl += ";";
+ if (MapUtils.isNotEmpty(hiveConnectionParam.getOther())) {
+ return jdbcUrl + "?" + transformOther(hiveConnectionParam.getOther());
}
- return jdbcUrl + otherParams;
+ return jdbcUrl;
}
@Override
@@ -140,47 +139,9 @@ public class HiveDataSourceProcessor extends AbstractDataSourceProcessor {
if (MapUtils.isEmpty(otherMap)) {
return null;
}
- StringBuilder stringBuilder = new StringBuilder();
- otherMap.forEach((key, value) -> stringBuilder.append(String.format("%s=%s;", key, value)));
- return stringBuilder.toString();
+ List<String> otherList = new ArrayList<>();
+ otherMap.forEach((key, value) -> otherList.add(String.format("%s=%s", key, value)));
+ return String.join(";", otherList);
}
- private String filterOther(String otherParams) {
- if (StringUtils.isBlank(otherParams)) {
- return "";
- }
-
- StringBuilder hiveConfListSb = new StringBuilder();
- hiveConfListSb.append("?");
- StringBuilder sessionVarListSb = new StringBuilder();
-
- String[] otherArray = otherParams.split(";", -1);
-
- for (String conf : otherArray) {
- sessionVarListSb.append(conf).append(";");
- }
-
- // remove the last ";"
- if (sessionVarListSb.length() > 0) {
- sessionVarListSb.deleteCharAt(sessionVarListSb.length() - 1);
- }
-
- if (hiveConfListSb.length() > 0) {
- hiveConfListSb.deleteCharAt(hiveConfListSb.length() - 1);
- }
-
- return sessionVarListSb.toString() + hiveConfListSb.toString();
- }
-
- private Map<String, String> parseOther(String other) {
- if (other == null) {
- return null;
- }
- Map<String, String> otherMap = new LinkedHashMap<>();
- String[] configs = other.split(";");
- for (String config : configs) {
- otherMap.put(config.split("=")[0], config.split("=")[1]);
- }
- return otherMap;
- }
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/mysql/MySQLDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/mysql/MySQLDataSourceProcessor.java
index a0c5347a7c..fb6f29ed77 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/mysql/MySQLDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/mysql/MySQLDataSourceProcessor.java
@@ -32,8 +32,9 @@ import org.apache.commons.collections4.MapUtils;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
+import java.util.ArrayList;
import java.util.HashMap;
-import java.util.LinkedHashMap;
+import java.util.List;
import java.util.Map;
import org.slf4j.Logger;
@@ -60,7 +61,7 @@ public class MySQLDataSourceProcessor extends AbstractDataSourceProcessor {
mysqlDatasourceParamDTO.setUserName(connectionParams.getUser());
mysqlDatasourceParamDTO.setDatabase(connectionParams.getDatabase());
- mysqlDatasourceParamDTO.setOther(parseOther(connectionParams.getOther()));
+ mysqlDatasourceParamDTO.setOther(connectionParams.getOther());
String address = connectionParams.getAddress();
String[] hostSeperator = address.split(Constants.DOUBLE_SLASH);
@@ -85,8 +86,7 @@ public class MySQLDataSourceProcessor extends AbstractDataSourceProcessor {
mysqlConnectionParam.setPassword(PasswordUtils.encodePassword(mysqlDatasourceParam.getPassword()));
mysqlConnectionParam.setDriverClassName(getDatasourceDriver());
mysqlConnectionParam.setValidationQuery(getValidationQuery());
- mysqlConnectionParam.setOther(transformOther(mysqlDatasourceParam.getOther()));
- mysqlConnectionParam.setProps(mysqlDatasourceParam.getOther());
+ mysqlConnectionParam.setOther(mysqlDatasourceParam.getOther());
return mysqlConnectionParam;
}
@@ -110,8 +110,8 @@ public class MySQLDataSourceProcessor extends AbstractDataSourceProcessor {
public String getJdbcUrl(ConnectionParam connectionParam) {
MySQLConnectionParam mysqlConnectionParam = (MySQLConnectionParam) connectionParam;
String jdbcUrl = mysqlConnectionParam.getJdbcUrl();
- if (!StringUtils.isEmpty(mysqlConnectionParam.getOther())) {
- return String.format("%s?%s&%s", jdbcUrl, mysqlConnectionParam.getOther(), APPEND_PARAMS);
+ if (MapUtils.isNotEmpty(mysqlConnectionParam.getOther())) {
+ return String.format("%s?%s&%s", jdbcUrl, transformOther(mysqlConnectionParam.getOther()), APPEND_PARAMS);
}
return String.format("%s?%s", jdbcUrl, APPEND_PARAMS);
}
@@ -152,9 +152,9 @@ public class MySQLDataSourceProcessor extends AbstractDataSourceProcessor {
if (MapUtils.isEmpty(otherMap)) {
return null;
}
- StringBuilder stringBuilder = new StringBuilder();
- otherMap.forEach((key, value) -> stringBuilder.append(String.format("%s=%s&", key, value)));
- return stringBuilder.toString();
+ List<String> otherList = new ArrayList<>();
+ otherMap.forEach((key, value) -> otherList.add(String.format("%s=%s", key, value)));
+ return String.join("&", otherList);
}
private static boolean checkKeyIsLegitimate(String key) {
@@ -164,15 +164,4 @@ public class MySQLDataSourceProcessor extends AbstractDataSourceProcessor {
&& !key.contains(ALLOW_URL_IN_LOCAL_IN_FILE_NAME);
}
- private Map<String, String> parseOther(String other) {
- if (StringUtils.isEmpty(other)) {
- return null;
- }
- Map<String, String> otherMap = new LinkedHashMap<>();
- for (String config : other.split("&")) {
- otherMap.put(config.split("=")[0], config.split("=")[1]);
- }
- return otherMap;
- }
-
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/oracle/OracleDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/oracle/OracleDataSourceProcessor.java
index 44c4ab8a46..9a3ad3ace2 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/oracle/OracleDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/oracle/OracleDataSourceProcessor.java
@@ -34,7 +34,6 @@ import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.ArrayList;
-import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -47,7 +46,7 @@ public class OracleDataSourceProcessor extends AbstractDataSourceProcessor {
oracleDatasourceParamDTO.setDatabase(connectionParams.getDatabase());
oracleDatasourceParamDTO.setUserName(connectionParams.getUser());
- oracleDatasourceParamDTO.setOther(parseOther(connectionParams.getOther()));
+ oracleDatasourceParamDTO.setOther(connectionParams.getOther());
String hostSeperator = Constants.DOUBLE_SLASH;
if (DbConnectType.ORACLE_SID.equals(connectionParams.connectType)) {
@@ -85,8 +84,7 @@ public class OracleDataSourceProcessor extends AbstractDataSourceProcessor {
oracleConnectionParam.setConnectType(oracleParam.getConnectType());
oracleConnectionParam.setDriverClassName(getDatasourceDriver());
oracleConnectionParam.setValidationQuery(getValidationQuery());
- oracleConnectionParam.setOther(transformOther(oracleParam.getOther()));
- oracleConnectionParam.setProps(oracleParam.getOther());
+ oracleConnectionParam.setOther(oracleParam.getOther());
return oracleConnectionParam;
}
@@ -109,8 +107,9 @@ public class OracleDataSourceProcessor extends AbstractDataSourceProcessor {
@Override
public String getJdbcUrl(ConnectionParam connectionParam) {
OracleConnectionParam oracleConnectionParam = (OracleConnectionParam) connectionParam;
- if (!StringUtils.isEmpty(oracleConnectionParam.getOther())) {
- return String.format("%s?%s", oracleConnectionParam.getJdbcUrl(), oracleConnectionParam.getOther());
+ if (MapUtils.isNotEmpty(oracleConnectionParam.getOther())) {
+ return String.format("%s?%s", oracleConnectionParam.getJdbcUrl(),
+ transformOther(oracleConnectionParam.getOther()));
}
return oracleConnectionParam.getJdbcUrl();
}
@@ -132,20 +131,9 @@ public class OracleDataSourceProcessor extends AbstractDataSourceProcessor {
if (MapUtils.isEmpty(otherMap)) {
return null;
}
- List<String> list = new ArrayList<>();
- otherMap.forEach((key, value) -> list.add(String.format("%s=%s", key, value)));
- return String.join("&", list);
+ List<String> otherList = new ArrayList<>();
+ otherMap.forEach((key, value) -> otherList.add(String.format("%s=%s", key, value)));
+ return String.join("&", otherList);
}
- private Map<String, String> parseOther(String other) {
- if (StringUtils.isEmpty(other)) {
- return null;
- }
- Map<String, String> otherMap = new LinkedHashMap<>();
- String[] configs = other.split("&");
- for (String config : configs) {
- otherMap.put(config.split("=")[0], config.split("=")[1]);
- }
- return otherMap;
- }
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/postgresql/PostgreSQLDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/postgresql/PostgreSQLDataSourceProcessor.java
index f14302f1e7..038ada68f1 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/postgresql/PostgreSQLDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/postgresql/PostgreSQLDataSourceProcessor.java
@@ -32,9 +32,14 @@ import org.apache.commons.lang.StringUtils;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.util.LinkedHashMap;
+import java.util.ArrayList;
+import java.util.List;
import java.util.Map;
+import com.google.auto.service.AutoService;
+import com.google.common.base.Strings;
+
+@AutoService(DataSourceProcessor.class)
public class PostgreSQLDataSourceProcessor extends AbstractDataSourceProcessor {
@Override
@@ -43,7 +48,7 @@ public class PostgreSQLDataSourceProcessor extends AbstractDataSourceProcessor {
PostgreSQLDataSourceParamDTO postgreSqlDatasourceParamDTO = new PostgreSQLDataSourceParamDTO();
postgreSqlDatasourceParamDTO.setDatabase(connectionParams.getDatabase());
postgreSqlDatasourceParamDTO.setUserName(connectionParams.getUser());
- postgreSqlDatasourceParamDTO.setOther(parseOther(connectionParams.getOther()));
+ postgreSqlDatasourceParamDTO.setOther(connectionParams.getOther());
String address = connectionParams.getAddress();
String[] hostSeperator = address.split(Constants.DOUBLE_SLASH);
@@ -68,8 +73,7 @@ public class PostgreSQLDataSourceProcessor extends AbstractDataSourceProcessor {
postgreSqlConnectionParam.setPassword(PasswordUtils.encodePassword(postgreSqlParam.getPassword()));
postgreSqlConnectionParam.setDriverClassName(getDatasourceDriver());
postgreSqlConnectionParam.setValidationQuery(getValidationQuery());
- postgreSqlConnectionParam.setOther(transformOther(postgreSqlParam.getOther()));
- postgreSqlConnectionParam.setProps(postgreSqlParam.getOther());
+ postgreSqlConnectionParam.setOther(postgreSqlParam.getOther());
return postgreSqlConnectionParam;
}
@@ -92,8 +96,9 @@ public class PostgreSQLDataSourceProcessor extends AbstractDataSourceProcessor {
@Override
public String getJdbcUrl(ConnectionParam connectionParam) {
PostgreSQLConnectionParam postgreSqlConnectionParam = (PostgreSQLConnectionParam) connectionParam;
- if (!StringUtils.isEmpty(postgreSqlConnectionParam.getOther())) {
- return String.format("%s?%s", postgreSqlConnectionParam.getJdbcUrl(), postgreSqlConnectionParam.getOther());
+ if (MapUtils.isNotEmpty(postgreSqlConnectionParam.getOther())) {
+ return String.format("%s?%s", postgreSqlConnectionParam.getJdbcUrl(),
+ transformOther(postgreSqlConnectionParam.getOther()));
}
return postgreSqlConnectionParam.getJdbcUrl();
}
@@ -115,20 +120,9 @@ public class PostgreSQLDataSourceProcessor extends AbstractDataSourceProcessor {
if (MapUtils.isEmpty(otherMap)) {
return null;
}
- StringBuilder stringBuilder = new StringBuilder();
- otherMap.forEach((key, value) -> stringBuilder.append(String.format("%s=%s&", key, value)));
- return stringBuilder.toString();
+ List<String> otherList = new ArrayList<>();
+ otherMap.forEach((key, value) -> otherList.add(String.format("%s=%s", key, value)));
+ return String.join("&", otherList);
}
- private Map<String, String> parseOther(String other) {
- if (StringUtils.isEmpty(other)) {
- return null;
- }
- Map<String, String> otherMap = new LinkedHashMap<>();
- for (String config : other.split("&")) {
- String[] split = config.split("=");
- otherMap.put(split[0], split[1]);
- }
- return otherMap;
- }
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/presto/PrestoDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/presto/PrestoDataSourceProcessor.java
index 8fe1a7231c..fb1e6f3189 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/presto/PrestoDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/presto/PrestoDataSourceProcessor.java
@@ -33,7 +33,6 @@ import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.ArrayList;
-import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -51,7 +50,7 @@ public class PrestoDataSourceProcessor extends AbstractDataSourceProcessor {
prestoDatasourceParamDTO.setHost(hostPortArray[0].split(Constants.COLON)[0]);
prestoDatasourceParamDTO.setDatabase(connectionParams.getDatabase());
prestoDatasourceParamDTO.setUserName(connectionParams.getUser());
- prestoDatasourceParamDTO.setOther(parseOther(connectionParams.getOther()));
+ prestoDatasourceParamDTO.setOther(connectionParams.getOther());
return prestoDatasourceParamDTO;
}
@@ -65,13 +64,12 @@ public class PrestoDataSourceProcessor extends AbstractDataSourceProcessor {
PrestoConnectionParam prestoConnectionParam = new PrestoConnectionParam();
prestoConnectionParam.setUser(prestoParam.getUserName());
prestoConnectionParam.setPassword(PasswordUtils.encodePassword(prestoParam.getPassword()));
- prestoConnectionParam.setOther(transformOther(prestoParam.getOther()));
+ prestoConnectionParam.setOther(prestoParam.getOther());
prestoConnectionParam.setAddress(address);
prestoConnectionParam.setJdbcUrl(jdbcUrl);
prestoConnectionParam.setDatabase(prestoParam.getDatabase());
prestoConnectionParam.setDriverClassName(getDatasourceDriver());
prestoConnectionParam.setValidationQuery(getValidationQuery());
- prestoConnectionParam.setProps(prestoParam.getOther());
return prestoConnectionParam;
}
@@ -94,8 +92,9 @@ public class PrestoDataSourceProcessor extends AbstractDataSourceProcessor {
@Override
public String getJdbcUrl(ConnectionParam connectionParam) {
PrestoConnectionParam prestoConnectionParam = (PrestoConnectionParam) connectionParam;
- if (!StringUtils.isEmpty(prestoConnectionParam.getOther())) {
- return String.format("%s?%s", prestoConnectionParam.getJdbcUrl(), prestoConnectionParam.getOther());
+ if (MapUtils.isNotEmpty(prestoConnectionParam.getOther())) {
+ return String.format("%s?%s", prestoConnectionParam.getJdbcUrl(),
+ transformOther(prestoConnectionParam.getOther()));
}
return prestoConnectionParam.getJdbcUrl();
}
@@ -122,15 +121,4 @@ public class PrestoDataSourceProcessor extends AbstractDataSourceProcessor {
return null;
}
- private Map<String, String> parseOther(String other) {
- if (StringUtils.isEmpty(other)) {
- return null;
- }
- Map<String, String> otherMap = new LinkedHashMap<>();
- String[] configs = other.split("&");
- for (String config : configs) {
- otherMap.put(config.split("=")[0], config.split("=")[1]);
- }
- return otherMap;
- }
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessor.java
index 5a9c521fce..0ae5cf2476 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessor.java
@@ -33,7 +33,6 @@ import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.ArrayList;
-import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -53,7 +52,7 @@ public class RedshiftDataSourceProcessor extends AbstractDataSourceProcessor {
redshiftDatasourceParamDTO.setHost(hostPortArray[0].split(Constants.COLON)[0]);
redshiftDatasourceParamDTO.setDatabase(connectionParams.getDatabase());
redshiftDatasourceParamDTO.setUserName(connectionParams.getUser());
- redshiftDatasourceParamDTO.setOther(parseOther(connectionParams.getOther()));
+ redshiftDatasourceParamDTO.setOther(connectionParams.getOther());
return redshiftDatasourceParamDTO;
}
@@ -68,13 +67,12 @@ public class RedshiftDataSourceProcessor extends AbstractDataSourceProcessor {
redshiftConnectionParam = new RedshiftConnectionParam();
redshiftConnectionParam.setUser(redshiftParam.getUserName());
redshiftConnectionParam.setPassword(PasswordUtils.encodePassword(redshiftParam.getPassword()));
- redshiftConnectionParam.setOther(transformOther(redshiftParam.getOther()));
+ redshiftConnectionParam.setOther(redshiftParam.getOther());
redshiftConnectionParam.setAddress(address);
redshiftConnectionParam.setJdbcUrl(jdbcUrl);
redshiftConnectionParam.setDatabase(redshiftParam.getDatabase());
redshiftConnectionParam.setDriverClassName(getDatasourceDriver());
redshiftConnectionParam.setValidationQuery(getValidationQuery());
- redshiftConnectionParam.setProps(redshiftParam.getOther());
return redshiftConnectionParam;
}
@@ -96,10 +94,10 @@ public class RedshiftDataSourceProcessor extends AbstractDataSourceProcessor {
@Override
public String getJdbcUrl(ConnectionParam connectionParam) {
- RedshiftConnectionParam
- redshiftConnectionParam = (RedshiftConnectionParam) connectionParam;
- if (!StringUtils.isEmpty(redshiftConnectionParam.getOther())) {
- return String.format("%s?%s", redshiftConnectionParam.getJdbcUrl(), redshiftConnectionParam.getOther());
+ RedshiftConnectionParam redshiftConnectionParam = (RedshiftConnectionParam) connectionParam;
+ if (MapUtils.isNotEmpty(redshiftConnectionParam.getOther())) {
+ return String.format("%s?%s", redshiftConnectionParam.getJdbcUrl(),
+ transformOther(redshiftConnectionParam.getOther()));
}
return redshiftConnectionParam.getJdbcUrl();
}
@@ -126,15 +124,4 @@ public class RedshiftDataSourceProcessor extends AbstractDataSourceProcessor {
return null;
}
- private Map<String, String> parseOther(String other) {
- Map<String, String> otherMap = new LinkedHashMap<>();
- if (StringUtils.isEmpty(other)) {
- return otherMap;
- }
- String[] configs = other.split(Constants.SEMICOLON);
- for (String config : configs) {
- otherMap.put(config.split(Constants.EQUAL_SIGN)[0], config.split(Constants.EQUAL_SIGN)[1]);
- }
- return otherMap;
- }
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/spark/SparkDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/spark/SparkDataSourceProcessor.java
index 69fcd4ff24..ef0078ff20 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/spark/SparkDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/spark/SparkDataSourceProcessor.java
@@ -35,7 +35,6 @@ import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Arrays;
-import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
@@ -49,7 +48,7 @@ public class SparkDataSourceProcessor extends AbstractDataSourceProcessor {
SparkDataSourceParamDTO sparkDatasourceParamDTO = new SparkDataSourceParamDTO();
sparkDatasourceParamDTO.setDatabase(connectionParams.getDatabase());
sparkDatasourceParamDTO.setUserName(connectionParams.getUser());
- sparkDatasourceParamDTO.setOther(parseOther(connectionParams.getOther()));
+ sparkDatasourceParamDTO.setOther(connectionParams.getOther());
sparkDatasourceParamDTO.setJavaSecurityKrb5Conf(connectionParams.getJavaSecurityKrb5Conf());
sparkDatasourceParamDTO.setLoginUserKeytabPath(connectionParams.getLoginUserKeytabPath());
sparkDatasourceParamDTO.setLoginUserKeytabUsername(connectionParams.getLoginUserKeytabUsername());
@@ -81,13 +80,12 @@ public class SparkDataSourceProcessor extends AbstractDataSourceProcessor {
SparkConnectionParam sparkConnectionParam = new SparkConnectionParam();
sparkConnectionParam.setPassword(PasswordUtils.encodePassword(sparkDatasourceParam.getPassword()));
sparkConnectionParam.setUser(sparkDatasourceParam.getUserName());
- sparkConnectionParam.setOther(transformOther(sparkDatasourceParam.getOther()));
+ sparkConnectionParam.setOther(sparkDatasourceParam.getOther());
sparkConnectionParam.setDatabase(sparkDatasourceParam.getDatabase());
sparkConnectionParam.setAddress(address.toString());
sparkConnectionParam.setJdbcUrl(jdbcUrl);
sparkConnectionParam.setDriverClassName(getDatasourceDriver());
sparkConnectionParam.setValidationQuery(getValidationQuery());
- sparkConnectionParam.setProps(sparkDatasourceParam.getOther());
if (CommonUtils.getKerberosStartupState()) {
sparkConnectionParam.setPrincipal(sparkDatasourceParam.getPrincipal());
@@ -117,8 +115,9 @@ public class SparkDataSourceProcessor extends AbstractDataSourceProcessor {
@Override
public String getJdbcUrl(ConnectionParam connectionParam) {
SparkConnectionParam sparkConnectionParam = (SparkConnectionParam) connectionParam;
- if (!StringUtils.isEmpty(sparkConnectionParam.getOther())) {
- return String.format("%s;%s", sparkConnectionParam.getJdbcUrl(), sparkConnectionParam.getOther());
+ if (MapUtils.isNotEmpty(sparkConnectionParam.getOther())) {
+ return String.format("%s;%s", sparkConnectionParam.getJdbcUrl(),
+ transformOther(sparkConnectionParam.getOther()));
}
return sparkConnectionParam.getJdbcUrl();
}
@@ -147,15 +146,4 @@ public class SparkDataSourceProcessor extends AbstractDataSourceProcessor {
return String.join(";", stringBuilder);
}
- private Map<String, String> parseOther(String other) {
- if (StringUtils.isEmpty(other)) {
- return null;
- }
- Map<String, String> otherMap = new LinkedHashMap<>();
- String[] configs = other.split(";");
- for (String config : configs) {
- otherMap.put(config.split("=")[0], config.split("=")[1]);
- }
- return otherMap;
- }
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/sqlserver/SQLServerDataSourceProcessor.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/sqlserver/SQLServerDataSourceProcessor.java
index 44855d1fb6..678e006099 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/sqlserver/SQLServerDataSourceProcessor.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/sqlserver/SQLServerDataSourceProcessor.java
@@ -32,7 +32,6 @@ import org.apache.commons.lang.StringUtils;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.util.LinkedHashMap;
import java.util.Map;
public class SQLServerDataSourceProcessor extends AbstractDataSourceProcessor {
@@ -46,7 +45,7 @@ public class SQLServerDataSourceProcessor extends AbstractDataSourceProcessor {
SQLServerDataSourceParamDTO sqlServerDatasourceParamDTO = new SQLServerDataSourceParamDTO();
sqlServerDatasourceParamDTO.setDatabase(connectionParams.getDatabase());
sqlServerDatasourceParamDTO.setUserName(connectionParams.getUser());
- sqlServerDatasourceParamDTO.setOther(parseOther(connectionParams.getOther()));
+ sqlServerDatasourceParamDTO.setOther(connectionParams.getOther());
sqlServerDatasourceParamDTO.setPort(Integer.parseInt(hostPortArray[0].split(Constants.COLON)[1]));
sqlServerDatasourceParamDTO.setHost(hostPortArray[0].split(Constants.COLON)[0]);
return sqlServerDatasourceParamDTO;
@@ -62,12 +61,11 @@ public class SQLServerDataSourceProcessor extends AbstractDataSourceProcessor {
sqlServerConnectionParam.setAddress(address);
sqlServerConnectionParam.setDatabase(sqlServerParam.getDatabase());
sqlServerConnectionParam.setJdbcUrl(jdbcUrl);
- sqlServerConnectionParam.setOther(transformOther(sqlServerParam.getOther()));
+ sqlServerConnectionParam.setOther(sqlServerParam.getOther());
sqlServerConnectionParam.setUser(sqlServerParam.getUserName());
sqlServerConnectionParam.setPassword(PasswordUtils.encodePassword(sqlServerParam.getPassword()));
sqlServerConnectionParam.setDriverClassName(getDatasourceDriver());
sqlServerConnectionParam.setValidationQuery(getValidationQuery());
- sqlServerConnectionParam.setProps(sqlServerParam.getOther());
return sqlServerConnectionParam;
}
@@ -90,8 +88,9 @@ public class SQLServerDataSourceProcessor extends AbstractDataSourceProcessor {
public String getJdbcUrl(ConnectionParam connectionParam) {
SQLServerConnectionParam sqlServerConnectionParam = (SQLServerConnectionParam) connectionParam;
- if (!StringUtils.isEmpty(sqlServerConnectionParam.getOther())) {
- return String.format("%s;%s", sqlServerConnectionParam.getJdbcUrl(), sqlServerConnectionParam.getOther());
+ if (MapUtils.isNotEmpty(sqlServerConnectionParam.getOther())) {
+ return String.format("%s;%s", sqlServerConnectionParam.getJdbcUrl(),
+ transformOther(sqlServerConnectionParam.getOther()));
}
return sqlServerConnectionParam.getJdbcUrl();
}
@@ -118,14 +117,4 @@ public class SQLServerDataSourceProcessor extends AbstractDataSourceProcessor {
return stringBuilder.toString();
}
- private Map<String, String> parseOther(String other) {
- if (StringUtils.isEmpty(other)) {
- return null;
- }
- Map<String, String> otherMap = new LinkedHashMap<>();
- for (String config : other.split(";")) {
- otherMap.put(config.split("=")[0], config.split("=")[1]);
- }
- return otherMap;
- }
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/provider/JDBCDataSourceProvider.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/provider/JDBCDataSourceProvider.java
index ef5db82cef..5b4ce6008d 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/provider/JDBCDataSourceProvider.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/provider/JDBCDataSourceProvider.java
@@ -24,6 +24,7 @@ import org.apache.dolphinscheduler.spi.enums.DbType;
import org.apache.dolphinscheduler.spi.utils.Constants;
import org.apache.dolphinscheduler.spi.utils.PropertyUtils;
import org.apache.dolphinscheduler.spi.utils.StringUtils;
+import org.apache.commons.collections4.MapUtils;
import java.sql.Driver;
@@ -56,8 +57,8 @@ public class JDBCDataSourceProvider {
dataSource.setMaximumPoolSize(PropertyUtils.getInt(Constants.SPRING_DATASOURCE_MAX_ACTIVE, 50));
dataSource.setConnectionTestQuery(properties.getValidationQuery());
- if (properties.getProps() != null) {
- properties.getProps().forEach(dataSource::addDataSourceProperty);
+ if (MapUtils.isNotEmpty(properties.getOther())) {
+ properties.getOther().forEach(dataSource::addDataSourceProperty);
}
logger.info("Creating HikariDataSource pool success.");
@@ -82,8 +83,8 @@ public class JDBCDataSourceProvider {
dataSource.setMaximumPoolSize(isOneSession ? 1 : PropertyUtils.getInt(Constants.SPRING_DATASOURCE_MAX_ACTIVE, 50));
dataSource.setConnectionTestQuery(properties.getValidationQuery());
- if (properties.getProps() != null) {
- properties.getProps().forEach(dataSource::addDataSourceProperty);
+ if (MapUtils.isNotEmpty(properties.getOther())) {
+ properties.getOther().forEach(dataSource::addDataSourceProperty);
}
logger.info("Creating OneSession HikariDataSource pool success.");
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/AbstractDataSourceProcessorTest.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/AbstractDataSourceProcessorTest.java
index 63534dfc70..745f68ffa2 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/AbstractDataSourceProcessorTest.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/AbstractDataSourceProcessorTest.java
@@ -17,10 +17,15 @@
package org.apache.dolphinscheduler.plugin.datasource.api.datasource;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
+import org.apache.dolphinscheduler.spi.datasource.ConnectionParam;
+import org.apache.dolphinscheduler.spi.enums.DbType;
+
+import java.sql.Connection;
import java.util.HashMap;
import java.util.Map;
@@ -43,4 +48,71 @@ public class AbstractDataSourceProcessorTest {
other.put("arg0", "%");
doThrow(new IllegalArgumentException()).when(mockDataSourceProcessor).checkOther(other);
}
+
+ @Test
+ public void shouldNotIncludeMaliciousParams() {
+ AbstractDataSourceProcessor mockDataSourceProcessor = mock(AbstractDataSourceProcessor.class);
+ Map<String, String> other = new HashMap<>();
+ other.put("allowLoadLocalInfile", "whatever");
+ doThrow(new IllegalArgumentException()).when(mockDataSourceProcessor).checkOther(other);
+ }
+
+ @org.junit.Test
+ public void transformOtherParamToMap() {
+ AbstractDataSourceProcessor abstractDataSourceProcessor = new AbstractDataSourceProcessor() {
+
+ @Override
+ public BaseDataSourceParamDTO castDatasourceParamDTO(String paramJson) {
+ return null;
+ }
+
+ @Override
+ public BaseDataSourceParamDTO createDatasourceParamDTO(String connectionJson) {
+ return null;
+ }
+
+ @Override
+ public ConnectionParam createConnectionParams(BaseDataSourceParamDTO datasourceParam) {
+ return null;
+ }
+
+ @Override
+ public ConnectionParam createConnectionParams(String connectionJson) {
+ return null;
+ }
+
+ @Override
+ public String getDatasourceDriver() {
+ return null;
+ }
+
+ @Override
+ public String getValidationQuery() {
+ return null;
+ }
+
+ @Override
+ public String getJdbcUrl(ConnectionParam connectionParam) {
+ return null;
+ }
+
+ @Override
+ public Connection getConnection(ConnectionParam connectionParam) {
+ return null;
+ }
+
+ @Override
+ public DbType getDbType() {
+ return null;
+ }
+
+ @Override
+ public DataSourceProcessor create() {
+ return null;
+ }
+ };
+ String other = "{\"useSSL\": \"true\"}";
+ Map<String, String> stringStringMap = abstractDataSourceProcessor.transformOtherParamToMap(other);
+ assertEquals("true", stringStringMap.get("useSSL"));
+ }
}
\ No newline at end of file
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/db2/Db2DataSourceProcessorTest.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/db2/Db2DataSourceProcessorTest.java
index 8789213351..95cde6f364 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/db2/Db2DataSourceProcessorTest.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/db2/Db2DataSourceProcessorTest.java
@@ -36,6 +36,8 @@ import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
+import com.google.common.collect.ImmutableMap;
+
@RunWith(PowerMockRunner.class)
@PrepareForTest({Class.class, DriverManager.class, DataSourceUtils.class, CommonUtils.class, DataSourceClientProvider.class, PasswordUtils.class})
public class Db2DataSourceProcessorTest {
@@ -83,7 +85,10 @@ public class Db2DataSourceProcessorTest {
public void testGetJdbcUrl() {
Db2ConnectionParam db2ConnectionParam = new Db2ConnectionParam();
db2ConnectionParam.setJdbcUrl("jdbc:db2://localhost:5142/default");
- db2ConnectionParam.setOther("other=other");
+ ImmutableMap<String, String> map = new ImmutableMap.Builder<String, String>()
+ .put("other", "other")
+ .build();
+ db2ConnectionParam.setOther(map);
String jdbcUrl = db2DatasourceProcessor.getJdbcUrl(db2ConnectionParam);
Assert.assertEquals("jdbc:db2://localhost:5142/default;other=other", jdbcUrl);
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/oracle/OracleDataSourceProcessorTest.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/oracle/OracleDataSourceProcessorTest.java
index a997e8728e..4b7cc752d5 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/oracle/OracleDataSourceProcessorTest.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/oracle/OracleDataSourceProcessorTest.java
@@ -37,6 +37,8 @@ import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
+import com.google.common.collect.ImmutableMap;
+
@RunWith(PowerMockRunner.class)
@PrepareForTest({Class.class, DriverManager.class, DataSourceUtils.class, CommonUtils.class, DataSourceClientProvider.class, PasswordUtils.class})
public class OracleDataSourceProcessorTest {
@@ -83,8 +85,11 @@ public class OracleDataSourceProcessorTest {
public void testGetJdbcUrl() {
OracleConnectionParam oracleConnectionParam = new OracleConnectionParam();
oracleConnectionParam.setJdbcUrl("jdbc:oracle:thin:@localhost:3308:default");
- oracleConnectionParam.setOther("other=other");
- Assert.assertEquals("jdbc:oracle:thin:@localhost:3308:default?other=other",
+ ImmutableMap<String, String> map = new ImmutableMap.Builder<String, String>()
+ .put("other", "other")
+ .build();
+ oracleConnectionParam.setOther(map);
+ Assertions.assertEquals("jdbc:oracle:thin:@localhost:3308:default?other=other",
oracleDatasourceProcessor.getJdbcUrl(oracleConnectionParam));
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/postgresql/PostgreSQLDataSourceProcessorTest.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/postgresql/PostgreSQLDataSourceProcessorTest.java
index 8342094c70..a784370880 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/postgresql/PostgreSQLDataSourceProcessorTest.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/postgresql/PostgreSQLDataSourceProcessorTest.java
@@ -36,6 +36,8 @@ import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
+import com.google.common.collect.ImmutableMap;
+
@RunWith(PowerMockRunner.class)
@PrepareForTest({Class.class, DriverManager.class, DataSourceUtils.class, CommonUtils.class, DataSourceClientProvider.class, PasswordUtils.class})
public class PostgreSQLDataSourceProcessorTest {
@@ -81,10 +83,13 @@ public class PostgreSQLDataSourceProcessorTest {
public void testGetJdbcUrl() {
PostgreSQLConnectionParam postgreSqlConnectionParam = new PostgreSQLConnectionParam();
postgreSqlConnectionParam.setJdbcUrl("jdbc:postgresql://localhost:3308/default");
- postgreSqlConnectionParam.setOther("other");
+ ImmutableMap<String, String> map = new ImmutableMap.Builder<String, String>()
+ .put("other", "other")
+ .build();
+ postgreSqlConnectionParam.setOther(map);
String jdbcUrl = postgreSqlDatasourceProcessor.getJdbcUrl(postgreSqlConnectionParam);
- Assert.assertEquals("jdbc:postgresql://localhost:3308/default?other", jdbcUrl);
+ Assertions.assertEquals("jdbc:postgresql://localhost:3308/default?other=other", jdbcUrl);
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/presto/PrestoDataSourceProcessorTest.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/presto/PrestoDataSourceProcessorTest.java
index f43e747bc2..24026c8075 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/presto/PrestoDataSourceProcessorTest.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/presto/PrestoDataSourceProcessorTest.java
@@ -80,8 +80,7 @@ public class PrestoDataSourceProcessorTest {
public void testGetJdbcUrl() {
PrestoConnectionParam prestoConnectionParam = new PrestoConnectionParam();
prestoConnectionParam.setJdbcUrl("jdbc:postgresql://localhost:1234/default");
- prestoConnectionParam.setOther("other");
- Assert.assertEquals("jdbc:postgresql://localhost:1234/default?other",
+ Assertions.assertEquals("jdbc:postgresql://localhost:1234/default",
prestoDatasourceProcessor.getJdbcUrl(prestoConnectionParam));
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessorTest.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessorTest.java
index a1756c3551..091919f1a8 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessorTest.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/redshift/RedshiftDataSourceProcessorTest.java
@@ -36,6 +36,8 @@ import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
+import com.google.common.collect.ImmutableMap;
+
@RunWith(PowerMockRunner.class)
@PrepareForTest({Class.class, DriverManager.class, DataSourceUtils.class, CommonUtils.class, DataSourceClientProvider.class, PasswordUtils.class})
public class RedshiftDataSourceProcessorTest {
@@ -80,9 +82,13 @@ public class RedshiftDataSourceProcessorTest {
public void testGetJdbcUrl() {
RedshiftConnectionParam redshiftConnectionParam = new RedshiftConnectionParam();
redshiftConnectionParam.setJdbcUrl("jdbc:redshift://localhost:5439/default");
- redshiftConnectionParam.setOther("DSILogLevel=6;defaultRowFetchSize=100");
- Assert.assertEquals("jdbc:redshift://localhost:5439/default?DSILogLevel=6;defaultRowFetchSize=100",
- redshiftDatasourceProcessor.getJdbcUrl(redshiftConnectionParam));
+ ImmutableMap<String, String> map = new ImmutableMap.Builder<String, String>()
+ .put("DSILogLevel", "6")
+ .put("defaultRowFetchSize", "100")
+ .build();
+ redshiftConnectionParam.setOther(map);
+ Assertions.assertEquals("jdbc:redshift://localhost:5439/default?DSILogLevel=6;defaultRowFetchSize=100",
+ redshiftDatasourceProcessor.getJdbcUrl(redshiftConnectionParam));
}
diff --git a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/spark/SparkDataSourceProcessorTest.java b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/spark/SparkDataSourceProcessorTest.java
index 7477ce7d24..f0acb9903f 100644
--- a/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/spark/SparkDataSourceProcessorTest.java
+++ b/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/test/java/org/apache/dolphinscheduler/plugin/datasource/api/datasource/spark/SparkDataSourceProcessorTest.java
@@ -82,8 +82,7 @@ public class SparkDataSourceProcessorTest {
public void testGetJdbcUrl() {
SparkConnectionParam sparkConnectionParam = new SparkConnectionParam();
sparkConnectionParam.setJdbcUrl("jdbc:hive2://localhost1:1234,localhost2:1234/default");
- sparkConnectionParam.setOther("other");
- Assert.assertEquals("jdbc:hive2://localhost1:1234,localhost2:1234/default;other",
+ Assertions.assertEquals("jdbc:hive2://localhost1:1234,localhost2:1234/default",
sparkDatasourceProcessor.getJdbcUrl(sparkConnectionParam));
}
diff --git a/dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/datasource/BaseConnectionParam.java b/dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/datasource/BaseConnectionParam.java
index b1df15c397..40e809c3d7 100644
--- a/dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/datasource/BaseConnectionParam.java
+++ b/dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/datasource/BaseConnectionParam.java
@@ -17,12 +17,14 @@
package org.apache.dolphinscheduler.spi.datasource;
-import java.util.HashMap;
import java.util.Map;
+import lombok.Data;
+
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonInclude.Include;
+@Data
@JsonInclude(Include.NON_NULL)
public abstract class BaseConnectionParam implements ConnectionParam {
@@ -42,87 +44,6 @@ public abstract class BaseConnectionParam implements ConnectionParam {
protected String validationQuery;
- protected String other;
-
- private Map<String, String> props = new HashMap<>();
-
- public String getUser() {
- return user;
- }
-
- public void setUser(String user) {
- this.user = user;
- }
-
- public String getPassword() {
- return password;
- }
-
- public void setPassword(String password) {
- this.password = password;
- }
-
- public String getAddress() {
- return address;
- }
-
- public void setAddress(String address) {
- this.address = address;
- }
-
- public String getDatabase() {
- return database;
- }
-
- public void setDatabase(String database) {
- this.database = database;
- }
-
- public String getJdbcUrl() {
- return jdbcUrl;
- }
-
- public void setJdbcUrl(String jdbcUrl) {
- this.jdbcUrl = jdbcUrl;
- }
-
- public String getDriverLocation() {
- return driverLocation;
- }
-
- public void setDriverLocation(String driverLocation) {
- this.driverLocation = driverLocation;
- }
-
- public String getDriverClassName() {
- return driverClassName;
- }
-
- public void setDriverClassName(String driverClassName) {
- this.driverClassName = driverClassName;
- }
-
- public String getValidationQuery() {
- return validationQuery;
- }
-
- public void setValidationQuery(String validationQuery) {
- this.validationQuery = validationQuery;
- }
-
- public String getOther() {
- return other;
- }
-
- public void setOther(String other) {
- this.other = other;
- }
-
- public Map<String, String> getProps() {
- return props;
- }
+ protected Map<String, String> other;
- public void setProps(Map<String, String> props) {
- this.props = props;
- }
}
diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-dataquality/src/test/java/org/apache/dolphinscheduler/plugin/task/dq/DataQualityTaskTest.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-dataquality/src/test/java/org/apache/dolphinscheduler/plugin/task/dq/DataQualityTaskTest.java
index e3bd5f55d3..1d981619a3 100644
--- a/dolphinscheduler-task-plugin/dolphinscheduler-task-dataquality/src/test/java/org/apache/dolphinscheduler/plugin/task/dq/DataQualityTaskTest.java
+++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-dataquality/src/test/java/org/apache/dolphinscheduler/plugin/task/dq/DataQualityTaskTest.java
@@ -494,7 +494,7 @@ public class DataQualityTaskTest {
+ "\"jdbcUrl\":\"jdbc:postgresql://localhost:5432/dolphinscheduler\","
+ "\"user\":\"test\","
+ "\"password\":\"test\","
- + "\"other\":\"stringtype=unspecified&characterEncoding=UTF-8&allowMultiQueries=true\"}");
+ + "\"other\":{\"stringtype\": \"unspecified\", \"characterEncoding\" : \"UTF-8\", \"allowMultiQueries\": true}}");
dataQualityTaskExecutionContext.setStatisticsValueConnectorType("JDBC");
dataQualityTaskExecutionContext.setStatisticsValueType(1);
@@ -505,7 +505,7 @@ public class DataQualityTaskTest {
+ "\"jdbcUrl\":\"jdbc:postgresql://localhost:5432/dolphinscheduler\","
+ "\"user\":\"test\","
+ "\"password\":\"test\","
- + "\"other\":\"stringtype=unspecified&characterEncoding=UTF-8&allowMultiQueries=true\"}");
+ + "\"other\":{\"stringtype\": \"unspecified\", \"characterEncoding\" : \"UTF-8\", \"allowMultiQueries\": true}}");
dataQualityTaskExecutionContext.setCompareWithFixedValue(true);
@@ -791,7 +791,7 @@ public class DataQualityTaskTest {
+ "\"jdbcUrl\":\"jdbc:postgresql://localhost:5432/dolphinscheduler\","
+ "\"user\":\"test\","
+ "\"password\":\"test\","
- + "\"other\":\"stringtype=unspecified&characterEncoding=UTF-8&allowMultiQueries=true\"}");
+ + "\"other\":{\"stringtype\": \"unspecified\", \"characterEncoding\": \"UTF-8\", \"allowMultiQueries\": true}}");
String expect = "{\"name\":\"跨表值比对\",\"env\":{\"type\":\"batch\",\"config\":null},\"readers\""
+ ":[{\"type\":\"JDBC\",\"config\":{\"database\":\"test\",\"password\":\"test\",\"driver\":"
@@ -1115,7 +1115,7 @@ public class DataQualityTaskTest {
+ "\"jdbcUrl\":\"jdbc:postgresql://localhost:5432/dolphinscheduler\","
+ "\"user\":\"test\","
+ "\"password\":\"test\","
- + "\"other\":\"stringtype=unspecified&characterEncoding=UTF-8&allowMultiQueries=true\"}");
+ + "\"other\":{\"stringtype\": \"unspecified\", \"characterEncoding\": \"UTF-8\", \"allowMultiQueries\": true}}");
dataQualityTaskExecutionContext.setStatisticsValueConnectorType("JDBC");
dataQualityTaskExecutionContext.setStatisticsValueType(1);
@@ -1126,7 +1126,7 @@ public class DataQualityTaskTest {
+ "\"jdbcUrl\":\"jdbc:postgresql://localhost:5432/dolphinscheduler\","
+ "\"user\":\"test\","
+ "\"password\":\"test\","
- + "\"other\":\"stringtype=unspecified&characterEncoding=UTF-8&allowMultiQueries=true\"}");
+ + "\"other\":{\"stringtype\": \"unspecified\", \"characterEncoding\": \"UTF-8\", \"allowMultiQueries\": true }}");
dataQualityTaskExecutionContext.setRuleName("跨表准确性");
dataQualityTaskExecutionContext.setRuleType(RuleType.MULTI_TABLE_ACCURACY.getCode());
[dolphinscheduler] 03/11: [Improvement-13084][UI] Make download log button disabled when host is null (#13090)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit d79dc31e31942cebb7fd500f8ca9cda1366f8029
Author: xla0 <11...@users.noreply.github.com>
AuthorDate: Sat Dec 10 12:34:35 2022 +0800
[Improvement-13084][UI] Make download log button disabled when host is null (#13090)
Co-authored-by: xiaojingxuan <13...@163.com>
(cherry picked from commit 58d21abfa166cb8c93ca4ff511f769dd147b2aff)
---
dolphinscheduler-ui/src/views/projects/task/instance/use-table.ts | 1 +
1 file changed, 1 insertion(+)
diff --git a/dolphinscheduler-ui/src/views/projects/task/instance/use-table.ts b/dolphinscheduler-ui/src/views/projects/task/instance/use-table.ts
index ec225762fd..5b99884ff5 100644
--- a/dolphinscheduler-ui/src/views/projects/task/instance/use-table.ts
+++ b/dolphinscheduler-ui/src/views/projects/task/instance/use-table.ts
@@ -241,6 +241,7 @@ export function useTable() {
circle: true,
type: 'info',
size: 'small',
+ disabled: !row.host,
onClick: () => downloadLog(row.id)
},
{
[dolphinscheduler] 10/11: [Fix][Doc] Replace the links. (#13276)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.4-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit f409749cb953699c372219dfe5ba07058c46eafb
Author: Amy0104 <am...@163.com>
AuthorDate: Mon Dec 26 14:32:40 2022 +0800
[Fix][Doc] Replace the links. (#13276)
* [Fix][Doc] Replace the links.
* [Fix][Doc] Revised `.dlc.json`
* [Fix] Fix the dead link.
(cherry picked from commit 1f7f900e7858067d9e72b01dbb6b10ffb8eec301)
---
.dlc.json | 13 ++++++-------
README.md | 4 ++--
README_zh_CN.md | 4 ++--
deploy/README.md | 4 ++--
docs/docs/en/contribute/architecture-design.md | 4 +---
docs/docs/en/contribute/development-environment-setup.md | 4 ++--
docs/docs/en/contribute/join/review.md | 4 ++--
docs/docs/en/contribute/release/release.md | 4 ++--
docs/docs/en/faq.md | 4 ++--
docs/docs/en/guide/installation/kubernetes.md | 2 +-
docs/docs/en/guide/installation/pseudo-cluster.md | 2 +-
docs/docs/en/guide/installation/standalone.md | 2 +-
docs/docs/en/guide/start/docker.md | 2 +-
docs/docs/en/guide/upgrade.md | 2 +-
docs/docs/zh/contribute/architecture-design.md | 2 +-
docs/docs/zh/contribute/join/review.md | 4 ++--
docs/docs/zh/faq.md | 6 +++---
docs/docs/zh/guide/installation/kubernetes.md | 2 +-
docs/docs/zh/guide/installation/pseudo-cluster.md | 16 ++++++++--------
docs/docs/zh/guide/installation/standalone.md | 2 +-
docs/docs/zh/guide/start/docker.md | 2 +-
docs/docs/zh/guide/upgrade.md | 2 +-
22 files changed, 44 insertions(+), 47 deletions(-)
diff --git a/.dlc.json b/.dlc.json
index da44b95b8e..2dcead4686 100644
--- a/.dlc.json
+++ b/.dlc.json
@@ -8,16 +8,19 @@
},
{
"pattern": "/community/community.html$"
+ },
+ {
+ "pattern": "^https://dolphinscheduler.apache.org"
}
],
"replacementPatterns": [
{
"pattern": "^/en-us/download/download.html$",
- "replacement": "https://dolphinscheduler.apache.org/en-us/download/download.html"
+ "replacement": "https://dolphinscheduler.apache.org/en-us/download"
},
{
"pattern": "^/zh-cn/download/download.html$",
- "replacement": "https://dolphinscheduler.apache.org/zh-cn/download/download.html"
+ "replacement": "https://dolphinscheduler.apache.org/zh-cn/download"
}
],
"httpHeaders": [
@@ -32,9 +35,5 @@
"retryOn429": true,
"retryCount": 10,
"fallbackRetryDelay": "1000s",
- "aliveStatusCodes": [
- 200,
- 401,
- 0
- ]
+ "aliveStatusCodes": [200, 401, 0]
}
diff --git a/README.md b/README.md
index 4f06d8d239..530c87dff9 100644
--- a/README.md
+++ b/README.md
@@ -55,11 +55,11 @@ supports custom task type
## QuickStart in Docker
-Please refer the official website document: [QuickStart in Docker](https://dolphinscheduler.apache.org/en-us/docs/3.0.2/user_doc/guide/start/docker.html)
+Please refer the official website document: [QuickStart in Docker](https://dolphinscheduler.apache.org/en-us/docs/3.0.3/guide/start/docker)
## QuickStart in Kubernetes
-Please refer to the official website document: [QuickStart in Kubernetes](https://dolphinscheduler.apache.org/en-us/docs/3.0.2/user_doc/guide/installation/kubernetes.html)
+Please refer to the official website document: [QuickStart in Kubernetes](https://dolphinscheduler.apache.org/en-us/docs/3.0.3/guide/installation/kubernetes)
## How to Build
diff --git a/README_zh_CN.md b/README_zh_CN.md
index ddf66202f4..8411fc1148 100644
--- a/README_zh_CN.md
+++ b/README_zh_CN.md
@@ -53,11 +53,11 @@ DolphinScheduler的工作计划：<a href="https://github.com/apache/dolphinsche
## 快速试用 Docker
-请参考官方文档: [快速试用 Docker 部署](https://dolphinscheduler.apache.org/#/zh-cn/docs/3.0.3/guide/start/docker)
+请参考官方文档: [快速试用 Docker 部署](https://dolphinscheduler.apache.org/zh-cn/docs/3.0.3/guide/start/docker)
## 快速试用 Kubernetes
-请参考官方文档: [快速试用 Kubernetes 部署](http://dolphinscheduler.apache.org/#/zh-cn/docs/3.0.3/guide/installation/kubernetes)
+请参考官方文档: [快速试用 Kubernetes 部署](https://dolphinscheduler.apache.org/zh-cn/docs/3.0.3/guide/installation/kubernetes)
## 如何构建
diff --git a/deploy/README.md b/deploy/README.md
index 80d8123ba9..2e568c1c89 100644
--- a/deploy/README.md
+++ b/deploy/README.md
@@ -1,4 +1,4 @@
# DolphinScheduler for Docker and Kubernetes
-- [Start Up DolphinScheduler with Docker](https://dolphinscheduler.apache.org/#/en-us/docs/3.0.3/guide/start/docker)
-- [Start Up DolphinScheduler with Kubernetes](https://dolphinscheduler.apache.org/#/en-us/docs/3.0.3/guide/installation/kubernetes)
+- [Start Up DolphinScheduler with Docker](https://dolphinscheduler.apache.org/en-us/docs/3.0.3/guide/start/docker)
+- [Start Up DolphinScheduler with Kubernetes](https://dolphinscheduler.apache.org/en-us/docs/3.0.3/guide/installation/kubernetes)
diff --git a/docs/docs/en/contribute/architecture-design.md b/docs/docs/en/contribute/architecture-design.md
index e48d53e9e1..3a4abf2a60 100644
--- a/docs/docs/en/contribute/architecture-design.md
+++ b/docs/docs/en/contribute/architecture-design.md
@@ -97,9 +97,7 @@ Before explaining the architecture of the schedule system, let us first understa
- **UI**
- The front-end page of the system provides various visual operation interfaces of the system. For details, see the [quick start](https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/about/introduction.html) section.
-
-
+ The front-end page of the system provides various visual operation interfaces of the system. For details, see the [quick start](https://dolphinscheduler.apache.org/en-us/docs/3.0.3/about/introduction) section.
#### 2.3 Architectural Design Ideas
diff --git a/docs/docs/en/contribute/development-environment-setup.md b/docs/docs/en/contribute/development-environment-setup.md
index 0940682d6e..9c659ad2cc 100644
--- a/docs/docs/en/contribute/development-environment-setup.md
+++ b/docs/docs/en/contribute/development-environment-setup.md
@@ -91,8 +91,8 @@ There are two ways to configure the DolphinScheduler development environment, st
> **_Note:_** Use standalone server only for development and debugging, because it uses H2 Database as default database and Zookeeper Testing Server which may not be stable in production.
>
> Standalone is only supported in DolphinScheduler 1.3.9 and later versions.
->
-> Standalone server is able to connect to external databases like mysql and postgresql, see [Standalone Deployment](https://dolphinscheduler.apache.org/en-us/docs/dev/user_doc/guide/installation/standalone.html) for instructions.
+>
+> Standalone server is able to connect to external databases like mysql and postgresql, see [Standalone Deployment](https://dolphinscheduler.apache.org/en-us/docs/3.0.3/guide/installation/standalone) for instructions.
### Git Branch Choose
diff --git a/docs/docs/en/contribute/join/review.md b/docs/docs/en/contribute/join/review.md
index 29a8b8465a..bbdea78272 100644
--- a/docs/docs/en/contribute/join/review.md
+++ b/docs/docs/en/contribute/join/review.md
@@ -1,6 +1,6 @@
# Community Review
-Beside submit Issues and pull requests to the GitHub repository mentioned in [team](https://dolphinscheduler.apache.org/#/en-us/community), another important way to
+Beside submit Issues and pull requests to the GitHub repository mentioned in [team](https://dolphinscheduler.apache.org/en-us/community), another important way to
contribute to DolphinScheduler is reviewing GitHub Issues or Pull Requests. You can not only know the latest new and
direction of the community, but also understand the good design in others during you reviewing. At the same time, you can
increase your exposure in the community and accumulate your honor.
@@ -12,7 +12,7 @@ in [GitHub Discussion][discussion-result-review-wanted].
> Note: It is only users mentioned in the [GitHub Discussion][discussion-result-review-wanted] can review Issues or Pull
> Requests, Community advocates **Anyone is encouraged to review Issues and Pull Requests**. Users in
> [GitHub Discussion][discussion-result-review-wanted] show their willing to review when we collect in the mail thread.
-> The advantage of this list is when the community has discussion, in addition to the mention Members in [team](https://dolphinscheduler.apache.org/#/en-us/community),
+> The advantage of this list is when the community has discussion, in addition to the mention Members in [team](https://dolphinscheduler.apache.org/en-us/community),
> you can also find some help in [GitHub Discussion][discussion-result-review-wanted] people. If you want to join the
> [GitHub Discussion][discussion-result-review-wanted], please comment in that discussion and leave a module you are interested
> in, and the maintainer will add you to the list.
diff --git a/docs/docs/en/contribute/release/release.md b/docs/docs/en/contribute/release/release.md
index 6a47abf991..52e25ee940 100644
--- a/docs/docs/en/contribute/release/release.md
+++ b/docs/docs/en/contribute/release/release.md
@@ -523,7 +523,7 @@ Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.
-Download Links: https://dolphinscheduler.apache.org/en-us/download/download.html
+Download Links: https://dolphinscheduler.apache.org/en-us/download
Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION>
@@ -532,5 +532,5 @@ Website: https://dolphinscheduler.apache.org/
DolphinScheduler Resources:
- Issue: https://github.com/apache/dolphinscheduler/issues/
- Mailing list: dev@dolphinscheduler.apache.org
-- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/user_doc/about/introduction.html
+- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/about/introduction
```
diff --git a/docs/docs/en/faq.md b/docs/docs/en/faq.md
index 3bb5b16efb..6378a06ccc 100644
--- a/docs/docs/en/faq.md
+++ b/docs/docs/en/faq.md
@@ -302,9 +302,9 @@ change into
## Q : how to add a worker server
-A: 1, Create deployment user and hosts mapping, please refer 1.3 part of [cluster deployment](https://dolphinscheduler.apache.org/#/en-us/docs/3.0.3/user_doc/installation/cluster)
+A: 1, Create deployment user and hosts mapping, please refer 1.3 part of [cluster deployment](https://dolphinscheduler.apache.org/en-us/docs/3.0.3/user_doc/installation/cluster)
- 2, Configure hosts mapping and ssh access and modify directory permissions. please refer 1.4 part of [cluster deployment](https://dolphinscheduler.apache.org/en-us/docs/laster/user_doc/installation/cluster.html)
+ 2, Configure hosts mapping and ssh access and modify directory permissions. please refer 1.4 part of [cluster deployment](https://dolphinscheduler.apache.org/en-us/docs/3.0.3/user_doc/installation/cluster)
3, Copy the deployment directory from worker server that has already deployed
diff --git a/docs/docs/en/guide/installation/kubernetes.md b/docs/docs/en/guide/installation/kubernetes.md
index bd8242cc67..fdab38ba01 100644
--- a/docs/docs/en/guide/installation/kubernetes.md
+++ b/docs/docs/en/guide/installation/kubernetes.md
@@ -12,7 +12,7 @@ If you are a new hand and want to experience DolphinScheduler functions, we reco
## Install DolphinScheduler
-Please download the source code package `apache-dolphinscheduler-3.0.3-src.tar.gz`, download address: [download address](/en-us/download/download.html)
+Please download the source code package `apache-dolphinscheduler-<version>-src.tar.gz`, download address: [download address](https://dolphinscheduler.apache.org/en-us/download)
To publish the release name `dolphinscheduler` version, please execute the following commands:
diff --git a/docs/docs/en/guide/installation/pseudo-cluster.md b/docs/docs/en/guide/installation/pseudo-cluster.md
index 603c0ca62b..1754d61475 100644
--- a/docs/docs/en/guide/installation/pseudo-cluster.md
+++ b/docs/docs/en/guide/installation/pseudo-cluster.md
@@ -9,7 +9,7 @@ If you are a new hand and want to experience DolphinScheduler functions, we reco
Pseudo-cluster deployment of DolphinScheduler requires external software support:
- JDK：download [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment.
-- Binary package: Download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/#/en-us/download)
+- Binary package: Download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download)
- Database: [PostgreSQL](https://www.postgresql.org/download/) (8.2.15+) or [MySQL](https://dev.mysql.com/downloads/mysql/) (5.7+), you can choose one of the two, such as MySQL requires JDBC Driver 8.0.16
- Registry Center: [ZooKeeper](https://zookeeper.apache.org/releases.html) (3.8.0+)，[download link][zookeeper]
- Process tree analysis
diff --git a/docs/docs/en/guide/installation/standalone.md b/docs/docs/en/guide/installation/standalone.md
index e5201c6438..5a0e55d017 100644
--- a/docs/docs/en/guide/installation/standalone.md
+++ b/docs/docs/en/guide/installation/standalone.md
@@ -12,7 +12,7 @@ If you want to deploy DolphinScheduler in production, we recommend you follow [c
## Preparation
- JDK：download [JDK][jdk] (1.8+), install and configure environment variable `JAVA_HOME` and append `bin` dir (included in `JAVA_HOME`) to `PATH` variable. You can skip this step if it already exists in your environment.
-- Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/#/en-us/download).
+- Binary package: download the DolphinScheduler binary package at [download page](https://dolphinscheduler.apache.org/en-us/download).
## Start DolphinScheduler Standalone Server
diff --git a/docs/docs/en/guide/start/docker.md b/docs/docs/en/guide/start/docker.md
index 8172af5d8f..dd43e2c30f 100644
--- a/docs/docs/en/guide/start/docker.md
+++ b/docs/docs/en/guide/start/docker.md
@@ -45,7 +45,7 @@ modify docker-compose's free memory up to 4 GB.
- Hyper-V mode: Click `Docker Desktop -> Settings -> Resources -> Memory` modified it
- WSL 2 mode: see [WSL 2 utility VM](https://docs.microsoft.com/zh-cn/windows/wsl/wsl-config#configure-global-options-with-wslconfig) for more detail.
-After complete the configuration, we can get the `docker-compose.yaml` file from [download page](https://dolphinscheduler.apache.org/#/en-us/download)
+After complete the configuration, we can get the `docker-compose.yaml` file from [download page](https://dolphinscheduler.apache.org/en-us/download)
form its source package, and make sure you get the right version. After download the package, you can run the commands as below.
```shell
diff --git a/docs/docs/en/guide/upgrade.md b/docs/docs/en/guide/upgrade.md
index 3af991c13b..92a54f5885 100644
--- a/docs/docs/en/guide/upgrade.md
+++ b/docs/docs/en/guide/upgrade.md
@@ -8,7 +8,7 @@ To prevent data loss by some miss-operation, it is recommended to back up data b
### Download the Latest Version Installation Package
-Download the latest binary distribute package from [download](https://dolphinscheduler.apache.org/#/en-us/download) and then put it in the different
+Download the latest binary distribute package from [download](https://dolphinscheduler.apache.org/en-us/download) and then put it in the different
directory where current service running. And all below command is running in this directory.
## Upgrade
diff --git a/docs/docs/zh/contribute/architecture-design.md b/docs/docs/zh/contribute/architecture-design.md
index 8e439ad20f..fc5fc94e3c 100644
--- a/docs/docs/zh/contribute/architecture-design.md
+++ b/docs/docs/zh/contribute/architecture-design.md
@@ -86,7 +86,7 @@
* **UI**
- 系统的前端页面，提供系统的各种可视化操作界面，详见 [快速开始](https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/about/introduction.html) 部分。
+ 系统的前端页面，提供系统的各种可视化操作界面，详见 [快速开始](https://dolphinscheduler.apache.org/zh-cn/docs/3.0.3/about/introduction) 部分。
#### 2.3 架构设计思想
diff --git a/docs/docs/zh/contribute/join/review.md b/docs/docs/zh/contribute/join/review.md
index d48ae791e4..14d78d5fd9 100644
--- a/docs/docs/zh/contribute/join/review.md
+++ b/docs/docs/zh/contribute/join/review.md
@@ -1,6 +1,6 @@
# 参与社区 review
-贡献 DolphinScheduler 的方式，除了向 [团队](https://dolphinscheduler.apache.org/#/zh-cn/community) 中提到的 GitHub 仓库提交 Issues 和 pull requests 外，另一非常重要的方式是
+贡献 DolphinScheduler 的方式，除了向 [团队](https://dolphinscheduler.apache.org/zh-cn/community) 中提到的 GitHub 仓库提交 Issues 和 pull requests 外，另一非常重要的方式是
review 社区的 Issues 或者 Pull Requests。通过别人 Issues 和 Pull Requests，你不仅能知道社区的最新进展和发展方向，还能了解别人代码的设
计思想，同时可以增加自己在社区的曝光、积累自己在社区的荣誉值。
@@ -9,7 +9,7 @@ review 社区的 Issues 或者 Pull Requests。通过别人 Issues 和 Pull Requ
> 注意: 这里并不是说只有 [GitHub Discussion][discussion-result-review-wanted] 中提及的用户才可以协助 review Issue 或者 Pull Requests，
> 请记住社区的主张是 **任何人都被鼓励去 review 社区的 Issues 和 Pull Requests**。只是那部分用户在邮件列表意见征集的时候，表达了愿意付
-> 出更多的时间，参与社区的 review。另一个好处是，当社区有不确定的问题的时，除了可以找 [团队](https://dolphinscheduler.apache.org/#/zh-cn/community) 中对应的 Members 外，还可以找
+> 出更多的时间，参与社区的 review。另一个好处是，当社区有不确定的问题的时，除了可以找 [团队](https://dolphinscheduler.apache.org/zh-cn/community) 中对应的 Members 外，还可以找
> [GitHub Discussion][discussion-result-review-wanted] 中提及的人解答对应的问题。如果你要想要加入到 [GitHub Discussion][discussion-result-review-wanted]
> 中，请在该 discussion 中评论并留下你感兴趣的模块，维护者会将你加入到对应的名单中。
diff --git a/docs/docs/zh/faq.md b/docs/docs/zh/faq.md
index 67b45302cd..b06c5c33f7 100644
--- a/docs/docs/zh/faq.md
+++ b/docs/docs/zh/faq.md
@@ -285,10 +285,10 @@ A： 将 hive pom
## Q：如何增加一台工作服务器
-A： 1，参考官网[部署文档](https://dolphinscheduler.apache.org/#/zh-cn/docs/3.0.3/installation/cluster) 1.3 小节，创建部署用户和 hosts 映射
+A： 1，参考官网[部署文档](https://dolphinscheduler.apache.org/zh-cn/docs/3.0.3/installation/cluster) 1.3 小节，创建部署用户和 hosts 映射
- 2，参考官网[部署文档](https://dolphinscheduler.apache.org/zh-cn/docs/laster/user_doc/installation/cluster.html) 1.4 小节，配置 hosts 映射和 ssh 打通及修改目录权限.
- 1.4 小节的最后一步是在当前新增机器上执行的，即需要给部署目录部署用户的权限
+ 2，参考官网[部署文档](https://dolphinscheduler.apache.org/zh-cn/docs/3.0.3/installation/cluster) 1.4 小节，配置 hosts 映射和 ssh 打通及修改目录权限.
+ 1.4 小节的最后一步是在当前新增机器上执行的，即需要给部署目录部署用户的权限
3，复制正在运行的服务器上的部署目录到新机器的同样的部署目录下
diff --git a/docs/docs/zh/guide/installation/kubernetes.md b/docs/docs/zh/guide/installation/kubernetes.md
index 3de1c51b09..52d1dd94c8 100644
--- a/docs/docs/zh/guide/installation/kubernetes.md
+++ b/docs/docs/zh/guide/installation/kubernetes.md
@@ -12,7 +12,7 @@ Kubernetes部署目的是在Kubernetes集群中部署 DolphinScheduler 服务，
## 安装 dolphinscheduler
-请下载源码包 apache-dolphinscheduler-<version>-src.tar.gz，下载地址: [下载](/zh-cn/download/download.html)
+请下载源码包 apache-dolphinscheduler-<version>-src.tar.gz，下载地址: [下载](https://dolphinscheduler.apache.org/zh-cn/download)
发布一个名为 `dolphinscheduler` 的版本(release)，请执行以下命令：
diff --git a/docs/docs/zh/guide/installation/pseudo-cluster.md b/docs/docs/zh/guide/installation/pseudo-cluster.md
index 5d1012d762..12e0c61880 100644
--- a/docs/docs/zh/guide/installation/pseudo-cluster.md
+++ b/docs/docs/zh/guide/installation/pseudo-cluster.md
@@ -8,13 +8,13 @@
伪分布式部署 DolphinScheduler 需要有外部软件的支持
-* JDK：下载[JDK][jdk] (1.8+)，安装并配置 `JAVA_HOME` 环境变量，并将其下的 `bin` 目录追加到 `PATH` 环境变量中。如果你的环境中已存在，可以跳过这步。
-* 二进制包：在[下载页面](https://dolphinscheduler.apache.org/#/zh-cn/download)下载 DolphinScheduler 二进制包
-* 数据库：[PostgreSQL](https://www.postgresql.org/download/) (8.2.15+) 或者 [MySQL](https://dev.mysql.com/downloads/mysql/) (5.7+)，两者任选其一即可，如 MySQL 则需要 JDBC Driver 8.0.16
-* 注册中心：[ZooKeeper](https://zookeeper.apache.org/releases.html) (3.8.0+)，[下载地址][zookeeper]
-* 进程树分析
- * macOS安装`pstree`
- * Fedora/Red/Hat/CentOS/Ubuntu/Debian安装`psmisc`
+- JDK：下载[JDK][jdk] (1.8+)，安装并配置 `JAVA_HOME` 环境变量，并将其下的 `bin` 目录追加到 `PATH` 环境变量中。如果你的环境中已存在，可以跳过这步。
+- 二进制包：在[下载页面](https://dolphinscheduler.apache.org/zh-cn/download)下载 DolphinScheduler 二进制包
+- 数据库：[PostgreSQL](https://www.postgresql.org/download/) (8.2.15+) 或者 [MySQL](https://dev.mysql.com/downloads/mysql/) (5.7+)，两者任选其一即可，如 MySQL 则需要 JDBC Driver 8.0.16
+- 注册中心：[ZooKeeper](https://zookeeper.apache.org/releases.html) (3.8.0+)，[下载地址][zookeeper]
+- 进程树分析
+ - macOS 安装`pstree`
+ - Fedora/Red/Hat/CentOS/Ubuntu/Debian 安装`psmisc`
> **_注意:_** DolphinScheduler 本身不依赖 Hadoop、Hive、Spark，但如果你运行的任务需要依赖他们，就需要有对应的环境支持
@@ -141,7 +141,7 @@ export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:
## 初始化数据库
-请参考 [数据源配置] `伪分布式/分布式安装初始化数据库` 创建并初始化数据库
+请参考 [数据源配置] `伪分布式/分布式安装初始化数据库` 创建并初始化数据库
## 启动 DolphinScheduler
diff --git a/docs/docs/zh/guide/installation/standalone.md b/docs/docs/zh/guide/installation/standalone.md
index e334293682..409c7f3c43 100644
--- a/docs/docs/zh/guide/installation/standalone.md
+++ b/docs/docs/zh/guide/installation/standalone.md
@@ -10,7 +10,7 @@ Standalone 仅适用于 DolphinScheduler 的快速体验.
## 前置准备工作
- JDK：下载[JDK][jdk] (1.8+)，安装并配置 `JAVA_HOME` 环境变量，并将其下的 `bin` 目录追加到 `PATH` 环境变量中。如果你的环境中已存在，可以跳过这步。
-- 二进制包：在[下载页面](https://dolphinscheduler.apache.org/#/zh-cn/download)下载 DolphinScheduler 二进制包
+- 二进制包：在[下载页面](https://dolphinscheduler.apache.org/zh-cn/download)下载 DolphinScheduler 二进制包
## 启动 DolphinScheduler Standalone Server
diff --git a/docs/docs/zh/guide/start/docker.md b/docs/docs/zh/guide/start/docker.md
index df3020d2f7..404429e861 100644
--- a/docs/docs/zh/guide/start/docker.md
+++ b/docs/docs/zh/guide/start/docker.md
@@ -39,7 +39,7 @@ $ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:2
- Hyper-V 模式：点击 `Docker Desktop -> Settings -> Resources -> Memory` 调整内存大小
- WSL 2 模式 模式：参考 [WSL 2 utility VM](https://docs.microsoft.com/zh-cn/windows/wsl/wsl-config#configure-global-options-with-wslconfig) 调整内存大小
-配置完成后我们需要获取 `docker-compose.yaml` 文件，通过[下载页面](https://dolphinscheduler.apache.org/#/zh-cn/download)下载对应版本源码包可能是最快的方法，
+配置完成后我们需要获取 `docker-compose.yaml` 文件，通过[下载页面](https://dolphinscheduler.apache.org/zh-cn/download)下载对应版本源码包可能是最快的方法，
源码包对应的值为 "Total Source Code"。当下载完源码后就可以运行命令进行部署了。
```shell
diff --git a/docs/docs/zh/guide/upgrade.md b/docs/docs/zh/guide/upgrade.md
index 917fa8f3f9..4c7bd9c299 100644
--- a/docs/docs/zh/guide/upgrade.md
+++ b/docs/docs/zh/guide/upgrade.md
@@ -8,7 +8,7 @@
### 下载新版本的安装包
-在[下载](https://dolphinscheduler.apache.org/#/zh-cn/download)页面下载最新版本的二进制安装包，并将二进制包放到与当前 dolphinscheduler 服务不一样的路径中，以下升级操作都需要在新版本的目录进行。
+在[下载](https://dolphinscheduler.apache.org/zh-cn/download)页面下载最新版本的二进制安装包，并将二进制包放到与当前 dolphinscheduler 服务不一样的路径中，以下升级操作都需要在新版本的目录进行。
## 升级步骤