You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by zh...@apache.org on 2022/05/23 07:00:49 UTC
[dolphinscheduler] branch 3.0.0-beta-prepare updated (1b080afa39 -> 68e6e45bb8)
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a change to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
discard 1b080afa39 [Cherry-Pick][Fix][UI] Add seatunnel icon (#10187)
omit 4cb307e955 [Fix]Change the HTTP or HTTPS regular expression (#10166) (#10176)
omit 0279391da7 [Cherry-Pick][Fix][UI] Fix the problem that the failure strategy field in the time management form is multilingual. (#10173)
omit 9d91d449fd [Cherry-Pick][Fix][UI] Limit task group resource capacity to only positive integers. (#10162)
omit 13d723bbe4 [Cherry-Pick][Build][UI] Solve vite config throws terser warning. (#10155)
omit 8b0f222762 [Fix][UI] Fix bug where username is empty in file manage (#10070) (#10073)
omit e3fd25d5e2 [Cherry Pick][Fix][UI] Fix in the Chinese state, there is a problem with the multi-language support of the form. (#10072)
omit b3196ccd85 [Fix][UI]Modify error message (#10054) (#10059)
omit 6af11e9b05 [Fix][UI]Modify error message (#10054)
new eca859546e [Fix][UI]Modify error message (#10054)
new 6bc9141c91 [Fix][UI] Fix in the Chinese state, there is a problem with the multi-language support of the form. (#10066)
new c2abc650a2 [Fix][UI] Fix bug where username is empty in file manage (#10070)
new aa51c66d91 [Bug][Master] fix master task failover (#10065)
new 4f11be7126 [Fix-10080]When the created tenant name is too long, an error message will be prompted (#10081)
new cb91fc72ee Create process definition with task group and task group priority cannot be save into db (#10078)
new f46a4d7dd8 [Fix][Docker] Support Multi-Architecture Docker Images (#10052)
new 63ab98e67f [Fix-10082]Resource userName is null and udf function manager add a username (#10064)
new 98576cb509 [Fix-10049] Conditions Task branch flow failed (#10077)
new 04c68e31e8 [Feat][UI] Add resource username for function and resource manage (#10086)
new 4f74db829e fix issue-10061 (#10062)
new 2797fcfc68 [Fix][UI] fulfill missing fields in DataQuality (#10098)
new cbfeb7f777 [fix] Enhance name pre checker in resource center (#10094)
new b016037a6f [BUG][TaskGroup] Task group does not take effect (#10093)
new 2c290b2f2b [Build][UI] Solve vite config throws terser warning. (#10105)
new 4c62ee76e4 Fix alert server curator health check not work (#10111)
new 418d422bb9 [Bug][doc] Fix the open-api docs request error (#10100)
new ebe76fb986 [doc] Show all the functions of the home page through screenshots (#10118)
new eaa75d6516 [doc] change document structure (#10124)
new 21625f57a7 [doc] Remove flink call docs (#10122)
new bb47ced474 [python] Change distribute name avoid confusion with main pkg (#10134)
new a521d49e94 [Fix][UI] Add the task status of pending. (#10129)
new eed50c3670 [Fix][UI] Limit task group resource capacity to only positive integers. (#10158)
new 05f209d5c0 [doc] Update editorial workflow content and images (#10130)
new 8ba5fa1afd [doc] Update website docs link (#10161)
new 713a7802a4 [Fix]Change the HTTP or HTTPS regular expression (#10166)
new bbf621cb05 [Fix][UI] Fix the problem that the failure strategy field in the time management form is multilingual. (#10169)
new e1ff8b87c6 [Fix-10057][api] api parameter name misspelling (#10063)
new 180ee3ba95 [Fix][UI] Add seatunnel icon (#10178)
new 8a52958f36 fix spcial week init bug (#10183)
new bb30b88306 [Fix-10076] queryUserByToken use server timezone (#10177)
new e737d8ee30 [doc] Refactor global parameter docment (#10045)
new 68e6e45bb8 [doc] Update project directory structure in docs (#10107)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (1b080afa39)
\
N -- N -- N refs/heads/3.0.0-beta-prepare (68e6e45bb8)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 33 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/publish-docker.yaml | 4 +-
docs/configs/docsdev.js | 140 ++++++++++---------
docs/configs/site.js | 24 +---
docs/docs/en/architecture/configuration.md | 119 ++++++++++------
.../development/development-environment-setup.md | 54 ++++++++
docs/docs/en/faq.md | 43 +-----
docs/docs/en/guide/flink-call.md | 123 -----------------
docs/docs/en/guide/open-api.md | 69 +++++-----
docs/docs/en/guide/parameter/global.md | 28 ++--
docs/docs/en/guide/project/workflow-instance.md | 2 +-
docs/docs/en/guide/resource/file-manage.md | 31 +++--
docs/docs/en/guide/task/conditions.md | 3 +-
docs/docs/zh/architecture/configuration.md | 124 +++++++++++------
.../development/development-environment-setup.md | 53 +++++++-
docs/docs/zh/faq.md | 36 +----
docs/docs/zh/guide/flink-call.md | 150 ---------------------
docs/docs/zh/guide/open-api.md | 82 +++++------
docs/docs/zh/guide/parameter/global.md | 29 ++--
docs/docs/zh/guide/project/workflow-instance.md | 2 +-
docs/docs/zh/guide/resource/file-manage.md | 31 +++--
docs/docs/zh/guide/task/conditions.md | 3 +-
docs/img/api-documentation-en.png | Bin 77247 -> 0 bytes
docs/img/api-documentation.png | Bin 61869 -> 0 bytes
docs/img/api/create_project1.png | Bin 38616 -> 0 bytes
docs/img/api/create_project2.png | Bin 38329 -> 0 bytes
docs/img/api/create_project3.png | Bin 73852 -> 0 bytes
docs/img/api/create_queue.png | Bin 74021 -> 0 bytes
docs/img/api/create_source1.png | Bin 360228 -> 0 bytes
docs/img/api/create_source2.png | Bin 278176 -> 0 bytes
docs/img/api/create_tenant.png | Bin 82987 -> 0 bytes
docs/img/api/create_user.png | Bin 92945 -> 0 bytes
docs/img/api/create_workflow1.png | Bin 55287 -> 0 bytes
docs/img/api/create_workflow2.png | Bin 96320 -> 0 bytes
docs/img/api/create_workflow3.png | Bin 214012 -> 0 bytes
docs/img/api/create_workflow4.png | Bin 69490 -> 0 bytes
docs/img/api/execution_result.png | Bin 75120 -> 0 bytes
docs/img/api/log.png | Bin 154299 -> 0 bytes
docs/img/api/upload_resource.png | Bin 68267 -> 0 bytes
docs/img/api/user_authorization.png | Bin 38817 -> 0 bytes
docs/img/api/user_login.png | Bin 184975 -> 0 bytes
docs/img/create-token-en1.png | Bin 8779 -> 0 bytes
docs/img/create-token.png | Bin 7699 -> 0 bytes
docs/img/editDag-en.png | Bin 421677 -> 1300064 bytes
docs/img/editDag.png | Bin 26907 -> 1300064 bytes
docs/img/local_parameter.png | Bin 42275 -> 0 bytes
docs/img/local_parameter_en.png | Bin 72458 -> 0 bytes
docs/img/new_ui/dev/homepage/homepage.png | Bin 161117 -> 746975 bytes
docs/img/new_ui/dev/open-api/api_doc.png | Bin 0 -> 210921 bytes
docs/img/new_ui/dev/open-api/api_source01.png | Bin 0 -> 257758 bytes
docs/img/new_ui/dev/open-api/api_source02.png | Bin 0 -> 233612 bytes
docs/img/new_ui/dev/open-api/api_test.png | Bin 0 -> 181011 bytes
docs/img/new_ui/dev/open-api/create_project01.png | Bin 0 -> 104392 bytes
docs/img/new_ui/dev/open-api/create_project02.png | Bin 0 -> 118638 bytes
docs/img/new_ui/dev/open-api/create_project03.png | Bin 0 -> 52968 bytes
docs/img/new_ui/dev/open-api/token_expiration.png | Bin 0 -> 261485 bytes
.../new_ui/dev/parameter/global_parameter01.png | Bin 0 -> 148147 bytes
.../new_ui/dev/parameter/global_parameter02.png | Bin 0 -> 141585 bytes
.../new_ui/dev/parameter/global_parameter03.png | Bin 0 -> 312613 bytes
docs/img/new_ui/dev/project/project-overview.png | Bin 167215 -> 790835 bytes
docs/img/supplement_global_parameter.png | Bin 295951 -> 0 bytes
docs/img/supplement_global_parameter_en.png | Bin 124936 -> 0 bytes
docs/img/tasks/demo/condition_task04.png | Bin 0 -> 33271 bytes
docs/img/test-api.png | Bin 40028 -> 0 bytes
docs/img/token-management-en.png | Bin 35282 -> 0 bytes
docs/img/token-management.png | Bin 34871 -> 0 bytes
.../apache/dolphinscheduler/alert/AlertServer.java | 2 +-
.../api/controller/ProcessInstanceController.java | 2 +-
.../apache/dolphinscheduler/api/enums/Status.java | 5 +-
.../api/interceptor/LoginHandlerInterceptor.java | 4 +-
.../api/service/impl/ResourcesServiceImpl.java | 21 ++-
.../api/service/impl/TenantServiceImpl.java | 9 +-
.../interceptor/LoginHandlerInterceptorTest.java | 6 +-
.../apache/dolphinscheduler/common/Constants.java | 9 ++
.../dolphinscheduler/common/utils/FileUtils.java | 21 +++
.../common/utils/FileUtilsTest.java | 30 +++++
.../dolphinscheduler/dao/entity/Resource.java | 17 +++
.../dolphinscheduler/dao/entity/TaskInstance.java | 2 +
.../dolphinscheduler/dao/entity/UdfFunc.java | 14 ++
.../dolphinscheduler/dao/mapper/UserMapper.java | 6 +-
.../dolphinscheduler/dao/mapper/ResourceMapper.xml | 5 +-
.../dao/mapper/TaskDefinitionLogMapper.xml | 4 +-
.../dolphinscheduler/dao/mapper/UdfFuncMapper.xml | 5 +-
.../dolphinscheduler/dao/mapper/UserMapper.xml | 2 +-
.../dolphinscheduler/dao/entity/UdfFuncTest.java | 2 +-
.../dao/mapper/UserMapperTest.java | 2 +-
.../master/runner/WorkflowExecuteThread.java | 10 +-
.../master/runner/task/CommonTaskProcessor.java | 2 +-
.../master/runner/task/ConditionTaskProcessor.java | 5 +-
.../master/runner/task/TaskProcessorFactory.java | 9 ++
.../server/master/service/FailoverService.java | 63 +++++----
.../consumer/TaskPriorityQueueConsumerTest.java | 12 +-
.../server/master/service/FailoverServiceTest.java | 103 +++++++++-----
dolphinscheduler-python/pom.xml | 31 +++++
.../queue/PeerTaskInstancePriorityQueue.java | 5 +
.../service/queue/TaskPriority.java | 33 ++++-
.../queue/PeerTaskInstancePriorityQueueTest.java | 57 ++++++--
.../service/queue/TaskPriorityQueueImplTest.java | 62 ++++++---
dolphinscheduler-ui/src/common/common.ts | 11 +-
dolphinscheduler-ui/src/common/types.ts | 1 +
.../src/components/crontab/modules/day.tsx | 8 +-
dolphinscheduler-ui/src/locales/modules/en_US.ts | 8 +-
dolphinscheduler-ui/src/locales/modules/zh_CN.ts | 8 +-
.../task/components/node/fields/use-conditions.ts | 2 +-
.../projects/task/components/node/format-data.ts | 14 ++
.../views/projects/task/components/node/types.ts | 7 +
.../src/views/resource/udf/function/use-table.ts | 5 +
.../src/views/resource/udf/resource/use-table.ts | 5 +
pom.xml | 2 +
108 files changed, 1012 insertions(+), 769 deletions(-)
delete mode 100644 docs/docs/en/guide/flink-call.md
delete mode 100644 docs/docs/zh/guide/flink-call.md
delete mode 100644 docs/img/api-documentation-en.png
delete mode 100644 docs/img/api-documentation.png
delete mode 100644 docs/img/api/create_project1.png
delete mode 100644 docs/img/api/create_project2.png
delete mode 100644 docs/img/api/create_project3.png
delete mode 100644 docs/img/api/create_queue.png
delete mode 100644 docs/img/api/create_source1.png
delete mode 100644 docs/img/api/create_source2.png
delete mode 100644 docs/img/api/create_tenant.png
delete mode 100644 docs/img/api/create_user.png
delete mode 100644 docs/img/api/create_workflow1.png
delete mode 100644 docs/img/api/create_workflow2.png
delete mode 100644 docs/img/api/create_workflow3.png
delete mode 100644 docs/img/api/create_workflow4.png
delete mode 100644 docs/img/api/execution_result.png
delete mode 100644 docs/img/api/log.png
delete mode 100644 docs/img/api/upload_resource.png
delete mode 100644 docs/img/api/user_authorization.png
delete mode 100644 docs/img/api/user_login.png
delete mode 100644 docs/img/create-token-en1.png
delete mode 100644 docs/img/create-token.png
delete mode 100644 docs/img/local_parameter.png
delete mode 100644 docs/img/local_parameter_en.png
create mode 100644 docs/img/new_ui/dev/open-api/api_doc.png
create mode 100644 docs/img/new_ui/dev/open-api/api_source01.png
create mode 100644 docs/img/new_ui/dev/open-api/api_source02.png
create mode 100644 docs/img/new_ui/dev/open-api/api_test.png
create mode 100644 docs/img/new_ui/dev/open-api/create_project01.png
create mode 100644 docs/img/new_ui/dev/open-api/create_project02.png
create mode 100644 docs/img/new_ui/dev/open-api/create_project03.png
create mode 100644 docs/img/new_ui/dev/open-api/token_expiration.png
create mode 100644 docs/img/new_ui/dev/parameter/global_parameter01.png
create mode 100644 docs/img/new_ui/dev/parameter/global_parameter02.png
create mode 100644 docs/img/new_ui/dev/parameter/global_parameter03.png
delete mode 100644 docs/img/supplement_global_parameter.png
delete mode 100644 docs/img/supplement_global_parameter_en.png
create mode 100644 docs/img/tasks/demo/condition_task04.png
delete mode 100644 docs/img/test-api.png
delete mode 100644 docs/img/token-management-en.png
delete mode 100644 docs/img/token-management.png
[dolphinscheduler] 04/33: [Bug][Master] fix master task failover (#10065)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit aa51c66d9196997bfdfd6cb42be07311652fa0e9
Author: caishunfeng <ca...@gmail.com>
AuthorDate: Tue May 17 16:36:24 2022 +0800
[Bug][Master] fix master task failover (#10065)
* fix master task failover
* ui
(cherry picked from commit 0cc0ee77faabf1bed495a55ccd7653c03c26a550)
---
.../master/runner/task/TaskProcessorFactory.java | 9 ++
.../server/master/service/FailoverService.java | 63 ++++++++-----
.../server/master/service/FailoverServiceTest.java | 103 ++++++++++++++-------
3 files changed, 114 insertions(+), 61 deletions(-)
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/TaskProcessorFactory.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/TaskProcessorFactory.java
index 2b9e9d644a..0129338649 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/TaskProcessorFactory.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/TaskProcessorFactory.java
@@ -52,4 +52,13 @@ public class TaskProcessorFactory {
return iTaskProcessor.getClass().newInstance();
}
+
+ /**
+ * if match master processor, then this task type is processed on the master
+ * @param type
+ * @return
+ */
+ public static boolean isMasterTask(String type) {
+ return PROCESS_MAP.containsKey(type);
+ }
}
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/service/FailoverService.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/service/FailoverService.java
index 4f25c78dad..2557b659e0 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/service/FailoverService.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/service/FailoverService.java
@@ -30,6 +30,7 @@ import org.apache.dolphinscheduler.plugin.task.api.enums.ExecutionStatus;
import org.apache.dolphinscheduler.server.builder.TaskExecutionContextBuilder;
import org.apache.dolphinscheduler.server.master.config.MasterConfig;
import org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThreadPool;
+import org.apache.dolphinscheduler.server.master.runner.task.TaskProcessorFactory;
import org.apache.dolphinscheduler.server.utils.ProcessUtils;
import org.apache.dolphinscheduler.service.process.ProcessService;
import org.apache.dolphinscheduler.service.registry.RegistryClient;
@@ -127,7 +128,11 @@ public class FailoverService {
long startTime = System.currentTimeMillis();
List<ProcessInstance> needFailoverProcessInstanceList = processService.queryNeedFailoverProcessInstances(masterHost);
LOGGER.info("start master[{}] failover, process list size:{}", masterHost, needFailoverProcessInstanceList.size());
- List<Server> workerServers = registryClient.getServerList(NodeType.WORKER);
+
+ // servers need to contains master hosts and worker hosts, otherwise the logic task will failover fail.
+ List<Server> servers = registryClient.getServerList(NodeType.WORKER);
+ servers.addAll(registryClient.getServerList(NodeType.MASTER));
+
for (ProcessInstance processInstance : needFailoverProcessInstanceList) {
if (Constants.NULL.equals(processInstance.getHost())) {
continue;
@@ -136,7 +141,7 @@ public class FailoverService {
List<TaskInstance> validTaskInstanceList = processService.findValidTaskListByProcessId(processInstance.getId());
for (TaskInstance taskInstance : validTaskInstanceList) {
LOGGER.info("failover task instance id: {}, process instance id: {}", taskInstance.getId(), taskInstance.getProcessInstanceId());
- failoverTaskInstance(processInstance, taskInstance, workerServers);
+ failoverTaskInstance(processInstance, taskInstance, servers);
}
if (serverStartupTime != null && processInstance.getRestartTime() != null
@@ -198,29 +203,37 @@ public class FailoverService {
/**
* failover task instance
* <p>
- * 1. kill yarn job if there are yarn jobs in tasks.
+ * 1. kill yarn job if run on worker and there are yarn jobs in tasks.
* 2. change task state from running to need failover.
* 3. try to notify local master
+ * @param processInstance
+ * @param taskInstance
+ * @param servers if failover master, servers container master servers and worker servers; if failover worker, servers contain worker servers.
*/
- private void failoverTaskInstance(ProcessInstance processInstance, TaskInstance taskInstance, List<Server> workerServers) {
+ private void failoverTaskInstance(ProcessInstance processInstance, TaskInstance taskInstance, List<Server> servers) {
if (processInstance == null) {
LOGGER.error("failover task instance error, processInstance {} of taskInstance {} is null",
taskInstance.getProcessInstanceId(), taskInstance.getId());
return;
}
- if (!checkTaskInstanceNeedFailover(workerServers, taskInstance)) {
+ if (!checkTaskInstanceNeedFailover(servers, taskInstance)) {
return;
}
+ boolean isMasterTask = TaskProcessorFactory.isMasterTask(taskInstance.getTaskType());
+
taskInstance.setProcessInstance(processInstance);
- TaskExecutionContext taskExecutionContext = TaskExecutionContextBuilder.get()
- .buildTaskInstanceRelatedInfo(taskInstance)
- .buildProcessInstanceRelatedInfo(processInstance)
- .create();
-
- if (masterConfig.isKillYarnJobWhenTaskFailover()) {
- // only kill yarn job if exists , the local thread has exited
- ProcessUtils.killYarnJob(taskExecutionContext);
+
+ if (!isMasterTask) {
+ TaskExecutionContext taskExecutionContext = TaskExecutionContextBuilder.get()
+ .buildTaskInstanceRelatedInfo(taskInstance)
+ .buildProcessInstanceRelatedInfo(processInstance)
+ .create();
+
+ if (masterConfig.isKillYarnJobWhenTaskFailover()) {
+ // only kill yarn job if exists , the local thread has exited
+ ProcessUtils.killYarnJob(taskExecutionContext);
+ }
}
taskInstance.setState(ExecutionStatus.NEED_FAULT_TOLERANCE);
@@ -256,13 +269,13 @@ public class FailoverService {
}
/**
- * task needs failover if task start before worker starts
+ * task needs failover if task start before server starts
*
- * @param workerServers worker servers
+ * @param servers servers, can container master servers or worker servers
* @param taskInstance task instance
* @return true if task instance need fail over
*/
- private boolean checkTaskInstanceNeedFailover(List<Server> workerServers, TaskInstance taskInstance) {
+ private boolean checkTaskInstanceNeedFailover(List<Server> servers, TaskInstance taskInstance) {
boolean taskNeedFailover = true;
@@ -279,14 +292,13 @@ public class FailoverService {
return false;
}
-
//now no host will execute this task instance,so no need to failover the task
if (taskInstance.getHost() == null) {
return false;
}
- //if task start after worker starts, there is no need to failover the task.
- if (checkTaskAfterWorkerStart(workerServers, taskInstance)) {
+ //if task start after server starts, there is no need to failover the task.
+ if (checkTaskAfterServerStart(servers, taskInstance)) {
taskNeedFailover = false;
}
@@ -296,19 +308,20 @@ public class FailoverService {
/**
* check task start after the worker server starts.
*
+ * @param servers servers, can contain master servers or worker servers
* @param taskInstance task instance
- * @return true if task instance start time after worker server start date
+ * @return true if task instance start time after server start date
*/
- private boolean checkTaskAfterWorkerStart(List<Server> workerServers, TaskInstance taskInstance) {
+ private boolean checkTaskAfterServerStart(List<Server> servers, TaskInstance taskInstance) {
if (StringUtils.isEmpty(taskInstance.getHost())) {
return false;
}
- Date workerServerStartDate = getServerStartupTime(workerServers, taskInstance.getHost());
- if (workerServerStartDate != null) {
+ Date serverStartDate = getServerStartupTime(servers, taskInstance.getHost());
+ if (serverStartDate != null) {
if (taskInstance.getStartTime() == null) {
- return taskInstance.getSubmitTime().after(workerServerStartDate);
+ return taskInstance.getSubmitTime().after(serverStartDate);
} else {
- return taskInstance.getStartTime().after(workerServerStartDate);
+ return taskInstance.getStartTime().after(serverStartDate);
}
}
return false;
diff --git a/dolphinscheduler-master/src/test/java/org/apache/dolphinscheduler/server/master/service/FailoverServiceTest.java b/dolphinscheduler-master/src/test/java/org/apache/dolphinscheduler/server/master/service/FailoverServiceTest.java
index 17a3798090..4e3d99347a 100644
--- a/dolphinscheduler-master/src/test/java/org/apache/dolphinscheduler/server/master/service/FailoverServiceTest.java
+++ b/dolphinscheduler-master/src/test/java/org/apache/dolphinscheduler/server/master/service/FailoverServiceTest.java
@@ -17,6 +17,9 @@
package org.apache.dolphinscheduler.server.master.service;
+import static org.apache.dolphinscheduler.common.Constants.COMMON_TASK_TYPE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.TASK_TYPE_DEPENDENT;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.TASK_TYPE_SWITCH;
import static org.mockito.BDDMockito.given;
import static org.mockito.Mockito.doNothing;
@@ -30,9 +33,11 @@ import org.apache.dolphinscheduler.dao.entity.TaskInstance;
import org.apache.dolphinscheduler.plugin.task.api.enums.ExecutionStatus;
import org.apache.dolphinscheduler.server.master.config.MasterConfig;
import org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThreadPool;
+import org.apache.dolphinscheduler.service.bean.SpringApplicationContext;
import org.apache.dolphinscheduler.service.process.ProcessService;
import org.apache.dolphinscheduler.service.registry.RegistryClient;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.Date;
@@ -46,6 +51,7 @@ import org.mockito.Mockito;
import org.powermock.core.classloader.annotations.PowerMockIgnore;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
+import org.springframework.context.ApplicationContext;
import org.springframework.test.util.ReflectionTestUtils;
import com.google.common.collect.Lists;
@@ -72,22 +78,34 @@ public class FailoverServiceTest {
@Mock
private WorkflowExecuteThreadPool workflowExecuteThreadPool;
- private String testHost;
+ private static int masterPort = 5678;
+ private static int workerPort = 1234;
+
+ private String testMasterHost;
+ private String testWorkerHost;
private ProcessInstance processInstance;
- private TaskInstance taskInstance;
+ private TaskInstance masterTaskInstance;
+ private TaskInstance workerTaskInstance;
@Before
public void before() throws Exception {
- given(masterConfig.getListenPort()).willReturn(8080);
+ // init spring context
+ ApplicationContext applicationContext = Mockito.mock(ApplicationContext.class);
+ SpringApplicationContext springApplicationContext = new SpringApplicationContext();
+ springApplicationContext.setApplicationContext(applicationContext);
+
+ given(masterConfig.getListenPort()).willReturn(masterPort);
- testHost = failoverService.getLocalAddress();
- String ip = testHost.split(":")[0];
- int port = Integer.valueOf(testHost.split(":")[1]);
- Assert.assertEquals(8080, port);
+ testMasterHost = failoverService.getLocalAddress();
+ String ip = testMasterHost.split(":")[0];
+ int port = Integer.valueOf(testMasterHost.split(":")[1]);
+ Assert.assertEquals(masterPort, port);
+
+ testWorkerHost = ip + ":" + workerPort;
given(registryClient.getLock(Mockito.anyString())).willReturn(true);
given(registryClient.releaseLock(Mockito.anyString())).willReturn(true);
- given(registryClient.getHostByEventDataPath(Mockito.anyString())).willReturn(testHost);
+ given(registryClient.getHostByEventDataPath(Mockito.anyString())).willReturn(testMasterHost);
given(registryClient.getStoppable()).willReturn(cause -> {
});
given(registryClient.checkNodeExists(Mockito.anyString(), Mockito.any())).willReturn(true);
@@ -95,30 +113,43 @@ public class FailoverServiceTest {
processInstance = new ProcessInstance();
processInstance.setId(1);
- processInstance.setHost(testHost);
+ processInstance.setHost(testMasterHost);
processInstance.setRestartTime(new Date());
processInstance.setHistoryCmd("xxx");
processInstance.setCommandType(CommandType.STOP);
- taskInstance = new TaskInstance();
- taskInstance.setId(1);
- taskInstance.setStartTime(new Date());
- taskInstance.setHost(testHost);
+ masterTaskInstance = new TaskInstance();
+ masterTaskInstance.setId(1);
+ masterTaskInstance.setStartTime(new Date());
+ masterTaskInstance.setHost(testMasterHost);
+ masterTaskInstance.setTaskType(TASK_TYPE_SWITCH);
+
+ workerTaskInstance = new TaskInstance();
+ workerTaskInstance.setId(2);
+ workerTaskInstance.setStartTime(new Date());
+ workerTaskInstance.setHost(testWorkerHost);
+ workerTaskInstance.setTaskType(COMMON_TASK_TYPE);
- given(processService.queryNeedFailoverTaskInstances(Mockito.anyString())).willReturn(Arrays.asList(taskInstance));
- given(processService.queryNeedFailoverProcessInstanceHost()).willReturn(Lists.newArrayList(testHost));
+ given(processService.queryNeedFailoverTaskInstances(Mockito.anyString())).willReturn(Arrays.asList(masterTaskInstance, workerTaskInstance));
+ given(processService.queryNeedFailoverProcessInstanceHost()).willReturn(Lists.newArrayList(testMasterHost));
given(processService.queryNeedFailoverProcessInstances(Mockito.anyString())).willReturn(Arrays.asList(processInstance));
doNothing().when(processService).processNeedFailoverProcessInstances(Mockito.any(ProcessInstance.class));
- given(processService.findValidTaskListByProcessId(Mockito.anyInt())).willReturn(Lists.newArrayList(taskInstance));
+ given(processService.findValidTaskListByProcessId(Mockito.anyInt())).willReturn(Lists.newArrayList(masterTaskInstance, workerTaskInstance));
given(processService.findProcessInstanceDetailById(Mockito.anyInt())).willReturn(processInstance);
Thread.sleep(1000);
- Server server = new Server();
- server.setHost(ip);
- server.setPort(port);
- server.setCreateTime(new Date());
- given(registryClient.getServerList(NodeType.WORKER)).willReturn(Arrays.asList(server));
- given(registryClient.getServerList(NodeType.MASTER)).willReturn(Arrays.asList(server));
+ Server masterServer = new Server();
+ masterServer.setHost(ip);
+ masterServer.setPort(masterPort);
+ masterServer.setCreateTime(new Date());
+
+ Server workerServer = new Server();
+ workerServer.setHost(ip);
+ workerServer.setPort(workerPort);
+ workerServer.setCreateTime(new Date());
+
+ given(registryClient.getServerList(NodeType.WORKER)).willReturn(new ArrayList<>(Arrays.asList(workerServer)));
+ given(registryClient.getServerList(NodeType.MASTER)).willReturn(new ArrayList<>(Arrays.asList(masterServer)));
ReflectionTestUtils.setField(failoverService, "registryClient", registryClient);
doNothing().when(workflowExecuteThreadPool).submitStateEvent(Mockito.any(StateEvent.class));
@@ -132,26 +163,26 @@ public class FailoverServiceTest {
@Test
public void failoverMasterTest() {
processInstance.setHost(Constants.NULL);
- taskInstance.setState(ExecutionStatus.RUNNING_EXECUTION);
- failoverService.failoverServerWhenDown(testHost, NodeType.MASTER);
- Assert.assertNotEquals(taskInstance.getState(), ExecutionStatus.NEED_FAULT_TOLERANCE);
-
- processInstance.setHost(testHost);
- taskInstance.setState(ExecutionStatus.SUCCESS);
- failoverService.failoverServerWhenDown(testHost, NodeType.MASTER);
- Assert.assertNotEquals(taskInstance.getState(), ExecutionStatus.NEED_FAULT_TOLERANCE);
+ masterTaskInstance.setState(ExecutionStatus.RUNNING_EXECUTION);
+ failoverService.failoverServerWhenDown(testMasterHost, NodeType.MASTER);
+ Assert.assertNotEquals(masterTaskInstance.getState(), ExecutionStatus.NEED_FAULT_TOLERANCE);
+
+ processInstance.setHost(testMasterHost);
+ masterTaskInstance.setState(ExecutionStatus.SUCCESS);
+ failoverService.failoverServerWhenDown(testMasterHost, NodeType.MASTER);
+ Assert.assertNotEquals(masterTaskInstance.getState(), ExecutionStatus.NEED_FAULT_TOLERANCE);
Assert.assertEquals(Constants.NULL, processInstance.getHost());
- processInstance.setHost(testHost);
- taskInstance.setState(ExecutionStatus.RUNNING_EXECUTION);
- failoverService.failoverServerWhenDown(testHost, NodeType.MASTER);
- Assert.assertEquals(taskInstance.getState(), ExecutionStatus.NEED_FAULT_TOLERANCE);
+ processInstance.setHost(testMasterHost);
+ masterTaskInstance.setState(ExecutionStatus.RUNNING_EXECUTION);
+ failoverService.failoverServerWhenDown(testMasterHost, NodeType.MASTER);
+ Assert.assertEquals(masterTaskInstance.getState(), ExecutionStatus.NEED_FAULT_TOLERANCE);
Assert.assertEquals(Constants.NULL, processInstance.getHost());
}
@Test
public void failoverWorkTest() {
- failoverService.failoverServerWhenDown(testHost, NodeType.WORKER);
- Assert.assertEquals(taskInstance.getState(), ExecutionStatus.NEED_FAULT_TOLERANCE);
+ failoverService.failoverServerWhenDown(testWorkerHost, NodeType.WORKER);
+ Assert.assertEquals(workerTaskInstance.getState(), ExecutionStatus.NEED_FAULT_TOLERANCE);
}
}
[dolphinscheduler] 28/33: [Fix-10057][api] api parameter name misspelling (#10063)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit e1ff8b87c6602db814321ff637d8339ebbf58e0d
Author: HeChuan <ti...@163.com>
AuthorDate: Sat May 21 12:25:32 2022 +0800
[Fix-10057][api] api parameter name misspelling (#10063)
correct parameter name from processDefiniteCode to processDefineCode
(cherry picked from commit 2251876021dea09786c418dfb1325366358a0601)
---
.../dolphinscheduler/api/controller/ProcessInstanceController.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessInstanceController.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessInstanceController.java
index aee4740d5c..2484d9ec7b 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessInstanceController.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProcessInstanceController.java
@@ -97,7 +97,7 @@ public class ProcessInstanceController extends BaseController {
*/
@ApiOperation(value = "queryProcessInstanceListPaging", notes = "QUERY_PROCESS_INSTANCE_LIST_NOTES")
@ApiImplicitParams({
- @ApiImplicitParam(name = "processDefiniteCode", value = "PROCESS_DEFINITION_CODE", dataType = "Long", example = "100"),
+ @ApiImplicitParam(name = "processDefineCode", value = "PROCESS_DEFINITION_CODE", dataType = "Long", example = "100"),
@ApiImplicitParam(name = "searchVal", value = "SEARCH_VAL", type = "String"),
@ApiImplicitParam(name = "executorName", value = "EXECUTOR_NAME", type = "String"),
@ApiImplicitParam(name = "stateType", value = "EXECUTION_STATUS", type = "ExecutionStatus"),
[dolphinscheduler] 27/33: [Fix][UI] Fix the problem that the failure strategy field in the time management form is multilingual. (#10169)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit bbf621cb059e3c0515ffc0ee8af0f3707873f47c
Author: songjianet <17...@qq.com>
AuthorDate: Fri May 20 18:29:40 2022 +0800
[Fix][UI] Fix the problem that the failure strategy field in the time management form is multilingual. (#10169)
(cherry picked from commit dc61370f302c6f2e8d70f50573818dc057ef2827)
---
.../src/views/projects/workflow/definition/timing/use-table.ts | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/dolphinscheduler-ui/src/views/projects/workflow/definition/timing/use-table.ts b/dolphinscheduler-ui/src/views/projects/workflow/definition/timing/use-table.ts
index 5219a21af4..39f08725e3 100644
--- a/dolphinscheduler-ui/src/views/projects/workflow/definition/timing/use-table.ts
+++ b/dolphinscheduler-ui/src/views/projects/workflow/definition/timing/use-table.ts
@@ -106,6 +106,13 @@ export function useTable() {
{
title: t('project.workflow.failure_strategy'),
key: 'failureStrategy',
+ render: (row: any) => {
+ if (row.failureStrategy === 'CONTINUE') {
+ return t('project.workflow.continue')
+ } else if (row.failureStrategy === 'END') {
+ return t('project.workflow.end')
+ }
+ },
width: 140
},
{
[dolphinscheduler] 32/33: [doc] Refactor global parameter docment (#10045)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit e737d8ee30cfaf38cb8d2285ac033b3eb405cce6
Author: QuakeWang <45...@users.noreply.github.com>
AuthorDate: Mon May 23 10:46:11 2022 +0800
[doc] Refactor global parameter docment (#10045)
(cherry picked from commit 0abfb5e74485d6658d8db33d92549d1f469ab752)
---
docs/docs/en/guide/parameter/global.md | 28 +++++++++++++-------
docs/docs/zh/guide/parameter/global.md | 29 ++++++++++++++-------
docs/img/local_parameter.png | Bin 42275 -> 0 bytes
docs/img/local_parameter_en.png | Bin 72458 -> 0 bytes
.../new_ui/dev/parameter/global_parameter01.png | Bin 0 -> 148147 bytes
.../new_ui/dev/parameter/global_parameter02.png | Bin 0 -> 141585 bytes
.../new_ui/dev/parameter/global_parameter03.png | Bin 0 -> 312613 bytes
docs/img/supplement_global_parameter.png | Bin 295951 -> 0 bytes
docs/img/supplement_global_parameter_en.png | Bin 124936 -> 0 bytes
9 files changed, 39 insertions(+), 18 deletions(-)
diff --git a/docs/docs/en/guide/parameter/global.md b/docs/docs/en/guide/parameter/global.md
index bd4b25d47b..882b1098d2 100644
--- a/docs/docs/en/guide/parameter/global.md
+++ b/docs/docs/en/guide/parameter/global.md
@@ -2,18 +2,28 @@
## Scope
-The parameters defined on the process definition page can apply to all the scope of the process tasks.
+Global parameters are parameters that are valid for all task nodes of the entire workflow. It can be configured on the workflow definition page.
## Usage
-Usage of global parameters is: at the process define page, click the '+' beside the 'Set global' and fill in the key and value to save:
+The specific use method can be determined according to the actual production situation. This example uses a shell task to print out the date value of yesterday.
-<p align="center">
- <img src="/img/supplement_global_parameter_en.png" width="80%" />
- </p>
+### Create a Shell task
-<p align="center">
- <img src="/img/local_parameter_en.png" width="80%" />
- </p>
+Create a shell task and enter `echo ${dt}` in the script content. In this case, dt is the global parameter we need to declare. As shown below:
-The `global_bizdate` parameter defined here can be referenced by local parameters of any other task node, and set the value of `global_bizdate` by referencing the system parameter `system.biz.date`.
\ No newline at end of file
+
+
+### Save the workflow and set global parameters
+
+You could follow this guide to set global parameter: On the workflow definition page, click the plus sign to the right of "Set Global", after filling in the variable name and value, then save it
+
+
+
+> Note: The dt parameter defined here can be referenced by the local parameters of any other node.
+
+### In task instance view execution result
+
+On the task instance page, you can check the log to verify the execution result of the task and determine whether the parameters are valid.
+
+
diff --git a/docs/docs/zh/guide/parameter/global.md b/docs/docs/zh/guide/parameter/global.md
index 57716d8afe..3d91c165ff 100644
--- a/docs/docs/zh/guide/parameter/global.md
+++ b/docs/docs/zh/guide/parameter/global.md
@@ -2,18 +2,29 @@
## 作用域
-在工作流定义页面配置的参数，作用于该工作流中全部的任务
+全局参数是指针对**整个工作流**的所有任务节点都有效的参数，在工作流定义页面配置。
## 使用方式
-全局参数配置方式如下：在工作流定义页面，点击“设置全局”右边的加号，填写对应的变量名称和对应的值，保存即可
+具体的使用方式可结合实际的生产情况而定，这里演示为使用 Shell 任务打印出前一天的日期。
-<p align="center">
- <img src="/img/supplement_global_parameter.png" width="80%" />
-</p>
+### 创建 Shell 任务
-<p align="center">
- <img src="/img/local_parameter.png" width="80%" />
-</p>
+创建一个 Shell 任务，并在脚本内容中输入 `echo ${dt}`。此时 dt 则为我们需要声明的全局参数。如下图所示：
+
+
+
+### 保存工作流，并设置全局参数
+
+全局参数配置方式如下：在工作流定义页面，点击“设置全局”右边的加号，填写对应的变量名称和对应的值，保存即可。如下图所示：
+
+
+
+> 注：这里定义的 dt 参数可以被其它任一节点的局部参数引用。
+
+### 任务实例查看执行结果
+
+进入任务实例页面，可以通过查看日志，验证任务的执行结果，判断参数是否有效。
+
+
-这里定义的global_bizdate参数可以被其它任一节点的局部参数引用，并设置global_bizdate的value为通过引用系统参数system.biz.date获得的值
diff --git a/docs/img/local_parameter.png b/docs/img/local_parameter.png
deleted file mode 100644
index 5bc6c097c6..0000000000
Binary files a/docs/img/local_parameter.png and /dev/null differ
diff --git a/docs/img/local_parameter_en.png b/docs/img/local_parameter_en.png
deleted file mode 100644
index 2d72f03cb8..0000000000
Binary files a/docs/img/local_parameter_en.png and /dev/null differ
diff --git a/docs/img/new_ui/dev/parameter/global_parameter01.png b/docs/img/new_ui/dev/parameter/global_parameter01.png
new file mode 100644
index 0000000000..f2d6c0303c
Binary files /dev/null and b/docs/img/new_ui/dev/parameter/global_parameter01.png differ
diff --git a/docs/img/new_ui/dev/parameter/global_parameter02.png b/docs/img/new_ui/dev/parameter/global_parameter02.png
new file mode 100644
index 0000000000..326f4954a2
Binary files /dev/null and b/docs/img/new_ui/dev/parameter/global_parameter02.png differ
diff --git a/docs/img/new_ui/dev/parameter/global_parameter03.png b/docs/img/new_ui/dev/parameter/global_parameter03.png
new file mode 100644
index 0000000000..2308b6a752
Binary files /dev/null and b/docs/img/new_ui/dev/parameter/global_parameter03.png differ
diff --git a/docs/img/supplement_global_parameter.png b/docs/img/supplement_global_parameter.png
deleted file mode 100644
index 2fd7d17cc6..0000000000
Binary files a/docs/img/supplement_global_parameter.png and /dev/null differ
diff --git a/docs/img/supplement_global_parameter_en.png b/docs/img/supplement_global_parameter_en.png
deleted file mode 100644
index f026f71e2e..0000000000
Binary files a/docs/img/supplement_global_parameter_en.png and /dev/null differ
[dolphinscheduler] 13/33: [fix] Enhance name pre checker in resource center (#10094)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit cbfeb7f7777e846346e4b46897cd32071ea1a9ef
Author: Jiajie Zhong <zh...@hotmail.com>
AuthorDate: Wed May 18 18:40:17 2022 +0800
[fix] Enhance name pre checker in resource center (#10094)
* [fix] Enhance name pre checker in resource center
Add file name and directory checker to avoid directory traversal
* add some missing change and change docs
* change var name in directoryTraversal
* Fix ci
(cherry picked from commit 63f835715f8ca8bff79c0e7177ebfa5917ebb3bd)
---
docs/docs/en/guide/resource/file-manage.md | 31 +++++++++++++---------
docs/docs/zh/guide/resource/file-manage.md | 31 +++++++++++++---------
.../apache/dolphinscheduler/api/enums/Status.java | 2 +-
.../api/service/impl/ResourcesServiceImpl.java | 21 +++++++++++++--
.../dolphinscheduler/common/utils/FileUtils.java | 21 +++++++++++++++
.../common/utils/FileUtilsTest.java | 30 +++++++++++++++++++++
6 files changed, 108 insertions(+), 28 deletions(-)
diff --git a/docs/docs/en/guide/resource/file-manage.md b/docs/docs/en/guide/resource/file-manage.md
index ae3e8d2ec4..87b965801b 100644
--- a/docs/docs/en/guide/resource/file-manage.md
+++ b/docs/docs/en/guide/resource/file-manage.md
@@ -4,40 +4,47 @@ When third party jars are used in the scheduling process or user defined scripts
-- Create a file
- > The file format supports the following types: txt, log, sh, conf, cfg, py, java, sql, xml, hql, properties.
+## Basic Operator
+
+### Create a File
+
+The file format supports the following types: txt, log, sh, conf, cfg, py, java, sql, xml, hql, properties.
-- upload files
+### Upload Files
-> Upload file: Click the "Upload File" button to upload, drag the file to the upload area, the file name will be automatically completed with the uploaded file name.
+Click the "Upload File" button to upload, drag the file to the upload area, the file name will be automatically completed with the uploaded file name.
-- File View
+### View File Content
-> For the files that can be viewed, click the file name to view the file details.
+ For the files that can be viewed, click the file name to view the file details.
-- Download file
+### Download file
> Click the "Download" button in the file list to download the file or click the "Download" button in the upper right corner of the file details to download the file.
-- File rename
+### Rename File
-- delete
- > File list -> Click the "Delete" button to delete the specified file.
+### Delete File
-- Re-upload file
+File list -> Click the "Delete" button to delete the specified file.
- > Re-upload file: Click the "Re-upload File" button to upload a new file to replace the old file, drag the file to the re-upload area, the file name will be automatically completed with the new uploaded file name.
+### Re-upload file
+
+Click the "Re-upload File" button to upload a new file to replace the old file, drag the file to the re-upload area, the file name will be automatically completed with the new uploaded file name.
+> Note: File name or source name of your local file can not contain specific characters like `.` or `/` when you trying to
+> upload, create or rename file in resource center.
+
## Example
The example uses a simple shell script to demonstrate the use of resource center files in workflow definitions. The same is true for tasks such as MR and Spark, which require jar packages.
diff --git a/docs/docs/zh/guide/resource/file-manage.md b/docs/docs/zh/guide/resource/file-manage.md
index 77d9509548..a0d2f0ab56 100644
--- a/docs/docs/zh/guide/resource/file-manage.md
+++ b/docs/docs/zh/guide/resource/file-manage.md
@@ -2,43 +2,48 @@
当在调度过程中需要使用到第三方的 jar 或者用户需要自定义脚本的情况，可以通过在该页面完成相关操作。可创建的文件类型包括：`txt/log/sh/conf/py/java` 等。并且可以对文件进行编辑、重命名、下载和删除等操作。
+## 基础操作
+
-* 创建文件
+### 创建文件
- > 文件格式支持以下几种类型：txt、log、sh、conf、cfg、py、java、sql、xml、hql、properties
+文件格式支持以下几种类型：txt、log、sh、conf、cfg、py、java、sql、xml、hql、properties
-* 上传文件
- > 上传文件：点击"上传文件"按钮进行上传，将文件拖拽到上传区域，文件名会自动以上传的文件名称补全
+### 上传文件
+
+上传文件：点击"上传文件"按钮进行上传，将文件拖拽到上传区域，文件名会自动以上传的文件名称补全
-* 文件查看
+### 文件查看
- > 对可查看的文件类型，点击文件名称，可查看文件详情
+对可查看的文件类型，点击文件名称，可查看文件详情
-* 下载文件
+### 下载文件
- > 点击文件列表的"下载"按钮下载文件或者在文件详情中点击右上角"下载"按钮下载文件
+点击文件列表的"下载"按钮下载文件或者在文件详情中点击右上角"下载"按钮下载文件
-* 文件重命名
+### 文件重命名
-* 删除
+### 删除文件
-> 文件列表->点击"删除"按钮，删除指定文件
+文件列表->点击"删除"按钮，删除指定文件
-* 重新上传文件
+### 重新上传文件
- > 点击文件列表中的”重新上传文件“按钮进行重新上传文件，将文件拖拽到上传区域，文件名会自动以上传的文件名称补全
+点击文件列表中的”重新上传文件“按钮进行重新上传文件，将文件拖拽到上传区域，文件名会自动以上传的文件名称补全
+> 注意：上传、创建、重命名文件时，文件名和源文件名（上传时）均不能带有 `.` 以及 `/` 特殊符号。
+
## 任务样例
该样例主要通过一个简单的 shell 脚本，来演示如何在工作流定义中使用资源中心的文件。像 MR、Spark 等任务需要用到 jar 包，也是同理。
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
index b315c313cb..0847d6920d 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
@@ -395,7 +395,7 @@ public enum Status {
K8S_CLIENT_OPS_ERROR(1300006, "k8s error with exception {0}", "k8s操作报错[{0}]"),
VERIFY_K8S_NAMESPACE_ERROR(1300007, "verify k8s and namespace error", "验证k8s命名空间信息错误"),
DELETE_K8S_NAMESPACE_BY_ID_ERROR(1300008, "delete k8s namespace by id error", "删除命名空间错误"),
- VERIFY_PARAMETER_NAME_FAILED(1300009, "The file name verify failed", "文件命名校验失败"),
+ VERIFY_PARAMETER_NAME_FAILED(1300009, "The file name verify failed", "文件命名校验失败"),
STORE_OPERATE_CREATE_ERROR(1300010, "create the resource failed", "存储操作失败"),
GRANT_K8S_NAMESPACE_ERROR(1300011, "grant namespace error", "授权资源错误"),
QUERY_UNAUTHORIZED_NAMESPACE_ERROR(1300012, "query unauthorized namespace error", "查询未授权命名空间错误"),
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
index 200aeb1c65..1c5804cd81 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ResourcesServiceImpl.java
@@ -129,8 +129,8 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
if (!result.getCode().equals(Status.SUCCESS.getCode())) {
return result;
}
- if (name.endsWith(FOLDER_SEPARATOR)) {
- result.setCode(Status.VERIFY_PARAMETER_NAME_FAILED.getCode());
+ if (FileUtils.directoryTraversal(name)) {
+ putMsg(result, Status.VERIFY_PARAMETER_NAME_FAILED);
return result;
}
String fullName = getFullName(currentDir, name);
@@ -517,6 +517,19 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
private Result<Object> verifyFile(String name, ResourceType type, MultipartFile file) {
Result<Object> result = new Result<>();
putMsg(result, Status.SUCCESS);
+
+ if (FileUtils.directoryTraversal(name)) {
+ logger.error("file alias name {} verify failed", name);
+ putMsg(result, Status.VERIFY_PARAMETER_NAME_FAILED);
+ return result;
+ }
+
+ if (file != null && FileUtils.directoryTraversal(Objects.requireNonNull(file.getOriginalFilename()))) {
+ logger.error("file original name {} verify failed", file.getOriginalFilename());
+ putMsg(result, Status.VERIFY_PARAMETER_NAME_FAILED);
+ return result;
+ }
+
if (file != null) {
// file is empty
if (file.isEmpty()) {
@@ -973,6 +986,10 @@ public class ResourcesServiceImpl extends BaseServiceImpl implements ResourcesSe
if (!result.getCode().equals(Status.SUCCESS.getCode())) {
return result;
}
+ if (FileUtils.directoryTraversal(fileName)) {
+ putMsg(result, Status.VERIFY_PARAMETER_NAME_FAILED);
+ return result;
+ }
//check file suffix
String nameSuffix = fileSuffix.trim();
diff --git a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/FileUtils.java b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/FileUtils.java
index e7817ffe80..bdcf62f76a 100644
--- a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/FileUtils.java
+++ b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/FileUtils.java
@@ -204,4 +204,25 @@ public class FileUtils {
}
}
+ /**
+ * Check whether the given string type of path can be traversal or not, return true if path could
+ * traversal, and return false if it is not.
+ *
+ * @param filename String type of filename
+ * @return whether file path could be traversal or not
+ */
+ public static boolean directoryTraversal(String filename){
+ if (filename.contains(FOLDER_SEPARATOR)) {
+ return true;
+ }
+ File file = new File(filename);
+ try {
+ File canonical = file.getCanonicalFile();
+ File absolute = file.getAbsoluteFile();
+ return !canonical.equals(absolute);
+ } catch (IOException e) {
+ return true;
+ }
+ }
+
}
diff --git a/dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/FileUtilsTest.java b/dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/FileUtilsTest.java
index d023b8651f..fdcaccd697 100644
--- a/dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/FileUtilsTest.java
+++ b/dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/FileUtilsTest.java
@@ -88,4 +88,34 @@ public class FileUtilsTest {
Assert.assertEquals(content, fileContent);
}
+ @Test
+ public void testDirectoryTraversal() {
+ // test case which do not directory traversal
+ String path;
+ path = "abc.txt";
+ Assert.assertFalse(FileUtils.directoryTraversal(path));
+
+ path = "abc...txt";
+ Assert.assertFalse(FileUtils.directoryTraversal(path));
+
+ path = "..abc.txt";
+ Assert.assertFalse(FileUtils.directoryTraversal(path));
+
+ // test case which will directory traversal
+ path = "../abc.txt";
+ Assert.assertTrue(FileUtils.directoryTraversal(path));
+
+ path = "../../abc.txt";
+ Assert.assertTrue(FileUtils.directoryTraversal(path));
+
+ path = "abc../def.txt";
+ Assert.assertTrue(FileUtils.directoryTraversal(path));
+
+ path = "abc./def.txt";
+ Assert.assertTrue(FileUtils.directoryTraversal(path));
+
+ path = "abc/def...txt";
+ Assert.assertTrue(FileUtils.directoryTraversal(path));
+ }
+
}
[dolphinscheduler] 14/33: [BUG][TaskGroup] Task group does not take effect (#10093)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit b016037a6f75686898b4a831be61fe11c5769e29
Author: BaoLiang <29...@users.noreply.github.com>
AuthorDate: Wed May 18 18:40:36 2022 +0800
[BUG][TaskGroup] Task group does not take effect (#10093)
* fix 10092: Task group does not take effect
* fix 10092: Task group does not take effect
* fix 10092: Task group does not take effect
(cherry picked from commit ee2b855ced2e46bcdaf3ae7068941ac7e1c03c17)
---
.../apache/dolphinscheduler/common/Constants.java | 4 ++
.../dolphinscheduler/dao/entity/TaskInstance.java | 2 +
.../master/runner/task/CommonTaskProcessor.java | 2 +-
.../consumer/TaskPriorityQueueConsumerTest.java | 12 ++---
.../queue/PeerTaskInstancePriorityQueue.java | 5 ++
.../service/queue/TaskPriority.java | 33 ++++++++++--
.../queue/PeerTaskInstancePriorityQueueTest.java | 57 ++++++++++++++++----
.../service/queue/TaskPriorityQueueImplTest.java | 62 ++++++++++++++++------
8 files changed, 140 insertions(+), 37 deletions(-)
diff --git a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
index 57b5c92970..19fbdb90ca 100644
--- a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
+++ b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
@@ -66,6 +66,8 @@ public final class Constants {
public static final String BUCKET_NAME = "dolphinscheduler-test";
+ public static final String EMPTY_STRING = "";
+
/**
* fs.defaultFS
*/
@@ -422,6 +424,8 @@ public final class Constants {
*/
public static final int DEFINITION_FAILURE = -1;
+ public static final int OPPOSITE_VALUE = -1;
+
/**
* process or task definition first version
*/
diff --git a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/TaskInstance.java b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/TaskInstance.java
index 6ab0c340f1..4ec1dfde2b 100644
--- a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/TaskInstance.java
+++ b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/TaskInstance.java
@@ -713,6 +713,8 @@ public class TaskInstance implements Serializable {
+ ", executorName='" + executorName + '\''
+ ", delayTime=" + delayTime
+ ", dryRun=" + dryRun
+ + ", taskGroupId=" + taskGroupId
+ + ", taskGroupPriority=" + taskGroupPriority
+ '}';
}
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/CommonTaskProcessor.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/CommonTaskProcessor.java
index 5833bc56c1..ffeb89a0d2 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/CommonTaskProcessor.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/CommonTaskProcessor.java
@@ -114,7 +114,7 @@ public class CommonTaskProcessor extends BaseTaskProcessor {
TaskPriority taskPriority = new TaskPriority(processInstance.getProcessInstancePriority().getCode(),
processInstance.getId(), taskInstance.getProcessInstancePriority().getCode(),
- taskInstance.getId(), org.apache.dolphinscheduler.common.Constants.DEFAULT_WORKER_GROUP);
+ taskInstance.getId(), taskInstance.getTaskGroupPriority(), org.apache.dolphinscheduler.common.Constants.DEFAULT_WORKER_GROUP);
TaskExecutionContext taskExecutionContext = getTaskExecutionContext(taskInstance);
if (taskExecutionContext == null) {
diff --git a/dolphinscheduler-master/src/test/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumerTest.java b/dolphinscheduler-master/src/test/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumerTest.java
index ec3f58aca0..05590e4541 100644
--- a/dolphinscheduler-master/src/test/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumerTest.java
+++ b/dolphinscheduler-master/src/test/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumerTest.java
@@ -98,7 +98,7 @@ public class TaskPriorityQueueConsumerTest {
processDefinition.setUserId(2);
taskInstance.setProcessDefine(processDefinition);
- TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, "default");
+ TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, 1, "default");
taskPriorityQueue.put(taskPriority);
TimeUnit.SECONDS.sleep(10);
@@ -125,7 +125,7 @@ public class TaskPriorityQueueConsumerTest {
ProcessDefinition processDefinition = new ProcessDefinition();
processDefinition.setUserId(2);
taskInstance.setProcessDefine(processDefinition);
- TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, "default");
+ TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, 1, "default");
taskPriorityQueue.put(taskPriority);
DataSource dataSource = new DataSource();
@@ -166,7 +166,7 @@ public class TaskPriorityQueueConsumerTest {
ProcessDefinition processDefinition = new ProcessDefinition();
processDefinition.setUserId(2);
taskInstance.setProcessDefine(processDefinition);
- TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, "default");
+ TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, 1, "default");
taskPriorityQueue.put(taskPriority);
DataSource dataSource = new DataSource();
@@ -205,7 +205,7 @@ public class TaskPriorityQueueConsumerTest {
ProcessDefinition processDefinition = new ProcessDefinition();
processDefinition.setUserId(2);
taskInstance.setProcessDefine(processDefinition);
- TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, "default");
+ TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, 1, "default");
taskPriorityQueue.put(taskPriority);
DataSource dataSource = new DataSource();
@@ -266,7 +266,7 @@ public class TaskPriorityQueueConsumerTest {
Mockito.doReturn(taskInstance).when(processService).findTaskInstanceById(1);
- TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, "NoWorkGroup");
+ TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, 1, "NoWorkGroup");
taskPriorityQueue.put(taskPriority);
TimeUnit.SECONDS.sleep(10);
@@ -335,7 +335,7 @@ public class TaskPriorityQueueConsumerTest {
Mockito.doReturn(taskInstance).when(processService).findTaskInstanceById(1);
- TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, "NoWorkGroup");
+ TaskPriority taskPriority = new TaskPriority(2, 1, 2, 1, 1, "NoWorkGroup");
taskPriorityQueue.put(taskPriority);
taskPriorityQueueConsumer.run();
diff --git a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/PeerTaskInstancePriorityQueue.java b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/PeerTaskInstancePriorityQueue.java
index 7502607bcf..231fd2a20f 100644
--- a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/PeerTaskInstancePriorityQueue.java
+++ b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/PeerTaskInstancePriorityQueue.java
@@ -17,6 +17,7 @@
package org.apache.dolphinscheduler.service.queue;
+import org.apache.dolphinscheduler.common.Constants;
import org.apache.dolphinscheduler.dao.entity.TaskInstance;
import org.apache.dolphinscheduler.service.exceptions.TaskPriorityQueueException;
@@ -168,6 +169,10 @@ public class PeerTaskInstancePriorityQueue implements TaskPriorityQueue<TaskInst
*/
@Override
public int compare(TaskInstance o1, TaskInstance o2) {
+ if(o1.getTaskInstancePriority().equals(o2.getTaskInstancePriority())){
+ // larger number, higher priority
+ return Constants.OPPOSITE_VALUE * Integer.compare(o1.getTaskGroupPriority(),o2.getTaskGroupPriority());
+ }
return o1.getTaskInstancePriority().compareTo(o2.getTaskInstancePriority());
}
}
diff --git a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/TaskPriority.java b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/TaskPriority.java
index 12a7258d6c..0aeec4609d 100644
--- a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/TaskPriority.java
+++ b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/TaskPriority.java
@@ -17,7 +17,9 @@
package org.apache.dolphinscheduler.service.queue;
+import org.apache.dolphinscheduler.common.Constants;
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
import java.util.Map;
import java.util.Objects;
@@ -67,6 +69,8 @@ public class TaskPriority implements Comparable<TaskPriority> {
*/
private long checkpoint;
+ private int taskGroupPriority;
+
public TaskPriority() {
this.checkpoint = System.currentTimeMillis();
}
@@ -74,11 +78,13 @@ public class TaskPriority implements Comparable<TaskPriority> {
public TaskPriority(int processInstancePriority,
int processInstanceId,
int taskInstancePriority,
- int taskId, String groupName) {
+ int taskId,
+ int taskGroupPriority, String groupName) {
this.processInstancePriority = processInstancePriority;
this.processInstanceId = processInstanceId;
this.taskInstancePriority = taskInstancePriority;
this.taskId = taskId;
+ this.taskGroupPriority = taskGroupPriority;
this.groupName = groupName;
this.checkpoint = System.currentTimeMillis();
}
@@ -147,6 +153,14 @@ public class TaskPriority implements Comparable<TaskPriority> {
this.checkpoint = checkpoint;
}
+ public int getTaskGroupPriority() {
+ return taskGroupPriority;
+ }
+
+ public void setTaskGroupPriority(int taskGroupPriority) {
+ this.taskGroupPriority = taskGroupPriority;
+ }
+
@Override
public int compareTo(TaskPriority other) {
if (this.getProcessInstancePriority() > other.getProcessInstancePriority()) {
@@ -169,15 +183,22 @@ public class TaskPriority implements Comparable<TaskPriority> {
if (this.getTaskInstancePriority() < other.getTaskInstancePriority()) {
return -1;
}
-
+ if(this.getTaskGroupPriority() != other.getTaskGroupPriority()){
+ // larger number, higher priority
+ return Constants.OPPOSITE_VALUE * Integer.compare(this.getTaskGroupPriority(), other.getTaskGroupPriority());
+ }
if (this.getTaskId() > other.getTaskId()) {
return 1;
}
if (this.getTaskId() < other.getTaskId()) {
return -1;
}
-
- return this.getGroupName().compareTo(other.getGroupName());
+ String thisGroupName = StringUtils.isNotBlank(this.getGroupName()) ? this.getGroupName() : Constants.EMPTY_STRING;
+ String otherGroupName = StringUtils.isNotBlank(other.getGroupName()) ? other.getGroupName() : Constants.EMPTY_STRING;
+ if(!thisGroupName.equals(otherGroupName)){
+ return thisGroupName.compareTo(otherGroupName);
+ }
+ return Long.compare(this.getCheckpoint(), other.getCheckpoint());
}
@Override
@@ -193,11 +214,13 @@ public class TaskPriority implements Comparable<TaskPriority> {
&& processInstanceId == that.processInstanceId
&& taskInstancePriority == that.taskInstancePriority
&& taskId == that.taskId
+ && taskGroupPriority == that.taskGroupPriority
&& Objects.equals(groupName, that.groupName);
}
@Override
public int hashCode() {
- return Objects.hash(processInstancePriority, processInstanceId, taskInstancePriority, taskId, groupName);
+ return Objects.hash(processInstancePriority, processInstanceId, taskInstancePriority, taskId, taskGroupPriority, groupName);
}
+
}
diff --git a/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/PeerTaskInstancePriorityQueueTest.java b/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/PeerTaskInstancePriorityQueueTest.java
index 7feb4c676b..8da3a6c194 100644
--- a/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/PeerTaskInstancePriorityQueueTest.java
+++ b/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/PeerTaskInstancePriorityQueueTest.java
@@ -31,8 +31,8 @@ public class PeerTaskInstancePriorityQueueTest {
@Test
public void put() throws TaskPriorityQueueException {
PeerTaskInstancePriorityQueue queue = new PeerTaskInstancePriorityQueue();
- TaskInstance taskInstanceHigPriority = createTaskInstance("high", Priority.HIGH);
- TaskInstance taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM);
+ TaskInstance taskInstanceHigPriority = createTaskInstance("high", Priority.HIGH, 1);
+ TaskInstance taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM, 1);
queue.put(taskInstanceHigPriority);
queue.put(taskInstanceMediumPriority);
Assert.assertEquals(2, queue.size());
@@ -60,10 +60,46 @@ public class PeerTaskInstancePriorityQueueTest {
public void peek() throws Exception {
PeerTaskInstancePriorityQueue queue = getPeerTaskInstancePriorityQueue();
int peekBeforeLength = queue.size();
- queue.peek();
Assert.assertEquals(peekBeforeLength, queue.size());
}
+ @Test
+ public void peekTaskGroupPriority() throws Exception{
+ PeerTaskInstancePriorityQueue queue = new PeerTaskInstancePriorityQueue();
+ TaskInstance taskInstanceHigPriority = createTaskInstance("high", Priority.HIGH, 2);
+ TaskInstance taskInstanceMediumPriority = createTaskInstance("medium", Priority.HIGH, 1);
+ queue.put(taskInstanceMediumPriority);
+ queue.put(taskInstanceHigPriority);
+ TaskInstance taskInstance = queue.peek();
+ queue.clear();
+ Assert.assertEquals(taskInstance.getName(), "high");
+
+ taskInstanceHigPriority = createTaskInstance("high", Priority.HIGH, 1);
+ taskInstanceMediumPriority = createTaskInstance("medium", Priority.HIGH, 2);
+ queue.put(taskInstanceMediumPriority);
+ queue.put(taskInstanceHigPriority);
+ taskInstance = queue.peek();
+ queue.clear();
+ Assert.assertEquals(taskInstance.getName(), "medium");
+
+ taskInstanceHigPriority = createTaskInstance("high", Priority.HIGH, 1);
+ taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM, 2);
+ queue.put(taskInstanceMediumPriority);
+ queue.put(taskInstanceHigPriority);
+ taskInstance = queue.peek();
+ queue.clear();
+ Assert.assertEquals(taskInstance.getName(), "high");
+
+ taskInstanceHigPriority = createTaskInstance("high", Priority.HIGH, 1);
+ taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM, 1);
+ queue.put(taskInstanceMediumPriority);
+ queue.put(taskInstanceHigPriority);
+ taskInstance = queue.peek();
+ queue.clear();
+ Assert.assertEquals(taskInstance.getName(), "high");
+
+ }
+
@Test
public void size() throws Exception {
Assert.assertEquals(2, getPeerTaskInstancePriorityQueue().size());
@@ -72,7 +108,7 @@ public class PeerTaskInstancePriorityQueueTest {
@Test
public void contains() throws Exception {
PeerTaskInstancePriorityQueue queue = new PeerTaskInstancePriorityQueue();
- TaskInstance taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM);
+ TaskInstance taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM, 1);
queue.put(taskInstanceMediumPriority);
Assert.assertTrue(queue.contains(taskInstanceMediumPriority));
}
@@ -80,7 +116,7 @@ public class PeerTaskInstancePriorityQueueTest {
@Test
public void remove() throws Exception {
PeerTaskInstancePriorityQueue queue = new PeerTaskInstancePriorityQueue();
- TaskInstance taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM);
+ TaskInstance taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM, 1);
queue.put(taskInstanceMediumPriority);
int peekBeforeLength = queue.size();
queue.remove(taskInstanceMediumPriority);
@@ -95,10 +131,12 @@ public class PeerTaskInstancePriorityQueueTest {
*/
private PeerTaskInstancePriorityQueue getPeerTaskInstancePriorityQueue() throws Exception {
PeerTaskInstancePriorityQueue queue = new PeerTaskInstancePriorityQueue();
- TaskInstance taskInstanceHigPriority = createTaskInstance("high", Priority.HIGH);
- TaskInstance taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM);
- queue.put(taskInstanceHigPriority);
+ TaskInstance taskInstanceHigPriority = createTaskInstance("high", Priority.HIGH, 1);
+ TaskInstance taskInstanceMediumPriority = createTaskInstance("medium", Priority.MEDIUM, 1);
+ taskInstanceHigPriority.setTaskGroupPriority(3);
+ taskInstanceMediumPriority.setTaskGroupPriority(2);
queue.put(taskInstanceMediumPriority);
+ queue.put(taskInstanceHigPriority);
return queue;
}
@@ -109,10 +147,11 @@ public class PeerTaskInstancePriorityQueueTest {
* @param priority priority
* @return
*/
- private TaskInstance createTaskInstance(String name, Priority priority) {
+ private TaskInstance createTaskInstance(String name, Priority priority, int taskGroupPriority) {
TaskInstance taskInstance = new TaskInstance();
taskInstance.setName(name);
taskInstance.setTaskInstancePriority(priority);
+ taskInstance.setTaskGroupPriority(taskGroupPriority);
return taskInstance;
}
}
diff --git a/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/TaskPriorityQueueImplTest.java b/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/TaskPriorityQueueImplTest.java
index 3888d3d93c..c2fb14acfa 100644
--- a/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/TaskPriorityQueueImplTest.java
+++ b/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/TaskPriorityQueueImplTest.java
@@ -31,9 +31,9 @@ public class TaskPriorityQueueImplTest {
@Test
public void testSort() {
- TaskPriority priorityOne = new TaskPriority(1, 0, 0, 0, "default");
- TaskPriority priorityTwo = new TaskPriority(2, 0, 0, 0, "default");
- TaskPriority priorityThree = new TaskPriority(3, 0, 0, 0, "default");
+ TaskPriority priorityOne = new TaskPriority(1, 0, 0, 0, 1, "default");
+ TaskPriority priorityTwo = new TaskPriority(2, 0, 0, 0, 1, "default");
+ TaskPriority priorityThree = new TaskPriority(3, 0, 0, 0, 1, "default");
List<TaskPriority> taskPrioritys = Arrays.asList(priorityOne, priorityThree, priorityTwo);
Collections.sort(taskPrioritys);
Assert.assertEquals(
@@ -41,9 +41,9 @@ public class TaskPriorityQueueImplTest {
taskPrioritys
);
- priorityOne = new TaskPriority(0, 1, 0, 0, "default");
- priorityTwo = new TaskPriority(0, 2, 0, 0, "default");
- priorityThree = new TaskPriority(0, 3, 0, 0, "default");
+ priorityOne = new TaskPriority(0, 1, 0, 0, 1, "default");
+ priorityTwo = new TaskPriority(0, 2, 0, 0, 1, "default");
+ priorityThree = new TaskPriority(0, 3, 0, 0, 1, "default");
taskPrioritys = Arrays.asList(priorityOne, priorityThree, priorityTwo);
Collections.sort(taskPrioritys);
Assert.assertEquals(
@@ -51,9 +51,9 @@ public class TaskPriorityQueueImplTest {
taskPrioritys
);
- priorityOne = new TaskPriority(0, 0, 1, 0, "default");
- priorityTwo = new TaskPriority(0, 0, 2, 0, "default");
- priorityThree = new TaskPriority(0, 0, 3, 0, "default");
+ priorityOne = new TaskPriority(0, 0, 1, 0, 1, "default");
+ priorityTwo = new TaskPriority(0, 0, 2, 0, 1, "default");
+ priorityThree = new TaskPriority(0, 0, 3, 0, 1, "default");
taskPrioritys = Arrays.asList(priorityOne, priorityThree, priorityTwo);
Collections.sort(taskPrioritys);
Assert.assertEquals(
@@ -61,9 +61,9 @@ public class TaskPriorityQueueImplTest {
taskPrioritys
);
- priorityOne = new TaskPriority(0, 0, 0, 1, "default");
- priorityTwo = new TaskPriority(0, 0, 0, 2, "default");
- priorityThree = new TaskPriority(0, 0, 0, 3, "default");
+ priorityOne = new TaskPriority(0, 0, 0, 1, 1, "default");
+ priorityTwo = new TaskPriority(0, 0, 0, 2, 1, "default");
+ priorityThree = new TaskPriority(0, 0, 0, 3, 1, "default");
taskPrioritys = Arrays.asList(priorityOne, priorityThree, priorityTwo);
Collections.sort(taskPrioritys);
Assert.assertEquals(
@@ -71,15 +71,45 @@ public class TaskPriorityQueueImplTest {
taskPrioritys
);
- priorityOne = new TaskPriority(0, 0, 0, 0, "default_1");
- priorityTwo = new TaskPriority(0, 0, 0, 0, "default_2");
- priorityThree = new TaskPriority(0, 0, 0, 0, "default_3");
+ priorityOne = new TaskPriority(0, 0, 0, 0, 1, "default_1");
+ priorityTwo = new TaskPriority(0, 0, 0, 0, 1,"default_2");
+ priorityThree = new TaskPriority(0, 0, 0, 0, 1, "default_3");
taskPrioritys = Arrays.asList(priorityOne, priorityThree, priorityTwo);
Collections.sort(taskPrioritys);
Assert.assertEquals(
Arrays.asList(priorityOne, priorityTwo, priorityThree),
taskPrioritys
);
+
+ priorityOne = new TaskPriority(0, 0, 0, 0, 2, "default_1");
+ priorityTwo = new TaskPriority(0, 0, 0, 0, 1,"default_2");
+ priorityThree = new TaskPriority(0, 0, 0, 0, 3, "default_3");
+ taskPrioritys = Arrays.asList(priorityOne, priorityThree, priorityTwo);
+ Collections.sort(taskPrioritys);
+ Assert.assertEquals(
+ Arrays.asList(priorityThree, priorityOne, priorityTwo),
+ taskPrioritys
+ );
+
+ priorityOne = new TaskPriority(0, 0, 0, 0, 1, "default_1");
+ priorityTwo = new TaskPriority(0, 0, 0, 0, 1,"default_2");
+ priorityThree = new TaskPriority(0, 0, 0, 0, 3, "default_3");
+ taskPrioritys = Arrays.asList(priorityOne, priorityThree, priorityTwo);
+ Collections.sort(taskPrioritys);
+ Assert.assertEquals(
+ Arrays.asList(priorityThree, priorityOne, priorityTwo),
+ taskPrioritys
+ );
+
+ priorityTwo = new TaskPriority(0, 0, 0, 0, 1,"default_1");
+ priorityOne = new TaskPriority(0, 0, 0, 0, 1, "default_1");
+ priorityThree = new TaskPriority(0, 0, 0, 0, 3, "default_1");
+ taskPrioritys = Arrays.asList(priorityTwo, priorityOne, priorityThree);
+ Collections.sort(taskPrioritys);
+ Assert.assertEquals(
+ Arrays.asList(priorityThree, priorityTwo, priorityOne),
+ taskPrioritys
+ );
}
@Test
@@ -134,7 +164,7 @@ public class TaskPriorityQueueImplTest {
* @return
*/
private TaskPriority createTaskPriority(Integer priority, Integer processInstanceId) {
- TaskPriority priorityOne = new TaskPriority(priority, processInstanceId, 0, 0, "default");
+ TaskPriority priorityOne = new TaskPriority(priority, processInstanceId, 0, 0, 1, "default");
return priorityOne;
}
}
\ No newline at end of file
[dolphinscheduler] 25/33: [doc] Update website docs link (#10161)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 8ba5fa1afdfca7d136c4508c2da83e406255f56c
Author: Mr.An <42...@users.noreply.github.com>
AuthorDate: Fri May 20 16:47:41 2022 +0800
[doc] Update website docs link (#10161)
(cherry picked from commit e2fac213991c30f0a7460e3576fa3bbb5e6e6d42)
---
docs/configs/site.js | 14 ++------------
1 file changed, 2 insertions(+), 12 deletions(-)
diff --git a/docs/configs/site.js b/docs/configs/site.js
index 982a32aa04..aa9cdf2124 100644
--- a/docs/configs/site.js
+++ b/docs/configs/site.js
@@ -57,7 +57,7 @@ export default {
key: 'docsHistory',
text: 'Older Versions',
link: '/en-us/docs/release/history-versions.html',
- },
+ }
],
},
{
@@ -233,17 +233,7 @@ export default {
key: 'docsHistory',
text: '历史版本',
link: '/zh-cn/docs/release/history-versions.html',
- },
- {
- key: 'pythonAPI',
- text: 'PyDolphinScheduler',
- link: '/python/index.html',
- },
- {
- key: 'docsdev',
- text: 'dev',
- link: '/zh-cn/docs/dev/user_doc/about/introduction.html',
- },
+ }
],
},
{
[dolphinscheduler] 10/33: [Feat][UI] Add resource username for function and resource manage (#10086)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 04c68e31e8382b5beb944a86581546ffc4737c23
Author: Devosend <de...@gmail.com>
AuthorDate: Wed May 18 12:00:51 2022 +0800
[Feat][UI] Add resource username for function and resource manage (#10086)
(cherry picked from commit 91f1d45948d5d53b790aefbf589d4a91fba7f817)
---
dolphinscheduler-ui/src/locales/modules/en_US.ts | 2 ++
dolphinscheduler-ui/src/locales/modules/zh_CN.ts | 2 ++
dolphinscheduler-ui/src/views/resource/udf/function/use-table.ts | 5 +++++
dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts | 5 +++++
4 files changed, 14 insertions(+)
diff --git a/dolphinscheduler-ui/src/locales/modules/en_US.ts b/dolphinscheduler-ui/src/locales/modules/en_US.ts
index 2d9555653b..20ae35e133 100644
--- a/dolphinscheduler-ui/src/locales/modules/en_US.ts
+++ b/dolphinscheduler-ui/src/locales/modules/en_US.ts
@@ -239,6 +239,7 @@ const resource = {
create_folder: 'Create Folder',
upload_udf_resources: 'Upload UDF Resources',
udf_source_name: 'UDF Resource Name',
+ user_name: 'Resource userName',
whether_directory: 'Whether directory',
file_name: 'File Name',
file_size: 'File Size',
@@ -266,6 +267,7 @@ const resource = {
create_udf_function: 'Create UDF Function',
edit_udf_function: 'Create UDF Function',
udf_function_name: 'UDF Function Name',
+ user_name: 'Resource userName',
class_name: 'Class Name',
type: 'Type',
description: 'Description',
diff --git a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
index b209c3844d..6ef3bd209d 100644
--- a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
+++ b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
@@ -238,6 +238,7 @@ const resource = {
create_folder: '创建文件夹',
upload_udf_resources: '上传UDF资源',
udf_source_name: 'UDF资源名称',
+ user_name: '所属用户',
whether_directory: '是否文件夹',
file_name: '文件名称',
file_size: '文件大小',
@@ -265,6 +266,7 @@ const resource = {
create_udf_function: '创建UDF函数',
edit_udf_function: '编辑UDF函数',
udf_function_name: 'UDF函数名称',
+ user_name: '所属用户',
class_name: '类名',
type: '类型',
description: '描述',
diff --git a/dolphinscheduler-ui/src/views/resource/udf/function/use-table.ts b/dolphinscheduler-ui/src/views/resource/udf/function/use-table.ts
index cf4a0407bb..a7d737c6db 100644
--- a/dolphinscheduler-ui/src/views/resource/udf/function/use-table.ts
+++ b/dolphinscheduler-ui/src/views/resource/udf/function/use-table.ts
@@ -65,6 +65,11 @@ export function useTable() {
key: 'funcName',
...COLUMN_WIDTH_CONFIG['name']
},
+ {
+ title: t('resource.function.user_name'),
+ ...COLUMN_WIDTH_CONFIG['userName'],
+ key: 'userName'
+ },
{
title: t('resource.function.class_name'),
key: 'className',
diff --git a/dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts b/dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts
index a02f9fa6b6..4001a1f303 100644
--- a/dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts
+++ b/dolphinscheduler-ui/src/views/resource/udf/resource/use-table.ts
@@ -102,6 +102,11 @@ export function useTable() {
)
}
},
+ {
+ title: t('resource.udf.user_name'),
+ ...COLUMN_WIDTH_CONFIG['userName'],
+ key: 'userName'
+ },
{
title: t('resource.udf.whether_directory'),
key: 'whether_directory',
[dolphinscheduler] 21/33: [python] Change distribute name avoid confusion with main pkg (#10134)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit bb47ced47440c271f023ae6b4881bde5438dc620
Author: Jiajie Zhong <zh...@hotmail.com>
AuthorDate: Thu May 19 20:27:38 2022 +0800
[python] Change distribute name avoid confusion with main pkg (#10134)
rename them and add `python` suffix in the end
close: #9121
(cherry picked from commit bf4b8ff0d98990a46dbc7adac01164c515b13ad3)
---
dolphinscheduler-python/pom.xml | 31 +++++++++++++++++++++++++++++++
1 file changed, 31 insertions(+)
diff --git a/dolphinscheduler-python/pom.xml b/dolphinscheduler-python/pom.xml
index 749c767e87..1a740fbd9d 100644
--- a/dolphinscheduler-python/pom.xml
+++ b/dolphinscheduler-python/pom.xml
@@ -92,6 +92,37 @@
</arguments>
</configuration>
</execution>
+ <!-- Rename Python dist package to avoid confusion with dolphinscheduler main package -->
+ <execution>
+ <id>python-pkg-rename-tar</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>bash</executable>
+ <workingDirectory>${project.basedir}/pydolphinscheduler</workingDirectory>
+ <arguments>
+ <argument>-c</argument>
+ <argument>mv dist/apache-dolphinscheduler-*.tar.gz dist/apache-dolphinscheduler-python-${project.version}.tar.gz</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ <execution>
+ <id>python-pkg-rename-whl</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>bash</executable>
+ <workingDirectory>${project.basedir}/pydolphinscheduler</workingDirectory>
+ <arguments>
+ <argument>-c</argument>
+ <argument>mv dist/apache_dolphinscheduler-*py3-none-any.whl dist/apache_dolphinscheduler-python-${project.version}-py3-none-any.whl</argument>
+ </arguments>
+ </configuration>
+ </execution>
<execution>
<id>sign-source</id>
<phase>prepare-package</phase>
[dolphinscheduler] 15/33: [Build][UI] Solve vite config throws terser warning. (#10105)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 2c290b2f2bff641f9f7be6919c8c7ab18fae93be
Author: songjianet <17...@qq.com>
AuthorDate: Wed May 18 22:12:52 2022 +0800
[Build][UI] Solve vite config throws terser warning. (#10105)
(cherry picked from commit bc71c3b0a0f7d93c3fccb850beb386fc5eb1a20c)
---
dolphinscheduler-ui/vite.config.ts | 8 --------
1 file changed, 8 deletions(-)
diff --git a/dolphinscheduler-ui/vite.config.ts b/dolphinscheduler-ui/vite.config.ts
index 1988ee8072..bad7b707f5 100644
--- a/dolphinscheduler-ui/vite.config.ts
+++ b/dolphinscheduler-ui/vite.config.ts
@@ -35,14 +35,6 @@ export default defineConfig({
deleteOriginFile: false
})
],
- build: {
- terserOptions: {
- compress: {
- drop_console: true,
- drop_debugger: true
- }
- }
- },
resolve: {
alias: {
'@': path.resolve(__dirname, 'src'),
[dolphinscheduler] 09/33: [Fix-10049] Conditions Task branch flow failed (#10077)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 98576cb509d20cd7b5e94a4d7b645631be2bf8b1
Author: xiangzihao <46...@qq.com>
AuthorDate: Wed May 18 11:40:32 2022 +0800
[Fix-10049] Conditions Task branch flow failed (#10077)
(cherry picked from commit 225cb332d18e2fef58bf653d644f415dfcce2e99)
---
docs/docs/en/guide/task/conditions.md | 3 ++-
docs/docs/zh/guide/task/conditions.md | 3 ++-
docs/img/tasks/demo/condition_task04.png | Bin 0 -> 33271 bytes
.../server/master/runner/WorkflowExecuteThread.java | 10 ++++++++--
.../master/runner/task/ConditionTaskProcessor.java | 5 ++---
dolphinscheduler-ui/src/locales/modules/en_US.ts | 1 +
dolphinscheduler-ui/src/locales/modules/zh_CN.ts | 1 +
.../task/components/node/fields/use-conditions.ts | 2 +-
8 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/docs/docs/en/guide/task/conditions.md b/docs/docs/en/guide/task/conditions.md
index 4bd8094648..df723b355b 100644
--- a/docs/docs/en/guide/task/conditions.md
+++ b/docs/docs/en/guide/task/conditions.md
@@ -47,7 +47,7 @@ Go to the workflow definition page, and then create the following task nodes:
After you finish creating the workflow, you can run the workflow online. You can view the execution status of each task on the workflow instance page. As shown below:
-
+
In the above figure, the task status marked with a green check mark is the successfully executed task node.
@@ -58,3 +58,4 @@ In the above figure, the task status marked with a green check mark is the succe
- The predecessor task of Conditions cannot connect to its branch nodes, which will cause logical confusion and does not conform to DAG scheduling. The situation shown below is **wrong**.
+
diff --git a/docs/docs/zh/guide/task/conditions.md b/docs/docs/zh/guide/task/conditions.md
index 66f11fa3ed..c7fef6d6f1 100644
--- a/docs/docs/zh/guide/task/conditions.md
+++ b/docs/docs/zh/guide/task/conditions.md
@@ -47,7 +47,7 @@ Conditions 是一个条件节点，根据上游任务运行状态，判断应该
当完成创建工作流之后，可以上线运行该工作流。在工作流实例页面可以查看到各个任务的执行状态。如下图所示：
-
+
上图中，任务状态标记为绿色对号的，即为成功执行的任务节点。
@@ -58,3 +58,4 @@ Conditions 是一个条件节点，根据上游任务运行状态，判断应该
- Conditions 的前置任务不能连接其分支节点，会造成逻辑混乱，不符合 DAG 调度。如下图所示的情况是**错误**的。
+
diff --git a/docs/img/tasks/demo/condition_task04.png b/docs/img/tasks/demo/condition_task04.png
new file mode 100644
index 0000000000..9da10be464
Binary files /dev/null and b/docs/img/tasks/demo/condition_task04.png differ
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java
index 3b7b5511ae..a40b7e5b27 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java
@@ -1376,10 +1376,16 @@ public class WorkflowExecuteThread {
if (depTaskState.typeIsPause() || depTaskState.typeIsCancel()) {
return DependResult.NON_EXEC;
}
- // ignore task state if current task is condition and block
- if (taskNode.isConditionsTask() || taskNode.isBlockingTask()) {
+ // ignore task state if current task is block
+ if (taskNode.isBlockingTask()) {
continue;
}
+
+ // always return success if current task is condition
+ if (taskNode.isConditionsTask()) {
+ continue;
+ }
+
if (!dependTaskSuccess(depsNode, taskCode)) {
return DependResult.FAILED;
}
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/ConditionTaskProcessor.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/ConditionTaskProcessor.java
index 0928d122ba..4749e20f0f 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/ConditionTaskProcessor.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/ConditionTaskProcessor.java
@@ -68,7 +68,7 @@ public class ConditionTaskProcessor extends BaseTaskProcessor {
}
this.setTaskExecutionLogger();
initTaskParameters();
- logger.info("dependent task start");
+ logger.info("condition task start");
return true;
}
@@ -170,8 +170,7 @@ public class ConditionTaskProcessor extends BaseTaskProcessor {
logger.info("depend item : {} expect status: {}, actual status: {}", item.getDepTaskCode(), item.getStatus(), executionStatus);
dependResult = DependResult.FAILED;
}
- logger.info("dependent item complete {} {},{}",
- Constants.DEPENDENT_SPLIT, item.getDepTaskCode(), dependResult);
+ logger.info("dependent item complete, dependentTaskCode: {}, dependResult: {}", item.getDepTaskCode(), dependResult);
return dependResult;
}
diff --git a/dolphinscheduler-ui/src/locales/modules/en_US.ts b/dolphinscheduler-ui/src/locales/modules/en_US.ts
index c01db94775..2d9555653b 100644
--- a/dolphinscheduler-ui/src/locales/modules/en_US.ts
+++ b/dolphinscheduler-ui/src/locales/modules/en_US.ts
@@ -822,6 +822,7 @@ const project = {
sea_tunnel_queue: 'Queue',
sea_tunnel_master_url_tips:
'Please enter the master url, e.g., 127.0.0.1:7077',
+ add_pre_task_check_condition: 'Add pre task check condition',
switch_condition: 'Condition',
switch_branch_flow: 'Branch Flow',
and: 'and',
diff --git a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
index 28a48dc565..b209c3844d 100644
--- a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
+++ b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
@@ -811,6 +811,7 @@ const project = {
sea_tunnel_master_url: 'Master URL',
sea_tunnel_queue: '队列',
sea_tunnel_master_url_tips: '请直接填写地址,例如:127.0.0.1:7077',
+ add_pre_task_check_condition: '添加前置检查条件',
switch_condition: '条件',
switch_branch_flow: '分支流转',
and: '且',
diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-conditions.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-conditions.ts
index 356e6e0dd3..53df0bba75 100644
--- a/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-conditions.ts
+++ b/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-conditions.ts
@@ -121,7 +121,7 @@ export function useConditions(model: { [field: string]: any }): IJsonItem[] {
]
},
childrenField: 'dependItemList',
- name: 'custom_parameters'
+ name: 'add_pre_task_check_condition'
})
]
}
[dolphinscheduler] 11/33: fix issue-10061 (#10062)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 4f74db829e642e8e482991e467614e43400892ba
Author: QuakeWang <45...@users.noreply.github.com>
AuthorDate: Wed May 18 12:21:35 2022 +0800
fix issue-10061 (#10062)
(cherry picked from commit 3bca2404466d9ff1a8772e89f50b0dbb27448338)
---
docs/configs/docsdev.js | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/configs/docsdev.js b/docs/configs/docsdev.js
index c6432e071d..71ec133d3a 100644
--- a/docs/configs/docsdev.js
+++ b/docs/configs/docsdev.js
@@ -276,7 +276,7 @@ export default {
},
{
title: 'Task Group Manage',
- link: '/en-us/docs/3.0.0/user_doc/guide/resource/task-manage.html'
+ link: '/en-us/docs/3.0.0/user_doc/guide/resource/task-group.html'
},
],
},
@@ -623,7 +623,7 @@ export default {
},
{
title: '任务组管理',
- link: '/zh-cn/docs/3.0.0/user_doc/guide/resource/task-manage.html'
+ link: '/zh-cn/docs/3.0.0/user_doc/guide/resource/task-group.html'
},
],
},
[dolphinscheduler] 02/33: [Fix][UI] Fix in the Chinese state, there is a problem with the multi-language support of the form. (#10066)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 6bc9141c91388131e26c01bf35d31ba9e57c7a0f
Author: songjianet <17...@qq.com>
AuthorDate: Tue May 17 14:57:04 2022 +0800
[Fix][UI] Fix in the Chinese state, there is a problem with the multi-language support of the form. (#10066)
(cherry picked from commit 359300380b0a2eb469beb0497271bda675860d7d)
---
.../src/views/security/tenant-manage/components/use-modalData.ts | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/dolphinscheduler-ui/src/views/security/tenant-manage/components/use-modalData.ts b/dolphinscheduler-ui/src/views/security/tenant-manage/components/use-modalData.ts
index 331573c56a..bd3c00bfc8 100644
--- a/dolphinscheduler-ui/src/views/security/tenant-manage/components/use-modalData.ts
+++ b/dolphinscheduler-ui/src/views/security/tenant-manage/components/use-modalData.ts
@@ -23,11 +23,13 @@ import {
createTenant,
updateTenant
} from '@/service/modules/tenants'
+import { useI18n } from 'vue-i18n'
export function useModalData(
props: any,
ctx: SetupContext<('cancelModal' | 'confirmModal')[]>
) {
+ const { t } = useI18n()
const variables = reactive({
tenantFormRef: ref(),
model: {
@@ -40,7 +42,12 @@ export function useModalData(
saving: false,
rules: {
tenantCode: {
- required: true
+ required: true,
+ validator() {
+ if (!variables.model.tenantCode) {
+ return new Error(t('security.tenant.tenant_code_tips'))
+ }
+ }
},
queueId: {
required: true
[dolphinscheduler] 18/33: [doc] Show all the functions of the home page through screenshots (#10118)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit ebe76fb986d630e977d618be623989d270e087bc
Author: Mr.An <42...@users.noreply.github.com>
AuthorDate: Thu May 19 14:45:44 2022 +0800
[doc] Show all the functions of the home page through screenshots (#10118)
(cherry picked from commit 73f13a9c9c3bbe821437ab4a01b648e58e6284ec)
---
docs/img/new_ui/dev/homepage/homepage.png | Bin 161117 -> 746975 bytes
1 file changed, 0 insertions(+), 0 deletions(-)
diff --git a/docs/img/new_ui/dev/homepage/homepage.png b/docs/img/new_ui/dev/homepage/homepage.png
index 5b852f4d2d..b969575465 100644
Binary files a/docs/img/new_ui/dev/homepage/homepage.png and b/docs/img/new_ui/dev/homepage/homepage.png differ
[dolphinscheduler] 22/33: [Fix][UI] Add the task status of pending. (#10129)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit a521d49e94e50895c429d5b8e58e4556702bfad0
Author: Amy0104 <97...@users.noreply.github.com>
AuthorDate: Fri May 20 10:06:43 2022 +0800
[Fix][UI] Add the task status of pending. (#10129)
(cherry picked from commit 3dd007ca599c5d3fdd6b32db8c2559991940ddc6)
---
dolphinscheduler-ui/src/common/common.ts | 11 ++++++++++-
dolphinscheduler-ui/src/common/types.ts | 1 +
dolphinscheduler-ui/src/locales/modules/en_US.ts | 1 +
dolphinscheduler-ui/src/locales/modules/zh_CN.ts | 1 +
4 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/dolphinscheduler-ui/src/common/common.ts b/dolphinscheduler-ui/src/common/common.ts
index 29a58ac627..ae644aeb01 100644
--- a/dolphinscheduler-ui/src/common/common.ts
+++ b/dolphinscheduler-ui/src/common/common.ts
@@ -31,7 +31,8 @@ import {
StopOutlined,
GlobalOutlined,
IssuesCloseOutlined,
- SendOutlined
+ SendOutlined,
+ HistoryOutlined
} from '@vicons/antd'
import { format, parseISO } from 'date-fns'
import _ from 'lodash'
@@ -275,6 +276,14 @@ export const tasksState = (t: any): ITaskStateConfig => ({
icon: SendOutlined,
isSpin: false,
classNames: 'dispatch'
+ },
+ PENDING: {
+ id: 18,
+ desc: `${t('project.workflow.pending')}`,
+ color: '#5101be',
+ icon: HistoryOutlined,
+ isSpin: false,
+ classNames: 'pending'
}
})
diff --git a/dolphinscheduler-ui/src/common/types.ts b/dolphinscheduler-ui/src/common/types.ts
index c766e1471f..20085cefc0 100644
--- a/dolphinscheduler-ui/src/common/types.ts
+++ b/dolphinscheduler-ui/src/common/types.ts
@@ -33,6 +33,7 @@ export type ITaskState =
| 'FORCED_SUCCESS'
| 'SERIAL_WAIT'
| 'DISPATCH'
+ | 'PENDING'
export type ITaskStateConfig = {
[key in ITaskState]: {
diff --git a/dolphinscheduler-ui/src/locales/modules/en_US.ts b/dolphinscheduler-ui/src/locales/modules/en_US.ts
index 20ae35e133..9c7664c5d2 100644
--- a/dolphinscheduler-ui/src/locales/modules/en_US.ts
+++ b/dolphinscheduler-ui/src/locales/modules/en_US.ts
@@ -498,6 +498,7 @@ const project = {
forced_success: 'Forced success',
serial_wait: 'Serial wait',
dispatch: 'Dispatch',
+ pending: 'Pending',
executing: 'Executing',
startup_type: 'Startup Type',
complement_range: 'Complement Range',
diff --git a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
index 6ef3bd209d..2d3e34ad39 100644
--- a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
+++ b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
@@ -495,6 +495,7 @@ const project = {
forced_success: '强制成功',
serial_wait: '串行等待',
dispatch: '派发',
+ pending: '挂起',
executing: '正在执行',
startup_type: '启动类型',
complement_range: '补数范围',
[dolphinscheduler] 12/33: [Fix][UI] fulfill missing fields in DataQuality (#10098)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 2797fcfc68fffbad6b7fbcdb32666b6b8d2a56dc
Author: rockfang <65...@qq.com>
AuthorDate: Wed May 18 18:28:24 2022 +0800
[Fix][UI] fulfill missing fields in DataQuality (#10098)
(cherry picked from commit 7a4a21d0b1832fe173bfa371af9bf7baeddbb036)
---
.../src/views/projects/task/components/node/format-data.ts | 14 ++++++++++++++
.../src/views/projects/task/components/node/types.ts | 7 +++++++
2 files changed, 21 insertions(+)
diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts
index 6bc512e5f4..c04c7daf7a 100644
--- a/dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts
+++ b/dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts
@@ -279,6 +279,13 @@ export function formatParams(data: INodeData): {
operator: data.operator,
src_connector_type: data.src_connector_type,
src_datasource_id: data.src_datasource_id,
+ field_length: data.field_length,
+ begin_time: data.begin_time,
+ deadline: data.deadline,
+ datetime_format: data.datetime_format,
+ enum_list: data.enum_list,
+ regexp_pattern: data.regexp_pattern,
+ target_filter: data.target_filter,
src_filter: data.src_filter,
src_field: data.src_field,
src_table: data.src_table,
@@ -499,6 +506,13 @@ export function formatModel(data: ITaskData) {
params.src_datasource_id =
data.taskParams.ruleInputParameter.src_datasource_id
params.src_table = data.taskParams.ruleInputParameter.src_table
+ params.field_length = data.taskParams.ruleInputParameter.field_length
+ params.begin_time = data.taskParams.ruleInputParameter.begin_time
+ params.deadline = data.taskParams.ruleInputParameter.deadline
+ params.datetime_format = data.taskParams.ruleInputParameter.datetime_format
+ params.target_filter = data.taskParams.ruleInputParameter.target_filter
+ params.regexp_pattern = data.taskParams.ruleInputParameter.regexp_pattern
+ params.enum_list = data.taskParams.ruleInputParameter.enum_list
params.src_filter = data.taskParams.ruleInputParameter.src_filter
params.src_field = data.taskParams.ruleInputParameter.src_field
params.statistics_execute_sql =
diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/types.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/types.ts
index 6750d84fc9..bbcaecbc08 100644
--- a/dolphinscheduler-ui/src/views/projects/task/components/node/types.ts
+++ b/dolphinscheduler-ui/src/views/projects/task/components/node/types.ts
@@ -198,6 +198,13 @@ interface IRuleParameters {
src_connector_type?: number
src_datasource_id?: number
src_table?: string
+ field_length?: number
+ begin_time?: string
+ deadline?: string
+ datetime_format?: string
+ target_filter?: string
+ regexp_pattern?: string
+ enum_list?: string
src_filter?: string
src_field?: string
statistics_execute_sql?: string
[dolphinscheduler] 30/33: fix spcial week init bug (#10183)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 8a52958f368451a34c50bc9381eb12baff74fac7
Author: Devosend <de...@gmail.com>
AuthorDate: Sun May 22 14:02:56 2022 +0800
fix spcial week init bug (#10183)
(cherry picked from commit d7d6699e22091896a9ea78c5a2e49c2a1f54a7e4)
---
dolphinscheduler-ui/src/components/crontab/modules/day.tsx | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/dolphinscheduler-ui/src/components/crontab/modules/day.tsx b/dolphinscheduler-ui/src/components/crontab/modules/day.tsx
index b4dd16d1af..9c67ff9db3 100644
--- a/dolphinscheduler-ui/src/components/crontab/modules/day.tsx
+++ b/dolphinscheduler-ui/src/components/crontab/modules/day.tsx
@@ -62,7 +62,7 @@ export default defineComponent({
const intervalDayStartRef = ref(1)
const intervalDayPerformRef = ref(1)
const WkspecificDayRef = ref<Array<number>>([])
- const WkspecificWeekRef = ref<Array<number>>([])
+ const WkspecificWeekRef = ref<Array<string>>([])
const monthLastDaysRef = ref('L')
const monthLastWorkingDaysRef = ref('LW')
const monthLastWeeksRef = ref('?')
@@ -108,9 +108,7 @@ export default defineComponent({
}
const hanleWeekTwo = () => {
- WkspecificWeekRef.value = $weekVal
- .split(',')
- .map((item) => parseInt(item))
+ WkspecificWeekRef.value = $weekVal.split(',')
radioRef.value = 'WkspecificWeek'
}
@@ -244,7 +242,7 @@ export default defineComponent({
}
// Specific day of the week (multiple choice)
- const onWkspecificWeek = (arr: Array<number>) => {
+ const onWkspecificWeek = (arr: Array<string>) => {
WkspecificWeekRef.value = arr
if (radioRef.value === 'WkspecificWeek') {
dayRef.value = '?'
[dolphinscheduler] 17/33: [Bug][doc] Fix the open-api docs request error (#10100)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 418d422bb94397f614096d71ce3ccdeabc241383
Author: QuakeWang <45...@users.noreply.github.com>
AuthorDate: Thu May 19 11:28:33 2022 +0800
[Bug][doc] Fix the open-api docs request error (#10100)
(cherry picked from commit 243ee8ba70d50c7d4b251fd43a4c150606014b8b)
---
docs/docs/en/guide/open-api.md | 69 +++++++++---------
docs/docs/zh/guide/open-api.md | 82 +++++++++++-----------
docs/img/api/create_project1.png | Bin 38616 -> 0 bytes
docs/img/api/create_project2.png | Bin 38329 -> 0 bytes
docs/img/api/create_project3.png | Bin 73852 -> 0 bytes
docs/img/api/create_source1.png | Bin 360228 -> 0 bytes
docs/img/api/create_source2.png | Bin 278176 -> 0 bytes
docs/img/new_ui/dev/open-api/api_doc.png | Bin 0 -> 210921 bytes
docs/img/new_ui/dev/open-api/api_source01.png | Bin 0 -> 257758 bytes
docs/img/new_ui/dev/open-api/api_source02.png | Bin 0 -> 233612 bytes
docs/img/new_ui/dev/open-api/api_test.png | Bin 0 -> 181011 bytes
docs/img/new_ui/dev/open-api/create_project01.png | Bin 0 -> 104392 bytes
docs/img/new_ui/dev/open-api/create_project02.png | Bin 0 -> 118638 bytes
docs/img/new_ui/dev/open-api/create_project03.png | Bin 0 -> 52968 bytes
docs/img/new_ui/dev/open-api/token_expiration.png | Bin 0 -> 261485 bytes
15 files changed, 76 insertions(+), 75 deletions(-)
diff --git a/docs/docs/en/guide/open-api.md b/docs/docs/en/guide/open-api.md
index 1bac59b494..656fb415b4 100644
--- a/docs/docs/en/guide/open-api.md
+++ b/docs/docs/en/guide/open-api.md
@@ -10,60 +10,59 @@ Generally, projects and processes are created through pages, but considering the
1. Log in to the scheduling system, click "Security", then click "Token manage" on the left, and click "Create token" to create a token.
-<p align="center">
- <img src="/img/token-management-en.png" width="80%" />
- </p>
+
2. Select the "Expiration time" (Token validity time), select "User" (choose the specified user to perform the API operation), click "Generate token", copy the `Token` string, and click "Submit".
-<p align="center">
- <img src="/img/create-token-en1.png" width="80%" />
- </p>
+
-### Token Usage
+### Examples
+
+#### Query project list
+
+1. Open the API documentation
+
+> Address：http://{API server ip}:12345/dolphinscheduler/doc.html?language=en_US&lang=en
+
+
-1. Open the API documentation page
- > Address：http://{API server ip}:12345/dolphinscheduler/doc.html?language=en_US&lang=en
-<p align="center">
- <img src="/img/api-documentation-en.png" width="80%" />
- </p>
-
2. select a test API, the API selected for this test is `queryAllProjectList`
- > projects/query-project-list
+
+ > projects/list
+ >
3. Open `Postman`, fill in the API address, enter the `Token` in `Headers`, and then send the request to view the result:
- ```
+
+ ```
token: The Token just generated
```
-<p align="center">
- <img src="/img/test-api.png" width="80%" />
- </p>
+
+
-### Create a Project
+#### Create a Project
-Here is an example of creating a project named "wudl-flink-test"：
-<p align="center">
- <img src="/img/api/create_project1.png" width="80%" />
- </p>
+This demonstrates how to use the calling api to create the corresponding project.
-<p align="center">
- <img src="/img/api/create_project2.png" width="80%" />
- </p>
+By consulting the api documentation, configure the KEY as Accept and VALUE as the parameter of application/json in the headers of Postman.
+
+
+
+And then configure the required projectName and description parameters in Body.
+
+
+
+Check the post request result.
+
+
-<p align="center">
- <img src="/img/api/create_project3.png" width="80%" />
- </p>
The returned `msg` information is "success", indicating that we have successfully created the project through API.
If you are interested in the source code of creating a project, please continue to read the following：
### Appendix： The Source Code of Creating a Project
-<p align="center">
- <img src="/img/api/create_source1.png" width="80%" />
- </p>
+
+
+
-<p align="center">
- <img src="/img/api/create_source2.png" width="80%" />
- </p>
diff --git a/docs/docs/zh/guide/open-api.md b/docs/docs/zh/guide/open-api.md
index c334358136..cd01d22868 100644
--- a/docs/docs/zh/guide/open-api.md
+++ b/docs/docs/zh/guide/open-api.md
@@ -1,65 +1,67 @@
# API 调用
## 背景
-一般都是通过页面来创建项目、流程等，但是与第三方系统集成就需要通过调用 API 来管理项目、流程
+
+一般都是通过页面来创建项目、流程等，但是与第三方系统集成就需要通过调用 API 来管理项目、流程。
## 操作步骤
### 创建 token
-1. 登录调度系统，点击 "安全中心"，再点击左侧的 "令牌管理"，点击 "令牌管理" 创建令牌
+1. 登录调度系统，点击 "安全中心"，再点击左侧的 "令牌管理"，点击 "令牌管理" 创建令牌。
-<p align="center">
- <img src="/img/token-management.png" width="80%" />
- </p>
+
-2. 选择 "失效时间" (Token有效期)，选择 "用户" (以指定的用户执行接口操作)，点击 "生成令牌" ，拷贝 Token 字符串，然后点击 "提交"
-<p align="center">
- <img src="/img/create-token.png" width="80%" />
- </p>
-
-### 使用 Token
-1. 打开 API文档页面
- > 地址：http://{api server ip}:12345/dolphinscheduler/doc.html?language=zh_CN&lang=cn
-<p align="center">
- <img src="/img/api-documentation.png" width="80%" />
- </p>
+2. 选择 "失效时间" (Token 有效期)，选择 "用户" (以指定的用户执行接口操作)，点击 "生成令牌" ，拷贝 Token 字符串，然后点击 "提交" 。
+
+
+
+### 使用案例
+
+#### 查询项目列表信息
+
+1. 打开 API 文档页面
+
+> 地址：http://{api server ip}:12345/dolphinscheduler/doc.html?language=zh_CN&lang=cn
+
+
2. 选一个测试的接口，本次测试选取的接口是：查询所有项目
- > projects/query-project-list
+
+> projects/list
3. 打开 Postman，填写接口地址，并在 Headers 中填写 Token，发送请求后即可查看结果
+
```
- token:刚刚生成的Token
+ token: 刚刚生成的 Token
```
-<p align="center">
- <img src="/img/test-api.png" width="80%" />
- </p>
-
-### 创建项目
-这里以创建名为 "wudl-flink-test" 的项目为例
-<p align="center">
- <img src="/img/api/create_project1.png" width="80%" />
- </p>
-
-<p align="center">
- <img src="/img/api/create_project2.png" width="80%" />
- </p>
+
+
-<p align="center">
- <img src="/img/api/create_project3.png" width="80%" />
- </p>
+#### 创建项目
+
+这里演示如何使用调用 api 来创建对应的项目。
+
+通过查阅 api 文档，在 Postman 的 Headers 中配置 KEY 为 Accept，VALUE 为 application/json 的参数。
+
+
+
+然后再 Body 中配置所需的 projectName 和 description 参数。
+
+
+
+检查 post 请求结果。
+
+
+
返回 msg 信息为 "success"，说明我们已经成功通过 API 的方式创建了项目。
如果您对创建项目的源码感兴趣，欢迎继续阅读下面内容
+
### 附:创建项目源码
-<p align="center">
- <img src="/img/api/create_source1.png" width="80%" />
- </p>
+
-<p align="center">
- <img src="/img/api/create_source2.png" width="80%" />
- </p>
+
diff --git a/docs/img/api/create_project1.png b/docs/img/api/create_project1.png
deleted file mode 100644
index 156ea3e00a..0000000000
Binary files a/docs/img/api/create_project1.png and /dev/null differ
diff --git a/docs/img/api/create_project2.png b/docs/img/api/create_project2.png
deleted file mode 100644
index ea9e678904..0000000000
Binary files a/docs/img/api/create_project2.png and /dev/null differ
diff --git a/docs/img/api/create_project3.png b/docs/img/api/create_project3.png
deleted file mode 100644
index c78487ef84..0000000000
Binary files a/docs/img/api/create_project3.png and /dev/null differ
diff --git a/docs/img/api/create_source1.png b/docs/img/api/create_source1.png
deleted file mode 100644
index 614b548758..0000000000
Binary files a/docs/img/api/create_source1.png and /dev/null differ
diff --git a/docs/img/api/create_source2.png b/docs/img/api/create_source2.png
deleted file mode 100644
index 130e7cccfc..0000000000
Binary files a/docs/img/api/create_source2.png and /dev/null differ
diff --git a/docs/img/new_ui/dev/open-api/api_doc.png b/docs/img/new_ui/dev/open-api/api_doc.png
new file mode 100644
index 0000000000..2666d8b99b
Binary files /dev/null and b/docs/img/new_ui/dev/open-api/api_doc.png differ
diff --git a/docs/img/new_ui/dev/open-api/api_source01.png b/docs/img/new_ui/dev/open-api/api_source01.png
new file mode 100644
index 0000000000..2148f166f4
Binary files /dev/null and b/docs/img/new_ui/dev/open-api/api_source01.png differ
diff --git a/docs/img/new_ui/dev/open-api/api_source02.png b/docs/img/new_ui/dev/open-api/api_source02.png
new file mode 100644
index 0000000000..70edbb1a48
Binary files /dev/null and b/docs/img/new_ui/dev/open-api/api_source02.png differ
diff --git a/docs/img/new_ui/dev/open-api/api_test.png b/docs/img/new_ui/dev/open-api/api_test.png
new file mode 100644
index 0000000000..32c885bc6e
Binary files /dev/null and b/docs/img/new_ui/dev/open-api/api_test.png differ
diff --git a/docs/img/new_ui/dev/open-api/create_project01.png b/docs/img/new_ui/dev/open-api/create_project01.png
new file mode 100644
index 0000000000..c03f1102db
Binary files /dev/null and b/docs/img/new_ui/dev/open-api/create_project01.png differ
diff --git a/docs/img/new_ui/dev/open-api/create_project02.png b/docs/img/new_ui/dev/open-api/create_project02.png
new file mode 100644
index 0000000000..61a26ac072
Binary files /dev/null and b/docs/img/new_ui/dev/open-api/create_project02.png differ
diff --git a/docs/img/new_ui/dev/open-api/create_project03.png b/docs/img/new_ui/dev/open-api/create_project03.png
new file mode 100644
index 0000000000..732e240203
Binary files /dev/null and b/docs/img/new_ui/dev/open-api/create_project03.png differ
diff --git a/docs/img/new_ui/dev/open-api/token_expiration.png b/docs/img/new_ui/dev/open-api/token_expiration.png
new file mode 100644
index 0000000000..9a75cad5a2
Binary files /dev/null and b/docs/img/new_ui/dev/open-api/token_expiration.png differ
[dolphinscheduler] 24/33: [doc] Update editorial workflow content and images (#10130)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 05f209d5c070d9198b4a4f9765da9ab97fa9ce5e
Author: Mr.An <42...@users.noreply.github.com>
AuthorDate: Fri May 20 16:44:56 2022 +0800
[doc] Update editorial workflow content and images (#10130)
(cherry picked from commit 7d5af7e7ec0efc01c81dab0384f7f80e103f0ec1)
---
docs/docs/en/guide/project/workflow-instance.md | 2 +-
docs/docs/zh/guide/project/workflow-instance.md | 2 +-
docs/img/editDag-en.png | Bin 421677 -> 1300064 bytes
docs/img/editDag.png | Bin 26907 -> 1300064 bytes
docs/img/new_ui/dev/project/project-overview.png | Bin 167215 -> 790835 bytes
dolphinscheduler-ui/src/locales/modules/en_US.ts | 4 ++--
dolphinscheduler-ui/src/locales/modules/zh_CN.ts | 4 ++--
7 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/docs/docs/en/guide/project/workflow-instance.md b/docs/docs/en/guide/project/workflow-instance.md
index bba65b9692..004322e1ab 100644
--- a/docs/docs/en/guide/project/workflow-instance.md
+++ b/docs/docs/en/guide/project/workflow-instance.md
@@ -40,7 +40,7 @@ Click Project Management -> Workflow -> Workflow Instance, enter the workflow in
-- **Edit：** only can edit terminated processes. Click the "Edit" button or the workflow instance name to enter the DAG edit page. After the edit, click the "Save" button to confirm, as shown in the figure below. In the pop-up box, check "Whether to update to workflow definition" to update the workflow definition; if not checked, no effect on the workflow definition
+- **Edit：** only processes with success/failed/stop status can be edited. Click the "Edit" button or the workflow instance name to enter the DAG edit page. After the edit, click the "Save" button to confirm, as shown in the figure below. In the pop-up box, check "Whether to update the workflow definition", after saving, the information modified by the instance will be updated to the workflow definition; if not checked, the workflow definition would not be updated.
<p align="center">
<img src="/img/editDag-en.png" width="80%" />
</p>
diff --git a/docs/docs/zh/guide/project/workflow-instance.md b/docs/docs/zh/guide/project/workflow-instance.md
index 4f4d0a5dca..1ca44c862d 100644
--- a/docs/docs/zh/guide/project/workflow-instance.md
+++ b/docs/docs/zh/guide/project/workflow-instance.md
@@ -40,7 +40,7 @@
-- **编辑：** 只能编辑已终止的流程。点击"编辑"按钮或工作流实例名称进入 DAG 编辑页面，编辑后点击"保存"按钮，弹出保存 DAG 弹框，如下图所示，在弹框中勾选"是否更新到工作流定义"，保存后则更新工作流定义；若不勾选，则不更新工作流定义。
+- **编辑：** 只能编辑 成功/失败/停止 状态的流程。点击"编辑"按钮或工作流实例名称进入 DAG 编辑页面，编辑后点击"保存"按钮，弹出保存 DAG 弹框，如下图所示，修改流程定义信息，在弹框中勾选"是否更新工作流定义"，保存后则将实例修改的信息更新到工作流定义；若不勾选，则不更新工作流定义。
<p align="center">
<img src="/img/editDag.png" width="80%" />
</p>
diff --git a/docs/img/editDag-en.png b/docs/img/editDag-en.png
index b3b62ae405..420ad6d9fa 100644
Binary files a/docs/img/editDag-en.png and b/docs/img/editDag-en.png differ
diff --git a/docs/img/editDag.png b/docs/img/editDag.png
index dbf8403d95..420ad6d9fa 100644
Binary files a/docs/img/editDag.png and b/docs/img/editDag.png differ
diff --git a/docs/img/new_ui/dev/project/project-overview.png b/docs/img/new_ui/dev/project/project-overview.png
index 67c5ee7cdc..8b2a5f899c 100644
Binary files a/docs/img/new_ui/dev/project/project-overview.png and b/docs/img/new_ui/dev/project/project-overview.png differ
diff --git a/dolphinscheduler-ui/src/locales/modules/en_US.ts b/dolphinscheduler-ui/src/locales/modules/en_US.ts
index aabd1c4cc4..f52932aa12 100644
--- a/dolphinscheduler-ui/src/locales/modules/en_US.ts
+++ b/dolphinscheduler-ui/src/locales/modules/en_US.ts
@@ -604,8 +604,8 @@ const project = {
value: 'Value',
success: 'Success',
delete_cell: 'Delete selected edges and nodes',
- online_directly: 'Whether to go online the process definition',
- update_directly: 'Whether to update the process definition',
+ online_directly: 'Whether to go online the workflow definition',
+ update_directly: 'Whether to update the workflow definition',
dag_name_empty: 'DAG graph name cannot be empty',
positive_integer: 'Please enter a positive integer greater than 0',
prop_empty: 'prop is empty',
diff --git a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
index f0f176cacb..b39d9df0d3 100644
--- a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
+++ b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
@@ -601,8 +601,8 @@ const project = {
value: '值',
success: '成功',
delete_cell: '删除选中的线或节点',
- online_directly: '是否上线流程定义',
- update_directly: '是否更新流程定义',
+ online_directly: '是否上线工作流定义',
+ update_directly: '是否更新工作流定义',
dag_name_empty: 'DAG图名称不能为空',
positive_integer: '请输入大于 0 的正整数',
prop_empty: '自定义参数prop不能为空',
[dolphinscheduler] 08/33: [Fix-10082]Resource userName is null and udf function manager add a username (#10064)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 63ab98e67f64ee780de81ad2b06c6a6a05fa997d
Author: hstdream <33...@users.noreply.github.com>
AuthorDate: Wed May 18 11:33:08 2022 +0800
[Fix-10082]Resource userName is null and udf function manager add a username (#10064)
* Resource userName is null
* [Fix-10082]Resource userName is null and udf function manager add a username
* [Fix-10082]Resource userName is null and udf function manager add a username
* [Fix-10082]Resource userName is null and udf function manager add a username
Co-authored-by: houshitao <sh...@163.com>
(cherry picked from commit 701b535ae6a9fdd7c67797528fb9ff8f650db514)
---
.../apache/dolphinscheduler/dao/entity/Resource.java | 17 +++++++++++++++++
.../org/apache/dolphinscheduler/dao/entity/UdfFunc.java | 14 ++++++++++++++
.../dolphinscheduler/dao/mapper/ResourceMapper.xml | 5 +++--
.../dolphinscheduler/dao/mapper/UdfFuncMapper.xml | 5 +++--
.../apache/dolphinscheduler/dao/entity/UdfFuncTest.java | 2 +-
5 files changed, 38 insertions(+), 5 deletions(-)
diff --git a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/Resource.java b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/Resource.java
index 750fbb4e05..1cafe09291 100644
--- a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/Resource.java
+++ b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/Resource.java
@@ -17,6 +17,7 @@
package org.apache.dolphinscheduler.dao.entity;
+import com.baomidou.mybatisplus.annotation.TableField;
import org.apache.dolphinscheduler.spi.enums.ResourceType;
import java.util.Date;
@@ -88,6 +89,13 @@ public class Resource {
*/
private Date updateTime;
+ /**
+ * user name
+ */
+ @TableField(exist = false)
+ private String userName;
+
+
public Resource() {
}
@@ -224,6 +232,14 @@ public class Resource {
this.updateTime = updateTime;
}
+ public String getUserName() {
+ return userName;
+ }
+
+ public void setUserName(String userName) {
+ this.userName = userName;
+ }
+
@Override
public String toString() {
return "Resource{" +
@@ -239,6 +255,7 @@ public class Resource {
", size=" + size +
", createTime=" + createTime +
", updateTime=" + updateTime +
+ ",userName=" + userName +
'}';
}
diff --git a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/UdfFunc.java b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/UdfFunc.java
index 949499d1e8..fb6e23b678 100644
--- a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/UdfFunc.java
+++ b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/UdfFunc.java
@@ -105,6 +105,12 @@ public class UdfFunc {
*/
private Date updateTime;
+ /**
+ * user name
+ */
+ @TableField(exist = false)
+ private String userName;
+
public int getId() {
return id;
}
@@ -202,6 +208,14 @@ public class UdfFunc {
this.updateTime = updateTime;
}
+ public String getUserName() {
+ return userName;
+ }
+
+ public void setUserName(String userName) {
+ this.userName = userName;
+ }
+
@Override
public boolean equals(Object o) {
if (this == o) {
diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ResourceMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ResourceMapper.xml
index 79e5d4cdb8..95620abb72 100644
--- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ResourceMapper.xml
+++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ResourceMapper.xml
@@ -58,8 +58,9 @@
<include refid="baseSqlV2">
<property name="alias" value="d"/>
</include>
- from t_ds_resources d
- where d.type=#{type} and d.pid=#{id}
+ ,u.user_name
+ from t_ds_resources d,t_ds_user u
+ where d.type=#{type} and d.pid=#{id} and d.user_id=u.id
<if test="userId != 0">
and (
d.user_id=#{userId}
diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/UdfFuncMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/UdfFuncMapper.xml
index 25cdfdd642..1f9e09e4d0 100644
--- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/UdfFuncMapper.xml
+++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/UdfFuncMapper.xml
@@ -56,8 +56,9 @@
<include refid="baseSql">
<property name="alias" value="udf"/>
</include>
- from t_ds_udfs udf
- where 1=1
+ ,u.user_name
+ from t_ds_udfs udf,t_ds_user u
+ where 1=1 and udf.user_id=u.id
<if test="searchVal!= null and searchVal != ''">
and udf.func_name like concat('%', #{searchVal}, '%')
</if>
diff --git a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/entity/UdfFuncTest.java b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/entity/UdfFuncTest.java
index 4231ba35f6..38479e9849 100644
--- a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/entity/UdfFuncTest.java
+++ b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/entity/UdfFuncTest.java
@@ -37,7 +37,7 @@ public class UdfFuncTest {
udfFunc.setClassName("org.apache.dolphinscheduler.test.mrUpdate");
Assert.assertEquals("{\"id\":0,\"userId\":0,\"resourceType\":\"UDF\",\"funcName\":null,\"className\":\"org.apache.dolphinscheduler.test.mrUpdate\",\"argTypes\":null,\"database\":null,"
- + "\"description\":null,\"resourceId\":2,\"resourceName\":\"dolphin_resource_update\",\"type\":null,\"createTime\":null,\"updateTime\":null}"
+ + "\"description\":null,\"resourceId\":2,\"resourceName\":\"dolphin_resource_update\",\"type\":null,\"createTime\":null,\"updateTime\":null,\"userName\":null}"
, udfFunc.toString());
}
[dolphinscheduler] 31/33: [Fix-10076] queryUserByToken use server timezone (#10177)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit bb30b88306e33fde05fb72b74b23597c5fe1f277
Author: longtb <67...@users.noreply.github.com>
AuthorDate: Mon May 23 09:54:11 2022 +0800
[Fix-10076] queryUserByToken use server timezone (#10177)
* [Fix-10076] queryUserByToken use server timezone
* [Fix-10076] queryUserByToken use server timezone
(cherry picked from commit bd34e66b8a493a2aa14af5884f2f7c13819004f2)
---
.../dolphinscheduler/api/interceptor/LoginHandlerInterceptor.java | 4 +++-
.../api/interceptor/LoginHandlerInterceptorTest.java | 6 +++++-
.../java/org/apache/dolphinscheduler/dao/mapper/UserMapper.java | 6 ++++--
.../resources/org/apache/dolphinscheduler/dao/mapper/UserMapper.xml | 2 +-
.../java/org/apache/dolphinscheduler/dao/mapper/UserMapperTest.java | 2 +-
5 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/LoginHandlerInterceptor.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/LoginHandlerInterceptor.java
index 2013348ed8..243928960b 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/LoginHandlerInterceptor.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/LoginHandlerInterceptor.java
@@ -37,6 +37,8 @@ import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.servlet.HandlerInterceptor;
import org.springframework.web.servlet.ModelAndView;
+import java.util.Date;
+
/**
* login interceptor, must log in first
*/
@@ -71,7 +73,7 @@ public class LoginHandlerInterceptor implements HandlerInterceptor {
return false;
}
} else {
- user = userMapper.queryUserByToken(token);
+ user = userMapper.queryUserByToken(token, new Date());
if (user == null) {
response.setStatus(HttpStatus.SC_UNAUTHORIZED);
logger.info("user token has expired");
diff --git a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/interceptor/LoginHandlerInterceptorTest.java b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/interceptor/LoginHandlerInterceptorTest.java
index 67d34192d4..467cfa0e07 100644
--- a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/interceptor/LoginHandlerInterceptorTest.java
+++ b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/interceptor/LoginHandlerInterceptorTest.java
@@ -17,6 +17,8 @@
package org.apache.dolphinscheduler.api.interceptor;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.when;
import org.apache.dolphinscheduler.api.ApiApplicationServer;
@@ -45,6 +47,8 @@ import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.transaction.annotation.Transactional;
+import java.util.Date;
+
@ActiveProfiles(value = {ProfileType.H2})
@RunWith(SpringRunner.class)
@SpringBootTest(classes = ApiApplicationServer.class)
@@ -81,7 +85,7 @@ public class LoginHandlerInterceptorTest {
// test token
String token = "123456";
when(request.getHeader("token")).thenReturn(token);
- when(userMapper.queryUserByToken(token)).thenReturn(mockUser);
+ when(userMapper.queryUserByToken(eq(token), any(Date.class))).thenReturn(mockUser);
Assert.assertTrue(interceptor.preHandle(request, response, null));
// test disable user
diff --git a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/UserMapper.java b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/UserMapper.java
index bd611417c1..d31f4b460b 100644
--- a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/UserMapper.java
+++ b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/UserMapper.java
@@ -21,6 +21,7 @@ import org.apache.dolphinscheduler.dao.entity.User;
import org.apache.ibatis.annotations.Param;
+import java.util.Date;
import java.util.List;
import org.springframework.cache.annotation.CacheConfig;
@@ -83,7 +84,7 @@ public interface UserMapper extends BaseMapper<User> {
/**
* user page
*
- * @param page page
+ * @param page page
* @param userName userName
* @return user IPage
*/
@@ -126,9 +127,10 @@ public interface UserMapper extends BaseMapper<User> {
* query user by token
*
* @param token token
+ * @param now now date
* @return user
*/
- User queryUserByToken(@Param("token") String token);
+ User queryUserByToken(@Param("token") String token, @Param("now") Date now);
/**
* query user by queue name
diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/UserMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/UserMapper.xml
index b105679970..e38688939c 100644
--- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/UserMapper.xml
+++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/UserMapper.xml
@@ -97,7 +97,7 @@
<property name="alias" value="u"/>
</include>
from t_ds_user u ,t_ds_access_token t
- where u.id = t.user_id and token=#{token} and t.expire_time > NOW()
+ where u.id = t.user_id and token=#{token} and t.expire_time > #{now}
</select>
<select id="queryUserListByQueue" resultType="org.apache.dolphinscheduler.dao.entity.User">
select
diff --git a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/UserMapperTest.java b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/UserMapperTest.java
index 0c66dd2486..1fd07f94c4 100644
--- a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/UserMapperTest.java
+++ b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/UserMapperTest.java
@@ -303,7 +303,7 @@ public class UserMapperTest extends BaseDaoTest {
//insertOneAccessToken
AccessToken accessToken = insertOneAccessToken(user);
//queryUserByToken
- User userToken = userMapper.queryUserByToken(accessToken.getToken());
+ User userToken = userMapper.queryUserByToken(accessToken.getToken(), new Date());
Assert.assertEquals(userToken, user);
}
[dolphinscheduler] 03/33: [Fix][UI] Fix bug where username is empty in file manage (#10070)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit c2abc650a2f54395800afc36c741567bc42015b5
Author: Devosend <de...@gmail.com>
AuthorDate: Tue May 17 15:17:46 2022 +0800
[Fix][UI] Fix bug where username is empty in file manage (#10070)
(cherry picked from commit c16424027e369c8cde2da966f08ded06ba0744df)
---
dolphinscheduler-ui/src/service/modules/resources/types.ts | 1 +
dolphinscheduler-ui/src/views/resource/file/use-file.ts | 1 +
2 files changed, 2 insertions(+)
diff --git a/dolphinscheduler-ui/src/service/modules/resources/types.ts b/dolphinscheduler-ui/src/service/modules/resources/types.ts
index 71edfa4943..85c6fef9f8 100644
--- a/dolphinscheduler-ui/src/service/modules/resources/types.ts
+++ b/dolphinscheduler-ui/src/service/modules/resources/types.ts
@@ -95,6 +95,7 @@ interface ResourceFile {
id: number
pid: number
alias: string
+ userName: string
userId: number
type: string
directory: boolean
diff --git a/dolphinscheduler-ui/src/views/resource/file/use-file.ts b/dolphinscheduler-ui/src/views/resource/file/use-file.ts
index e2727bfeb6..b16fb2b744 100644
--- a/dolphinscheduler-ui/src/views/resource/file/use-file.ts
+++ b/dolphinscheduler-ui/src/views/resource/file/use-file.ts
@@ -49,6 +49,7 @@ export function useFileState(
alias: item.alias,
fullName: item.fullName,
type: item.type,
+ user_name: item.userName,
directory: item.directory,
file_name: item.fileName,
description: item.description,
[dolphinscheduler] 29/33: [Fix][UI] Add seatunnel icon (#10178)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 180ee3ba95d4f75de5301ba1f0a7e26471b7de20
Author: songjianet <17...@qq.com>
AuthorDate: Sat May 21 14:36:27 2022 +0800
[Fix][UI] Add seatunnel icon (#10178)
(cherry picked from commit 0ef02e80935d9f6588233186e01ae5ca7f8b1e7a)
---
.../public/images/task-icons/seatunnel.png | Bin 1086 -> 5405 bytes
.../public/images/task-icons/seatunnel_hover.png | Bin 1054 -> 5396 bytes
2 files changed, 0 insertions(+), 0 deletions(-)
diff --git a/dolphinscheduler-ui/public/images/task-icons/seatunnel.png b/dolphinscheduler-ui/public/images/task-icons/seatunnel.png
index bf2f83e206..4b11920c5f 100644
Binary files a/dolphinscheduler-ui/public/images/task-icons/seatunnel.png and b/dolphinscheduler-ui/public/images/task-icons/seatunnel.png differ
diff --git a/dolphinscheduler-ui/public/images/task-icons/seatunnel_hover.png b/dolphinscheduler-ui/public/images/task-icons/seatunnel_hover.png
index e6affd6ff1..74616c2021 100644
Binary files a/dolphinscheduler-ui/public/images/task-icons/seatunnel_hover.png and b/dolphinscheduler-ui/public/images/task-icons/seatunnel_hover.png differ
[dolphinscheduler] 33/33: [doc] Update project directory structure in docs (#10107)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 68e6e45bb8f740e95e91b909b001a1cf6b614af3
Author: Eric Gao <er...@gmail.com>
AuthorDate: Mon May 23 11:11:45 2022 +0800
[doc] Update project directory structure in docs (#10107)
(cherry picked from commit 7a8f3f1810ec283fc5acd66077d46db88b0400af)
---
docs/docs/en/architecture/configuration.md | 119 ++++++++++++++++++---------
docs/docs/zh/architecture/configuration.md | 124 +++++++++++++++++++----------
2 files changed, 163 insertions(+), 80 deletions(-)
diff --git a/docs/docs/en/architecture/configuration.md b/docs/docs/en/architecture/configuration.md
index b56ed1b119..e3689f9ae9 100644
--- a/docs/docs/en/architecture/configuration.md
+++ b/docs/docs/en/architecture/configuration.md
@@ -15,44 +15,87 @@ This document only describes DolphinScheduler configurations and other topics ar
[Note: the DolphinScheduler (hereinafter called the ‘DS’) .]
```
-├─bin DS application commands directory
-│ ├─dolphinscheduler-daemon.sh startup or shutdown DS application
-│ ├─start-all.sh startup all DS services with configurations
-│ ├─stop-all.sh shutdown all DS services with configurations
-├─conf configurations directory
-│ ├─application-api.properties API-service config properties
-│ ├─datasource.properties datasource config properties
-│ ├─zookeeper.properties ZooKeeper config properties
-│ ├─master.properties master-service config properties
-│ ├─worker.properties worker-service config properties
-│ ├─quartz.properties quartz config properties
-│ ├─common.properties common-service [storage] config properties
-│ ├─alert.properties alert-service config properties
-│ ├─config environment variables config directory
-│ ├─install_config.conf DS environment variables configuration script [install or start DS]
-│ ├─env load environment variables configs script directory
-│ ├─dolphinscheduler_env.sh load environment variables configs [eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
-│ ├─org mybatis mapper files directory
-│ ├─i18n i18n configs directory
-│ ├─logback-api.xml API-service log config
-│ ├─logback-master.xml master-service log config
-│ ├─logback-worker.xml worker-service log config
-│ ├─logback-alert.xml alert-service log config
-├─sql .sql files to create or upgrade DS metadata
-│ ├─create create SQL scripts directory
-│ ├─upgrade upgrade SQL scripts directory
-│ ├─dolphinscheduler_postgre.sql PostgreSQL database init script
-│ ├─dolphinscheduler_mysql.sql MySQL database init script
-│ ├─soft_version current DS version-id file
-├─script DS services deployment, database create or upgrade scripts directory
-│ ├─create-dolphinscheduler.sh DS database init script
-│ ├─upgrade-dolphinscheduler.sh DS database upgrade script
-│ ├─monitor-server.sh DS monitor-server start script
-│ ├─scp-hosts.sh transfer installation files script
-│ ├─remove-zk-node.sh cleanup ZooKeeper caches script
-├─ui front-end web resources directory
-├─lib DS .jar dependencies directory
-├─install.sh auto-setup DS services script
+├── LICENSE
+│
+├── NOTICE
+│
+├── licenses directory of licenses
+│
+├── bin directory of DolphinScheduler application commands, configrations scripts
+│ ├── dolphinscheduler-daemon.sh script to start or shut down DolphinScheduler application
+│ ├── env directory of scripts to load environment variables
+│ │ ├── dolphinscheduler_env.sh script to export environment variables [eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...] when you start or stop service using script `dolphinscheduler-daemon.sh`
+│ │ └── install_env.sh script to export environment variables for DolphinScheduler installation when you use scripts `install.sh` `start-all.sh` `stop-all.sh` `status-all.sh`
+│ ├── install.sh script to auto-setup services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
+│ ├── remove-zk-node.sh script to cleanup ZooKeeper caches
+│ ├── scp-hosts.sh script to copy installation files to target hosts
+│ ├── start-all.sh script to start all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
+│ ├── status-all.sh script to check the status of all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
+│ └── stop-all.sh script to shut down all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
+│
+├── alert-server directory of DolphinScheduler alert-server commands, configrations scripts and libs
+│ ├── bin
+│ │ └── start.sh script to start DolphinScheduler alert-server
+│ ├── conf
+│ │ ├── application.yaml configurations of alert-server
+│ │ ├── common.properties configurations of common-service like storage, credentials, etc.
+│ │ ├── dolphinscheduler_env.sh script to load environment variables for alert-server
+│ │ └── logback-spring.xml configurations of alert-service log
+│ └── libs directory of alert-server libs
+│
+├── api-server directory of DolphinScheduler api-server commands, configrations scripts and libs
+│ ├── bin
+│ │ └── start.sh script to start DolphinScheduler api-server
+│ ├── conf
+│ │ ├── application.yaml configurations of api-server
+│ │ ├── common.properties configurations of common-service like storage, credentials, etc.
+│ │ ├── dolphinscheduler_env.sh script to load environment variables for api-server
+│ │ └── logback-spring.xml configurations of api-service log
+│ ├── libs directory of api-server libs
+│ └── ui directory of api-server related front-end web resources
+│
+├── master-server directory of DolphinScheduler master-server commands, configrations scripts and libs
+│ ├── bin
+│ │ └── start.sh script to start DolphinScheduler master-server
+│ ├── conf
+│ │ ├── application.yaml configurations of master-server
+│ │ ├── common.properties configurations of common-service like storage, credentials, etc.
+│ │ ├── dolphinscheduler_env.sh script to load environment variables for master-server
+│ │ └── logback-spring.xml configurations of master-service log
+│ └── libs directory of master-server libs
+│
+├── standalone-server directory of DolphinScheduler standalone-server commands, configrations scripts and libs
+│ ├── bin
+│ │ └── start.sh script to start DolphinScheduler standalone-server
+│ ├── conf
+│ │ ├── application.yaml configurations of standalone-server
+│ │ ├── common.properties configurations of common-service like storage, credentials, etc.
+│ │ ├── dolphinscheduler_env.sh script to load environment variables for standalone-server
+│ │ ├── logback-spring.xml configurations of standalone-service log
+│ │ └── sql .sql files to create or upgrade DolphinScheduler metadata
+│ ├── libs directory of standalone-server libs
+│ └── ui directory of standalone-server related front-end web resources
+│
+├── tools directory of DolphinScheduler metadata tools commands, configrations scripts and libs
+│ ├── bin
+│ │ └── upgrade-schema.sh script to initialize or upgrade DolphinScheduler metadata
+│ ├── conf
+│ │ ├── application.yaml configurations of tools
+│ │ └── common.properties configurations of common-service like storage, credentials, etc.
+│ ├── libs directory of tool libs
+│ └── sql .sql files to create or upgrade DolphinScheduler metadata
+│
+├── worker-server directory of DolphinScheduler worker-server commands, configrations scripts and libs
+│ ├── bin
+│ │ └── start.sh script to start DolphinScheduler worker-server
+│ ├── conf
+│ │ ├── application.yaml configurations of worker-server
+│ │ ├── common.properties configurations of common-service like storage, credentials, etc.
+│ │ ├── dolphinscheduler_env.sh script to load environment variables for worker-server
+│ │ └── logback-spring.xml configurations of worker-service log
+│ └── libs directory of worker-server libs
+│
+└── ui directory of front-end web resources
```
## Configurations in Details
diff --git a/docs/docs/zh/architecture/configuration.md b/docs/docs/zh/architecture/configuration.md
index 7eba76ea2c..698a1c4aa2 100644
--- a/docs/docs/zh/architecture/configuration.md
+++ b/docs/docs/zh/architecture/configuration.md
@@ -9,50 +9,90 @@
本文主要讲述dolphinscheduler的配置文件.其他部分先不做赘述.
[注:以下 dolphinscheduler 简称为DS.]
-```
-
-├─bin DS命令存放目录
-│ ├─dolphinscheduler-daemon.sh 启动/关闭DS服务脚本
-│ ├─start-all.sh 根据配置文件启动所有DS服务
-│ ├─stop-all.sh 根据配置文件关闭所有DS服务
-├─conf 配置文件目录
-│ ├─application-api.properties api服务配置文件
-│ ├─datasource.properties 数据库配置文件
-│ ├─zookeeper.properties zookeeper配置文件
-│ ├─master.properties master服务配置文件
-│ ├─worker.properties worker服务配置文件
-│ ├─quartz.properties quartz服务配置文件
-│ ├─common.properties 公共服务[存储]配置文件
-│ ├─alert.properties alert服务配置文件
-│ ├─config 环境变量配置文件夹
-│ ├─install_config.conf DS环境变量配置脚本[用于DS安装/启动]
-│ ├─env 运行脚本环境变量配置目录
-│ ├─dolphinscheduler_env.sh 运行脚本加载环境变量配置文件[如: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
-│ ├─org mybatis mapper文件目录
-│ ├─i18n i18n配置文件目录
-│ ├─logback-api.xml api服务日志配置文件
-│ ├─logback-master.xml master服务日志配置文件
-│ ├─logback-worker.xml worker服务日志配置文件
-│ ├─logback-alert.xml alert服务日志配置文件
-├─sql DS的元数据创建升级sql文件
-│ ├─create 创建SQL脚本目录
-│ ├─upgrade 升级SQL脚本目录
-│ ├─dolphinscheduler_postgre.sql postgre数据库初始化脚本
-│ ├─dolphinscheduler_mysql.sql mysql数据库初始化脚本
-│ ├─soft_version 当前DS版本标识文件
-├─script DS服务部署,数据库创建/升级脚本目录
-│ ├─create-dolphinscheduler.sh DS数据库初始化脚本
-│ ├─upgrade-dolphinscheduler.sh DS数据库升级脚本
-│ ├─monitor-server.sh DS服务监控启动脚本
-│ ├─scp-hosts.sh 安装文件传输脚本
-│ ├─remove-zk-node.sh 清理zookeeper缓存文件脚本
-├─ui 前端WEB资源目录
-├─lib DS依赖的jar存放目录
-├─install.sh 自动安装DS服务脚本
-
```
-
+├── LICENSE
+│
+├── NOTICE
+│
+├── licenses licenses存放目录
+│
+├── bin DolphinScheduler命令和环境变量配置存放目录
+│ ├── dolphinscheduler-daemon.sh 启动/关闭DolphinScheduler服务脚本
+│ ├── env 环境变量配置存放目录
+│ │ ├── dolphinscheduler_env.sh 当使用`dolphinscheduler-daemon.sh`脚本起停服务时，运行此脚本加载环境变量配置文件 [如：JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
+│ │ └── install_env.sh 当使用`install.sh` `start-all.sh` `stop-all.sh` `status-all.sh`脚本时，运行此脚本为DolphinScheduler安装加载环境变量配置
+│ ├── install.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本自动安装服务
+│ ├── remove-zk-node.sh 清理zookeeper缓存文件脚本
+│ ├── scp-hosts.sh 安装文件传输脚本
+│ ├── start-all.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本启动所有服务
+│ ├── status-all.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本获取所有服务状态
+│ └── stop-all.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本终止所有服务
+│
+├── alert-server DolphinScheduler alert-server命令、配置和依赖存放目录
+│ ├── bin
+│ │ └── start.sh DolphinScheduler alert-server启动脚本
+│ ├── conf
+│ │ ├── application.yaml alert-server配置文件
+│ │ ├── common.properties 公共服务（存储等信息）配置文件
+│ │ ├── dolphinscheduler_env.sh alert-server环境变量配置加载脚本
+│ │ └── logback-spring.xml alert-service日志配置文件
+│ └── libs alert-server依赖jar包存放目录
+│
+├── api-server DolphinScheduler api-server命令、配置和依赖存放目录
+│ ├── bin
+│ │ └── start.sh DolphinScheduler api-server启动脚本
+│ ├── conf
+│ │ ├── application.yaml api-server配置文件
+│ │ ├── common.properties 公共服务（存储等信息）配置文件
+│ │ ├── dolphinscheduler_env.sh api-server环境变量配置加载脚本
+│ │ └── logback-spring.xml api-service日志配置文件
+│ ├── libs api-server依赖jar包存放目录
+│ └── ui api-server相关前端WEB资源存放目录
+│
+├── master-server DolphinScheduler master-server命令、配置和依赖存放目录
+│ ├── bin
+│ │ └── start.sh DolphinScheduler master-server启动脚本
+│ ├── conf
+│ │ ├── application.yaml master-server配置文件
+│ │ ├── common.properties 公共服务（存储等信息）配置文件
+│ │ ├── dolphinscheduler_env.sh master-server环境变量配置加载脚本
+│ │ └── logback-spring.xml master-service日志配置文件
+│ └── libs master-server依赖jar包存放目录
+│
+├── standalone-server DolphinScheduler standalone-server命令、配置和依赖存放目录
+│ ├── bin
+│ │ └── start.sh DolphinScheduler standalone-server启动脚本
+│ ├── conf
+│ │ ├── application.yaml standalone-server配置文件
+│ │ ├── common.properties 公共服务（存储等信息）配置文件
+│ │ ├── dolphinscheduler_env.sh standalone-server环境变量配置加载脚本
+│ │ ├── logback-spring.xml standalone-service日志配置文件
+│ │ └── sql DolphinScheduler元数据创建/升级sql文件
+│ ├── libs standalone-server依赖jar包存放目录
+│ └── ui standalone-server相关前端WEB资源存放目录
+│
+├── tools DolphinScheduler元数据工具命令、配置和依赖存放目录
+│ ├── bin
+│ │ └── upgrade-schema.sh DolphinScheduler元数据创建/升级脚本
+│ ├── conf
+│ │ ├── application.yaml 元数据工具配置文件
+│ │ └── common.properties 公共服务（存储等信息）配置文件
+│ ├── libs 元数据工具依赖jar包存放目录
+│ └── sql DolphinScheduler元数据创建/升级sql文件
+│
+├── worker-server DolphinScheduler worker-server命令、配置和依赖存放目录
+│ ├── bin
+│ │ └── start.sh DolphinScheduler worker-server启动脚本
+│ ├── conf
+│ │ ├── application.yaml worker-server配置文件
+│ │ ├── common.properties 公共服务（存储等信息）配置文件
+│ │ ├── dolphinscheduler_env.sh worker-server环境变量配置加载脚本
+│ │ └── logback-spring.xml worker-service日志配置文件
+│ └── libs worker-server依赖jar包存放目录
+│
+└── ui 前端WEB资源目录
+```
# 配置文件详解
[dolphinscheduler] 05/33: [Fix-10080]When the created tenant name is too long, an error message will be prompted (#10081)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 4f11be7126bf1fa6f2eee2a9ccfde6bfde0b1dc6
Author: hstdream <33...@users.noreply.github.com>
AuthorDate: Wed May 18 01:36:40 2022 +0800
[Fix-10080]When the created tenant name is too long, an error message will be prompted (#10081)
* [Fix-10080]When the created tenant name is too long, an error message will be prompted
* [Fix-10080]When the created tenant name is too long, an error message will be prompted
* [Fix-10080]When the created tenant name is too long, an error message will be prompted
Co-authored-by: houshitao <sh...@163.com>
(cherry picked from commit 67cf7b280087d4424bf0c2a698c9c12ab874bd2e)
---
.../main/java/org/apache/dolphinscheduler/api/enums/Status.java | 3 ++-
.../dolphinscheduler/api/service/impl/TenantServiceImpl.java | 9 ++++++++-
.../main/java/org/apache/dolphinscheduler/common/Constants.java | 5 +++++
3 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
index a755a75e1c..b315c313cb 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java
@@ -401,7 +401,8 @@ public enum Status {
QUERY_UNAUTHORIZED_NAMESPACE_ERROR(1300012, "query unauthorized namespace error", "查询未授权命名空间错误"),
QUERY_AUTHORIZED_NAMESPACE_ERROR(1300013, "query authorized namespace error", "查询授权命名空间错误"),
QUERY_CAN_USE_K8S_CLUSTER_ERROR(1300014, "login user query can used k8s cluster list error", "查询可用k8s集群错误"),
- RESOURCE_FULL_NAME_TOO_LONG_ERROR(1300015, "resource's fullname is too long error", "资源文件名过长");
+ RESOURCE_FULL_NAME_TOO_LONG_ERROR(1300015, "resource's fullname is too long error", "资源文件名过长"),
+ TENANT_FULL_NAME_TOO_LONG_ERROR(1300016, "tenant's fullname is too long error", "租户名过长");
private final int code;
private final String enMsg;
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TenantServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TenantServiceImpl.java
index fb92c1583d..6bb2870452 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TenantServiceImpl.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TenantServiceImpl.java
@@ -40,12 +40,13 @@ import org.apache.dolphinscheduler.dao.mapper.UserMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
-
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import static org.apache.dolphinscheduler.common.Constants.TENANT_FULL_NAME_MAX_LENGTH;
+
/**
* tenant service impl
*/
@@ -89,11 +90,17 @@ public class TenantServiceImpl extends BaseServiceImpl implements TenantService
return result;
}
+ if(StringUtils.length(tenantCode) > TENANT_FULL_NAME_MAX_LENGTH){
+ putMsg(result, Status.TENANT_FULL_NAME_TOO_LONG_ERROR);
+ return result;
+ }
+
if (!RegexUtils.isValidLinuxUserName(tenantCode)) {
putMsg(result, Status.CHECK_OS_TENANT_CODE_ERROR);
return result;
}
+
if (checkTenantExists(tenantCode)) {
putMsg(result, Status.OS_TENANT_CODE_EXIST, tenantCode);
return result;
diff --git a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
index d1d240f28c..57b5c92970 100644
--- a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
+++ b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
@@ -823,4 +823,9 @@ public final class Constants {
*/
public static final String SCHEDULE_TIMEZONE = "schedule_timezone";
public static final int RESOURCE_FULL_NAME_MAX_LENGTH = 128;
+
+ /**
+ * tenant
+ */
+ public static final int TENANT_FULL_NAME_MAX_LENGTH = 30;
}
[dolphinscheduler] 06/33: Create process definition with task group and task group priority cannot be save into db (#10078)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit cb91fc72ee8b530615d1f09bcf4ad86dca163c38
Author: hstdream <33...@users.noreply.github.com>
AuthorDate: Wed May 18 09:41:03 2022 +0800
Create process definition with task group and task group priority cannot be save into db (#10078)
Co-authored-by: houshitao <sh...@163.com>
(cherry picked from commit cd70ad584aae91958a187800a7083868e778d49b)
---
.../apache/dolphinscheduler/dao/mapper/TaskDefinitionLogMapper.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionLogMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionLogMapper.xml
index 212b213607..14314d0fd9 100644
--- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionLogMapper.xml
+++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionLogMapper.xml
@@ -51,7 +51,7 @@
<insert id="batchInsert">
insert into t_ds_task_definition_log (code, name, version, description, project_code, user_id,
task_type, task_params, flag, task_priority, worker_group, environment_code, fail_retry_times, fail_retry_interval,
- timeout_flag, timeout_notify_strategy, timeout, delay_time, resource_ids, operator, operate_time, create_time, update_time,task_group_id)
+ timeout_flag, timeout_notify_strategy, timeout, delay_time, resource_ids, operator, operate_time, create_time, update_time,task_group_id,task_group_priority)
values
<foreach collection="taskDefinitionLogs" item="taskDefinitionLog" separator=",">
(#{taskDefinitionLog.code},#{taskDefinitionLog.name},#{taskDefinitionLog.version},#{taskDefinitionLog.description},
@@ -59,7 +59,7 @@
#{taskDefinitionLog.flag},#{taskDefinitionLog.taskPriority},#{taskDefinitionLog.workerGroup},#{taskDefinitionLog.environmentCode},
#{taskDefinitionLog.failRetryTimes},#{taskDefinitionLog.failRetryInterval},#{taskDefinitionLog.timeoutFlag},#{taskDefinitionLog.timeoutNotifyStrategy},
#{taskDefinitionLog.timeout},#{taskDefinitionLog.delayTime},#{taskDefinitionLog.resourceIds},#{taskDefinitionLog.operator},#{taskDefinitionLog.operateTime},
- #{taskDefinitionLog.createTime},#{taskDefinitionLog.updateTime}, #{taskDefinitionLog.taskGroupId})
+ #{taskDefinitionLog.createTime},#{taskDefinitionLog.updateTime}, #{taskDefinitionLog.taskGroupId},#{taskDefinitionLog.taskGroupPriority})
</foreach>
</insert>
<delete id="deleteByCodeAndVersion">
[dolphinscheduler] 23/33: [Fix][UI] Limit task group resource capacity to only positive integers. (#10158)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit eed50c36709e1b36f7f65e9663e6e1a430e5d935
Author: songjianet <17...@qq.com>
AuthorDate: Fri May 20 15:19:43 2022 +0800
[Fix][UI] Limit task group resource capacity to only positive integers. (#10158)
(cherry picked from commit c07339b07dea2182bf6a3ba1ff0bcc1fe1f4af55)
---
dolphinscheduler-ui/src/locales/modules/en_US.ts | 3 +--
dolphinscheduler-ui/src/locales/modules/zh_CN.ts | 2 +-
.../src/views/resource/task-group/option/components/form-modal.tsx | 6 ++++--
.../src/views/resource/task-group/option/use-form.ts | 6 ++++++
4 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/dolphinscheduler-ui/src/locales/modules/en_US.ts b/dolphinscheduler-ui/src/locales/modules/en_US.ts
index 9c7664c5d2..aabd1c4cc4 100644
--- a/dolphinscheduler-ui/src/locales/modules/en_US.ts
+++ b/dolphinscheduler-ui/src/locales/modules/en_US.ts
@@ -308,8 +308,6 @@ const resource = {
name: 'Task group name',
project_name: 'Project name',
resource_pool_size: 'Resource pool size',
- resource_pool_size_be_a_number:
- 'The size of the task group resource pool should be more than 1',
resource_used_pool_size: 'Used resource',
desc: 'Task group desc',
status: 'Task group status',
@@ -319,6 +317,7 @@ const resource = {
please_enter_desc: 'Please enter task group description',
please_enter_resource_pool_size:
'Please enter task group resource pool size',
+ positive_integer_tips: 'should be a positive integer',
please_select_project: 'Please select a project',
create_time: 'Create time',
update_time: 'Update time',
diff --git a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
index 2d3e34ad39..f0f176cacb 100644
--- a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
+++ b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
@@ -314,7 +314,7 @@ const resource = {
please_enter_name: '请输入任务组名称',
please_enter_desc: '请输入任务组描述',
please_enter_resource_pool_size: '请输入资源容量大小',
- resource_pool_size_be_a_number: '资源容量大小必须大于等于1的数值',
+ positive_integer_tips: '应为正整数',
please_select_project: '请选择项目',
create_time: '创建时间',
update_time: '更新时间',
diff --git a/dolphinscheduler-ui/src/views/resource/task-group/option/components/form-modal.tsx b/dolphinscheduler-ui/src/views/resource/task-group/option/components/form-modal.tsx
index 47be759f58..259d0a361e 100644
--- a/dolphinscheduler-ui/src/views/resource/task-group/option/components/form-modal.tsx
+++ b/dolphinscheduler-ui/src/views/resource/task-group/option/components/form-modal.tsx
@@ -16,7 +16,7 @@
*/
import { defineComponent, PropType, toRefs, onMounted, ref, Ref } from 'vue'
-import { NForm, NFormItem, NInput, NSelect } from 'naive-ui'
+import { NForm, NFormItem, NInput, NSelect, NInputNumber } from 'naive-ui'
import { useForm } from '../use-form'
import Modal from '@/components/modal'
import { createTaskGroup, updateTaskGroup } from '@/service/modules/task-group'
@@ -133,8 +133,10 @@ const FormModal = defineComponent({
label={t('resource.task_group_option.resource_pool_size')}
path='groupSize'
>
- <NInput
+ <NInputNumber
v-model:value={this.formData.groupSize}
+ style={{width: '100%'}}
+ min={1}
placeholder={t(
'resource.task_group_option.please_enter_resource_pool_size'
)}
diff --git a/dolphinscheduler-ui/src/views/resource/task-group/option/use-form.ts b/dolphinscheduler-ui/src/views/resource/task-group/option/use-form.ts
index 6f6f0e69c3..b5a77cf199 100644
--- a/dolphinscheduler-ui/src/views/resource/task-group/option/use-form.ts
+++ b/dolphinscheduler-ui/src/views/resource/task-group/option/use-form.ts
@@ -53,6 +53,12 @@ export function useForm() {
t('resource.task_group_option.please_enter_resource_pool_size')
)
}
+ if (!/^[1-9]\d*$/.test(state.formData.groupSize)) {
+ return new Error(
+ t('resource.task_group_option.resource_pool_size') +
+ t('resource.task_group_option.positive_integer_tips')
+ )
+ }
}
},
projectCode: {
[dolphinscheduler] 26/33: [Fix]Change the HTTP or HTTPS regular expression (#10166)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 713a7802a4bf77751b0f2014dfaf0ee25e5a1fca
Author: labbomb <73...@qq.com>
AuthorDate: Fri May 20 18:05:05 2022 +0800
[Fix]Change the HTTP or HTTPS regular expression (#10166)
(cherry picked from commit 13a83e50f22bb9be2674a2f1edcb7a4193e372b5)
---
.../src/views/projects/task/components/node/fields/use-http.ts | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-http.ts b/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-http.ts
index eaa29fdf1d..855d34c226 100644
--- a/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-http.ts
+++ b/dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-http.ts
@@ -56,7 +56,7 @@ export function useHttp(model: { [field: string]: any }): IJsonItem[] {
return new Error(t('project.node.http_url_tips'))
}
if (
- value.search(new RegExp(/http[s]{0,1}:\/\/([\w.]+\/?)\S*/, 'i'))
+ value.search(new RegExp(/http[s]{0,1}:\/\/\S*/, 'i'))
) {
return new Error(t('project.node.http_url_validator'))
}
[dolphinscheduler] 20/33: [doc] Remove flink call docs (#10122)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 21625f57a734641706811ee6652d3de4aac9333f
Author: Jiajie Zhong <zh...@gmail.com>
AuthorDate: Thu May 19 16:41:33 2022 +0800
[doc] Remove flink call docs (#10122)
Flink call docs is only teld users
how to use flink task in dolphinscheduler
and it should be including in task/flink
(cherry picked from commit 94c515555e90e591acd013aa55d05982ff62efb1)
---
docs/configs/docsdev.js | 8 --
docs/docs/en/guide/flink-call.md | 123 -----------------------------
docs/docs/zh/guide/flink-call.md | 150 ------------------------------------
docs/img/api-documentation-en.png | Bin 77247 -> 0 bytes
docs/img/api-documentation.png | Bin 61869 -> 0 bytes
docs/img/api/create_queue.png | Bin 74021 -> 0 bytes
docs/img/api/create_tenant.png | Bin 82987 -> 0 bytes
docs/img/api/create_user.png | Bin 92945 -> 0 bytes
docs/img/api/create_workflow1.png | Bin 55287 -> 0 bytes
docs/img/api/create_workflow2.png | Bin 96320 -> 0 bytes
docs/img/api/create_workflow3.png | Bin 214012 -> 0 bytes
docs/img/api/create_workflow4.png | Bin 69490 -> 0 bytes
docs/img/api/execution_result.png | Bin 75120 -> 0 bytes
docs/img/api/log.png | Bin 154299 -> 0 bytes
docs/img/api/upload_resource.png | Bin 68267 -> 0 bytes
docs/img/api/user_authorization.png | Bin 38817 -> 0 bytes
docs/img/api/user_login.png | Bin 184975 -> 0 bytes
docs/img/create-token-en1.png | Bin 8779 -> 0 bytes
docs/img/create-token.png | Bin 7699 -> 0 bytes
docs/img/test-api.png | Bin 40028 -> 0 bytes
docs/img/token-management-en.png | Bin 35282 -> 0 bytes
docs/img/token-management.png | Bin 34871 -> 0 bytes
22 files changed, 281 deletions(-)
diff --git a/docs/configs/docsdev.js b/docs/configs/docsdev.js
index cbebe0a81a..ee1f2ded57 100644
--- a/docs/configs/docsdev.js
+++ b/docs/configs/docsdev.js
@@ -276,10 +276,6 @@ export default {
title: 'Data Quality',
link: '/en-us/docs/3.0.0/user_doc/guide/data-quality.html',
},
- {
- title: 'Flink',
- link: '/en-us/docs/3.0.0/user_doc/guide/flink-call.html',
- },
{
title: 'Upgrade',
link: '/en-us/docs/3.0.0/user_doc/guide/upgrade.html',
@@ -632,10 +628,6 @@ export default {
title: '数据质量',
link: '/zh-cn/docs/3.0.0/user_doc/guide/data-quality.html',
},
- {
- title: 'Flink调用',
- link: '/zh-cn/docs/3.0.0/user_doc/guide/flink-call.html',
- },
{
title: '升级',
link: '/zh-cn/docs/3.0.0/user_doc/guide/upgrade.html',
diff --git a/docs/docs/en/guide/flink-call.md b/docs/docs/en/guide/flink-call.md
deleted file mode 100644
index 5f71b69e9a..0000000000
--- a/docs/docs/en/guide/flink-call.md
+++ /dev/null
@@ -1,123 +0,0 @@
-# Flink Calls Operating Steps
-
-## Create a Queue
-
-1. Log in to the scheduling system, click `Security`, then click `Queue manage` on the left, and click `Create queue` to create a queue.
-2. Fill in the name and value of the queue, and click "Submit"
-
-<p align="center">
- <img src="/img/api/create_queue.png" width="80%" />
- </p>
-
-## Create a Tenant
-
-```
-1. The tenant corresponds to a Linux user, which the user worker uses to submit jobs. If the Linux OS environment does not have this user, the worker will create this user when executing the script.
-2. Both the tenant and the tenant code are unique and cannot be repeated, just like a person only has one name and one ID number.
-3. After creating a tenant, there will be a folder in the HDFS relevant directory.
-```
-
-<p align="center">
- <img src="/img/api/create_tenant.png" width="80%" />
- </p>
-
-## Create a User
-
-<p align="center">
- <img src="/img/api/create_user.png" width="80%" />
- </p>
-
-## Create a Token
-
-1. Log in to the scheduling system, click `Security`, then click `Token manage` on the left, and click `Create token` to create a token.
-
-<p align="center">
- <img src="/img/token-management-en.png" width="80%" />
- </p>
-
-
-2. Select the `Expiration time` (token validity time), select `User` (choose the specified user to perform the API operation), click "Generate token", copy the `Token` string, and click "Submit".
-
-<p align="center">
- <img src="/img/create-token-en1.png" width="80%" />
- </p>
-
-## Token Usage
-
-1. Open the API documentation page
-
- > Address：http://{api server ip}:12345/dolphinscheduler/doc.html?language=en_US&lang=en
-
-<p align="center">
- <img src="/img/api-documentation-en.png" width="80%" />
- </p>
-
-
-2. Select a test API, the API selected for this test is `queryAllProjectList`
-
- > projects/query-project-list
-
-3. Open `Postman`, fill in the API address, and enter the `Token` in `Headers`, and then send the request to view the result:
-
- ```
- token: The Token just generated
- ```
-
-<p align="center">
- <img src="/img/test-api.png" width="80%" />
- </p>
-
-## User Authorization
-
-<p align="center">
- <img src="/img/api/user_authorization.png" width="80%" />
- </p>
-
-## User Login
-
-```
-http://192.168.1.163:12345/dolphinscheduler/ui/#/monitor/servers/master
-```
-
-<p align="center">
- <img src="/img/api/user_login.png" width="80%" />
- </p>
-
-## Upload the Resource
-
-<p align="center">
- <img src="/img/api/upload_resource.png" width="80%" />
- </p>
-
-## Create a Workflow
-
-<p align="center">
- <img src="/img/api/create_workflow1.png" width="80%" />
- </p>
-
-
-<p align="center">
- <img src="/img/api/create_workflow2.png" width="80%" />
- </p>
-
-
-<p align="center">
- <img src="/img/api/create_workflow3.png" width="80%" />
- </p>
-
-
-<p align="center">
- <img src="/img/api/create_workflow4.png" width="80%" />
- </p>
-
-## View the Execution Result
-
-<p align="center">
- <img src="/img/api/execution_result.png" width="80%" />
- </p>
-
-## View Log
-
-<p align="center">
- <img src="/img/api/log.png" width="80%" />
- </p>
\ No newline at end of file
diff --git a/docs/docs/zh/guide/flink-call.md b/docs/docs/zh/guide/flink-call.md
deleted file mode 100644
index 547efa064f..0000000000
--- a/docs/docs/zh/guide/flink-call.md
+++ /dev/null
@@ -1,150 +0,0 @@
-# 调用 flink 操作步骤
-
-### 创建队列
-
-1. 登录调度系统，点击 "安全中心"，再点击左侧的 "队列管理"，点击 "队列管理" 创建队列
-2. 填写队列名称和队列值，然后点击 "提交"
-
-<p align="center">
- <img src="/img/api/create_queue.png" width="80%" />
- </p>
-
-
-
-
-### 创建租户
-
-```
-1.租户对应的是 linux 用户, 用户 worker 提交作业所使用的的用户， 如果 linux 没有这个用户， worker 会在执行脚本的时候创建这个用户
-2.租户和租户编码都是唯一不能重复，好比一个人有名字有身份证号。
-3.创建完租户会在 hdfs 对应的目录上有相关的文件夹。
-```
-
-<p align="center">
- <img src="/img/api/create_tenant.png" width="80%" />
- </p>
-
-
-
-
-### 创建用户
-
-<p align="center">
- <img src="/img/api/create_user.png" width="80%" />
- </p>
-
-
-
-
-### 创建 Token
-
-1. 登录调度系统，点击 "安全中心"，再点击左侧的 "令牌管理"，点击 "令牌管理" 创建令牌
-
-<p align="center">
- <img src="/img/token-management.png" width="80%" />
- </p>
-
-
-2. 选择 "失效时间" (Token有效期)，选择 "用户" (以指定的用户执行接口操作)，点击 "生成令牌" ，拷贝 Token 字符串，然后点击 "提交"
-
-<p align="center">
- <img src="/img/create-token.png" width="80%" />
- </p>
-
-
-### 使用 Token
-
-1. 打开 API文档页面
-
- > 地址：http://{api server ip}:12345/dolphinscheduler/doc.html?language=zh_CN&lang=cn
-
-<p align="center">
- <img src="/img/api-documentation.png" width="80%" />
- </p>
-
-
-2. 选一个测试的接口，本次测试选取的接口是：查询所有项目
-
- > projects/query-project-list
-
-3. 打开 Postman，填写接口地址，并在 Headers 中填写 Token，发送请求后即可查看结果
-
- ```
- token: 刚刚生成的 Token
- ```
-
-<p align="center">
- <img src="/img/test-api.png" width="80%" />
- </p>
-
-
-
-
-### 用户授权
-
-<p align="center">
- <img src="/img/api/user_authorization.png" width="80%" />
- </p>
-
-
-### 用户登录
-
-```
-http://192.168.1.163:12345/dolphinscheduler/ui/#/monitor/servers/master
-```
-
-<p align="center">
- <img src="/img/api/user_login.png" width="80%" />
- </p>
-
-
-
-
-### 资源上传
-
-<p align="center">
- <img src="/img/api/upload_resource.png" width="80%" />
- </p>
-
-
-
-
-### 创建工作流
-
-<p align="center">
- <img src="/img/api/create_workflow1.png" width="80%" />
- </p>
-
-
-<p align="center">
- <img src="/img/api/create_workflow2.png" width="80%" />
- </p>
-
-
-<p align="center">
- <img src="/img/api/create_workflow3.png" width="80%" />
- </p>
-
-
-<p align="center">
- <img src="/img/api/create_workflow4.png" width="80%" />
- </p>
-
-
-
-
-### 查看执行结果
-
-<p align="center">
- <img src="/img/api/execution_result.png" width="80%" />
- </p>
-
-
-
-
-### 查看日志结果
-
-<p align="center">
- <img src="/img/api/log.png" width="80%" />
- </p>
-
diff --git a/docs/img/api-documentation-en.png b/docs/img/api-documentation-en.png
deleted file mode 100644
index 81f6b78dc8..0000000000
Binary files a/docs/img/api-documentation-en.png and /dev/null differ
diff --git a/docs/img/api-documentation.png b/docs/img/api-documentation.png
deleted file mode 100644
index 97dc027b60..0000000000
Binary files a/docs/img/api-documentation.png and /dev/null differ
diff --git a/docs/img/api/create_queue.png b/docs/img/api/create_queue.png
deleted file mode 100644
index 1ad1a0dca2..0000000000
Binary files a/docs/img/api/create_queue.png and /dev/null differ
diff --git a/docs/img/api/create_tenant.png b/docs/img/api/create_tenant.png
deleted file mode 100644
index 8cfda08969..0000000000
Binary files a/docs/img/api/create_tenant.png and /dev/null differ
diff --git a/docs/img/api/create_user.png b/docs/img/api/create_user.png
deleted file mode 100644
index 2bbd095006..0000000000
Binary files a/docs/img/api/create_user.png and /dev/null differ
diff --git a/docs/img/api/create_workflow1.png b/docs/img/api/create_workflow1.png
deleted file mode 100644
index 479c6cdccc..0000000000
Binary files a/docs/img/api/create_workflow1.png and /dev/null differ
diff --git a/docs/img/api/create_workflow2.png b/docs/img/api/create_workflow2.png
deleted file mode 100644
index d87384f51e..0000000000
Binary files a/docs/img/api/create_workflow2.png and /dev/null differ
diff --git a/docs/img/api/create_workflow3.png b/docs/img/api/create_workflow3.png
deleted file mode 100644
index c721d7e907..0000000000
Binary files a/docs/img/api/create_workflow3.png and /dev/null differ
diff --git a/docs/img/api/create_workflow4.png b/docs/img/api/create_workflow4.png
deleted file mode 100644
index 3c7051f535..0000000000
Binary files a/docs/img/api/create_workflow4.png and /dev/null differ
diff --git a/docs/img/api/execution_result.png b/docs/img/api/execution_result.png
deleted file mode 100644
index b93556c111..0000000000
Binary files a/docs/img/api/execution_result.png and /dev/null differ
diff --git a/docs/img/api/log.png b/docs/img/api/log.png
deleted file mode 100644
index 0807079f33..0000000000
Binary files a/docs/img/api/log.png and /dev/null differ
diff --git a/docs/img/api/upload_resource.png b/docs/img/api/upload_resource.png
deleted file mode 100644
index 357211ea31..0000000000
Binary files a/docs/img/api/upload_resource.png and /dev/null differ
diff --git a/docs/img/api/user_authorization.png b/docs/img/api/user_authorization.png
deleted file mode 100644
index 6bcb8f3618..0000000000
Binary files a/docs/img/api/user_authorization.png and /dev/null differ
diff --git a/docs/img/api/user_login.png b/docs/img/api/user_login.png
deleted file mode 100644
index b663d1faf1..0000000000
Binary files a/docs/img/api/user_login.png and /dev/null differ
diff --git a/docs/img/create-token-en1.png b/docs/img/create-token-en1.png
deleted file mode 100644
index 7542464dc5..0000000000
Binary files a/docs/img/create-token-en1.png and /dev/null differ
diff --git a/docs/img/create-token.png b/docs/img/create-token.png
deleted file mode 100644
index 987a591e64..0000000000
Binary files a/docs/img/create-token.png and /dev/null differ
diff --git a/docs/img/test-api.png b/docs/img/test-api.png
deleted file mode 100644
index 71ae6f4c3e..0000000000
Binary files a/docs/img/test-api.png and /dev/null differ
diff --git a/docs/img/token-management-en.png b/docs/img/token-management-en.png
deleted file mode 100644
index 6d31e5ffe9..0000000000
Binary files a/docs/img/token-management-en.png and /dev/null differ
diff --git a/docs/img/token-management.png b/docs/img/token-management.png
deleted file mode 100644
index d3de6a9e38..0000000000
Binary files a/docs/img/token-management.png and /dev/null differ
[dolphinscheduler] 16/33: Fix alert server curator health check not work (#10111)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 4c62ee76e434c644411180f74ab30f2bbbdc5de5
Author: 旺阳 <wa...@lqwang.net>
AuthorDate: Thu May 19 10:51:37 2022 +0800
Fix alert server curator health check not work (#10111)
(cherry picked from commit 844ae42ee4ba2b2423906891c5dba102aa89a0e3)
---
.../src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java
index 1e17afefd1..ee9d5b3f62 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java
@@ -63,7 +63,7 @@ public class AlertServer implements Closeable {
*/
public static void main(String[] args) {
Thread.currentThread().setName(Constants.THREAD_NAME_ALERT_SERVER);
- new SpringApplicationBuilder(AlertServer.class).web(WebApplicationType.NONE).run(args);
+ new SpringApplicationBuilder(AlertServer.class).run(args);
}
@EventListener
[dolphinscheduler] 07/33: [Fix][Docker] Support Multi-Architecture Docker Images (#10052)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit f46a4d7dd8e77883ef2efcfa0dd4155f463939c7
Author: 旺阳 <wa...@lqwang.net>
AuthorDate: Wed May 18 10:20:24 2022 +0800
[Fix][Docker] Support Multi-Architecture Docker Images (#10052)
Co-authored-by: kezhenxu94 <ke...@apache.org>
(cherry picked from commit 537c206c2da92e114397dd29a98b70d6c78544dd)
---
.github/workflows/publish-docker.yaml | 4 +-
.../development/development-environment-setup.md | 54 ++++++++++++++++++++++
docs/docs/en/faq.md | 43 ++---------------
.../development/development-environment-setup.md | 53 ++++++++++++++++++++-
docs/docs/zh/faq.md | 36 ++-------------
pom.xml | 2 +
6 files changed, 119 insertions(+), 73 deletions(-)
diff --git a/.github/workflows/publish-docker.yaml b/.github/workflows/publish-docker.yaml
index 54032a0396..0b7d6ff20c 100644
--- a/.github/workflows/publish-docker.yaml
+++ b/.github/workflows/publish-docker.yaml
@@ -48,9 +48,9 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up QEMU
- uses: docker/setup-qemu-action@v1
+ uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
- uses: docker/setup-buildx-action@v1
+ uses: docker/setup-buildx-action@v2
- name: Build and push docker images
run: |
./mvnw -B clean deploy \
diff --git a/docs/docs/en/development/development-environment-setup.md b/docs/docs/en/development/development-environment-setup.md
index 15babb7210..102dead641 100644
--- a/docs/docs/en/development/development-environment-setup.md
+++ b/docs/docs/en/development/development-environment-setup.md
@@ -23,6 +23,58 @@ git clone git@github.com:apache/dolphinscheduler.git
ii. Run `mvn clean install -Prelease -Dmaven.test.skip=true`
+## Docker image build
+
+DolphinScheduler will release new Docker images after it released, you could find them in [Docker Hub](https://hub.docker.com/search?q=DolphinScheduler).
+
+* If you want to modify DolphinScheduler source code, and build Docker images locally, you can run when finished the modification
+```shell
+cd dolphinscheduler
+./mvnw -B clean package \
+ -Dmaven.test.skip \
+ -Dmaven.javadoc.skip \
+ -Dmaven.checkstyle.skip \
+ -Ddocker.tag=<TAG> \
+ -Pdocker,release
+```
+
+When the command is finished you could find them by command `docker imaegs`.
+
+* If you want to modify DolphinScheduler source code, build and push Docker images to your registry <HUB_URL>，you can run when finished the modification
+```shell
+cd dolphinscheduler
+./mvnw -B clean deploy \
+ -Dmaven.test.skip \
+ -Dmaven.javadoc.skip \
+ -Dmaven.checkstyle.skip \
+ -Dmaven.deploy.skip \
+ -Ddocker.tag=<TAG> \
+ -Ddocker.hub=<HUB_URL> \
+ -Pdocker,release
+```
+
+* If you want to modify DolphinScheduler source code, and also want to add customize dependencies of Docker image, you can modify the definition of Dockerfile after modifying the source code. You can run the following command to find all Dockerfile files.
+
+```shell
+cd dolphinscheduler
+find . -iname 'Dockerfile'
+```
+
+Then run the Docker build command above
+
+* You could create custom Docker images base on those images if you want to change image like add some dependencies or upgrade package.
+
+```Dockerfile
+FROM dolphinscheduler-standalone-server
+RUN apt update ; \
+ apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \
+```
+
+> **_Note：_** Docker will build and push linux/amd64,linux/arm64 multi-architecture images by default
+>
+> Have to use version after Docker 19.03, because after 19.03 docker contains buildx
+
+
## Notice
There are two ways to configure the DolphinScheduler development environment, standalone mode and normal mode
@@ -33,7 +85,9 @@ There are two ways to configure the DolphinScheduler development environment, st
## DolphinScheduler Standalone Quick Start
> **_Note:_** Use standalone server only for development and debugging, because it uses H2 Database as default database and Zookeeper Testing Server which may not be stable in production.
+>
> Standalone is only supported in DolphinScheduler 1.3.9 and later versions.
+>
> Standalone server is able to connect to external databases like mysql and postgresql, see [Standalone Deployment](https://dolphinscheduler.apache.org/en-us/docs/dev/user_doc/guide/installation/standalone.html) for instructions.
### Git Branch Choose
diff --git a/docs/docs/en/faq.md b/docs/docs/en/faq.md
index a16cfa4137..31b7ef47a9 100644
--- a/docs/docs/en/faq.md
+++ b/docs/docs/en/faq.md
@@ -700,47 +700,14 @@ A：The repair can be completed by executing the following SQL in the database:
update t_ds_version set version='2.0.1';
```
-## Can not find python-gateway-server in distribute package
+---
+
+## Q：Can not find python-gateway-server in distribute package
-After version 3.0.0-alpha, Python gateway server integrate into API server, and Python gateway service will start when you
+A：After version 3.0.0-alpha, Python gateway server integrate into API server, and Python gateway service will start when you
start API server. If you want disabled when Python gateway service you could change API server configuration in path
`api-server/conf/application.yaml` and change attribute `python-gateway.enabled : false`.
-## How to Build Custom Docker Image
-
-DolphinScheduler will release new Docker images after it released, you could find them in DockerHub. You could create
-custom Docker images base on those images if you want to change image like add some dependencies or upgrade package.
-
-```Dockerfile
-FROM apache/dolphinscheduler-standalone-server
-RUN apt update ; \
- apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \
-```
-
-If you want to modify DolphinScheduler source code, then build and distribute your own images, you can run below command
-to build Docker images and install them locally, which you could find them by command `docker imaegs`.
-
-```shell
-./mvnw -B clean install \
- -Dmaven.test.skip \
- -Dmaven.javadoc.skip \
- -Dmaven.checkstyle.skip \
- -Dmaven.deploy.skip \
- -Ddocker.tag=latest \
- -Pdocker,release
-```
-
-If you want to modify DolphinScheduler source code, but also want to add customize dependencies of Docker image, you can
-modify the definition of Dockerfile after modifying the source code. You can run the following command in root source code
-directory to find all Dockerfile files.
-
-```shell
-find . -iname 'Dockerfile'
-```
-
-Then run the command above start with `./mvnw -B clean install`. You can see all docker images you just created with
-command `docker images` after finish commnand `mvnw`.
-
---
-## We will collect more FAQ later
\ No newline at end of file
+We will collect more FAQ later
diff --git a/docs/docs/zh/development/development-environment-setup.md b/docs/docs/zh/development/development-environment-setup.md
index e315e89fb9..af5c4feb92 100644
--- a/docs/docs/zh/development/development-environment-setup.md
+++ b/docs/docs/zh/development/development-environment-setup.md
@@ -23,6 +23,56 @@ git clone git@github.com:apache/dolphinscheduler.git
* 运行 `mvn clean install -Prelease -Dmaven.test.skip=true`
+## Docker镜像构建
+
+DolphinScheduler 每次发版都会同时发布 Docker 镜像，你可以在 [Docker Hub](https://hub.docker.com/search?q=DolphinScheduler) 中找到这些镜像
+
+* 如果你想基于源码进行改造，然后在本地构建Docker镜像，可以在代码改造完成后运行
+```shell
+cd dolphinscheduler
+./mvnw -B clean package \
+ -Dmaven.test.skip \
+ -Dmaven.javadoc.skip \
+ -Dmaven.checkstyle.skip \
+ -Ddocker.tag=<TAG> \
+ -Pdocker,release
+```
+当命令运行完了后你可以通过 `docker images` 命令查看刚刚创建的镜像
+
+* 如果你想基于源码进行改造，然后构建Docker镜像并推送到 <HUB_URL>，可以在代码改造完成后运行
+```shell
+cd dolphinscheduler
+./mvnw -B clean deploy \
+ -Dmaven.test.skip \
+ -Dmaven.javadoc.skip \
+ -Dmaven.checkstyle.skip \
+ -Dmaven.deploy.skip \
+ -Ddocker.tag=<TAG> \
+ -Ddocker.hub=<HUB_URL> \
+ -Pdocker,release
+```
+
+* 如果你不仅需要改造源码，还想要自定义 Docker 镜像打包的依赖，可以在修改源码的同时修改 Dockerfile 的定义。你可以运行以下命令找到所有的 Dockerfile 文件
+
+```shell
+cd dolphinscheduler
+find . -iname 'Dockerfile'
+```
+
+之后再运行上面的构建镜像命令
+
+* 如果你因为个性化需求想要自己打包 Docker 镜像，最佳实践是基于 DolphinScheduler 对应镜像编写 Dockerfile 文件
+
+```Dockerfile
+FROM dolphinscheduler-standalone-server
+RUN apt update ; \
+ apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \
+```
+
+> **_注意：_** Docker默认会构建并推送 linux/amd64,linux/arm64 多架构镜像
+>
+> 必须使用Docker 19.03及以后的版本，因为19.03及以后的版本包含 buildx
+
## 开发者须知
DolphinScheduler 开发环境配置有两个方式，分别是standalone模式，以及普通模式
@@ -33,6 +83,7 @@ DolphinScheduler 开发环境配置有两个方式，分别是standalone模式
## DolphinScheduler Standalone快速开发模式
> **_注意：_** 仅供单机开发调试使用，默认使用 H2 Database,Zookeeper Testing Server
+>
> Standalone 仅在 DolphinScheduler 1.3.9 及以后的版本支持
### 分支选择
@@ -143,4 +194,4 @@ pnpm install
pnpm run dev
```
-截止目前，前后端已成功运行起来，浏览器访问[http://localhost:3000](http://localhost:3000)，并使用默认账户密码 **admin/dolphinscheduler123** 即可完成登录
\ No newline at end of file
+截止目前，前后端已成功运行起来，浏览器访问[http://localhost:3000](http://localhost:3000)，并使用默认账户密码 **admin/dolphinscheduler123** 即可完成登录
diff --git a/docs/docs/zh/faq.md b/docs/docs/zh/faq.md
index 5f380f3c8d..f16701fae8 100644
--- a/docs/docs/zh/faq.md
+++ b/docs/docs/zh/faq.md
@@ -682,40 +682,12 @@ update t_ds_version set version='2.0.1';
---
-## 在二进制分发包中找不到 python-gateway-server 文件夹
+## Q：在二进制分发包中找不到 python-gateway-server 文件夹
-在 3.0.0-alpha 版本之后，Python gateway server 集成到 api server 中，当您启动 api server 后，Python gateway server 将启动。
+A：在 3.0.0-alpha 版本之后，Python gateway server 集成到 api server 中，当您启动 api server 后，Python gateway server 将启动。
如果您不想在 api server 启动的时候启动 Python gateway server，您可以修改 api server 中的配置文件 `api-server/conf/application.yaml`
并更改可选项 `python-gateway.enabled` 中的值设置为 `false`。
-## 如何构建自定义的 Docker 镜像
+---
-DolphinScheduler 每次发版都会同时发布 Docker 镜像，你可以在 docker hub 中找到这些镜像，如果你因为个性化需求想要自己打包 docker 镜像，最佳实践是基于 DolphinScheduler 对应镜像编写 Dockerfile 文件
-
-```Dockerfile
-FROM dolphinscheduler-standalone-server
-RUN apt update ; \
- apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \
-```
-
-如果你想基于源码进行改造，打包并分发你的镜像，可以在代码改造完成后运行。当命令运行完了后你可以通过 `docker images` 命令查看刚刚创建的镜像
-
-```shell
-./mvnw -B clean install \
- -Dmaven.test.skip \
- -Dmaven.javadoc.skip \
- -Dmaven.checkstyle.skip \
- -Dmaven.deploy.skip \
- -Ddocker.tag=latest \
- -Pdocker,release
-```
-
-如果你不仅需要改造源码，还想要自定义 Docker 镜像打包的依赖，可以在修改源码的同时修改 Dockerfile 的定义，你可以在源码项目根目录中运行以下命令找到所有的 Dockerfile 文件
-
-```shell
-find . -iname 'Dockerfile'
-```
-
-之后再运行上面的 `./mvnw -B clean install` 命令，当命令运行完成后，你可以通过命令 `docker images` 查看刚刚创建的 docker 镜像
-
-我们会持续收集更多的 FAQ。
\ No newline at end of file
+我们会持续收集更多的 FAQ。
diff --git a/pom.xml b/pom.xml
index fd61b0b1fb..3ffd4135eb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -965,6 +965,8 @@
<arguments>
<argument>buildx</argument>
<argument>build</argument>
+ <argument>--platform</argument>
+ <argument>linux/amd64,linux/arm64</argument>
<argument>--no-cache</argument>
<argument>--push</argument>
<argument>-t</argument>
[dolphinscheduler] 01/33: [Fix][UI]Modify error message (#10054)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit eca859546e8de3854b4c04a481b47c85528d408b
Author: labbomb <73...@qq.com>
AuthorDate: Tue May 17 11:11:56 2022 +0800
[Fix][UI]Modify error message (#10054)
(cherry picked from commit a088ca10489ade01485f7bf3b4fbb3b81d97ceab)
---
dolphinscheduler-ui/src/locales/modules/en_US.ts | 2 +-
dolphinscheduler-ui/src/locales/modules/zh_CN.ts | 2 +-
.../src/views/projects/workflow/instance/components/table-action.tsx | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/dolphinscheduler-ui/src/locales/modules/en_US.ts b/dolphinscheduler-ui/src/locales/modules/en_US.ts
index badaf5bbf7..c01db94775 100644
--- a/dolphinscheduler-ui/src/locales/modules/en_US.ts
+++ b/dolphinscheduler-ui/src/locales/modules/en_US.ts
@@ -478,7 +478,7 @@ const project = {
recovery_waiting_thread: 'Recovery waiting thread',
recover_serial_wait: 'Recover serial wait',
recovery_suspend: 'Recovery Suspend',
- failed_to_retry: 'Failed to retry',
+ recovery_failed: 'Recovery Failed',
gantt: 'Gantt',
name: 'Name',
all_status: 'AllStatus',
diff --git a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
index c75a83427d..28a48dc565 100644
--- a/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
+++ b/dolphinscheduler-ui/src/locales/modules/zh_CN.ts
@@ -475,7 +475,7 @@ const project = {
recovery_waiting_thread: '恢复等待线程',
recover_serial_wait: '串行恢复',
recovery_suspend: '恢复运行',
- failed_to_retry: '重跑失败任务',
+ recovery_failed: '重跑失败任务',
gantt: '甘特图',
name: '名称',
all_status: '全部状态',
diff --git a/dolphinscheduler-ui/src/views/projects/workflow/instance/components/table-action.tsx b/dolphinscheduler-ui/src/views/projects/workflow/instance/components/table-action.tsx
index a75cd41a62..b5afa82ad5 100644
--- a/dolphinscheduler-ui/src/views/projects/workflow/instance/components/table-action.tsx
+++ b/dolphinscheduler-ui/src/views/projects/workflow/instance/components/table-action.tsx
@@ -167,7 +167,7 @@ export default defineComponent({
</NTooltip>
<NTooltip trigger={'hover'}>
{{
- default: () => t('project.workflow.failed_to_retry'),
+ default: () => t('project.workflow.recovery_failed'),
trigger: () => (
<NButton
tag='div'
[dolphinscheduler] 19/33: [doc] change document structure (#10124)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-beta-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit eaa75d6516152e125b53773c23c64978f5585f88
Author: Jiajie Zhong <zh...@gmail.com>
AuthorDate: Thu May 19 16:30:53 2022 +0800
[doc] change document structure (#10124)
This patch change the structure and make it more sense
* Remove dev doc in tab Docs, and for now you could only see it in
history-version page
* Move PyDolphinScheduler from tab DOCS to sidebar and combine with open
API
* Move installation after Functions
* Move data quality after HOW-TOS
(cherry picked from commit 2eff5ee10ce8073a5e648066d4a5e26933035c08)
---
docs/configs/docsdev.js | 128 +++++++++++++++++++++++++++---------------------
docs/configs/site.js | 10 ----
2 files changed, 73 insertions(+), 65 deletions(-)
diff --git a/docs/configs/docsdev.js b/docs/configs/docsdev.js
index 71ec133d3a..cbebe0a81a 100644
--- a/docs/configs/docsdev.js
+++ b/docs/configs/docsdev.js
@@ -50,27 +50,6 @@ export default {
},
],
},
- {
- title: 'Installation',
- children: [
- {
- title: 'Standalone Deployment',
- link: '/en-us/docs/3.0.0/user_doc/guide/installation/standalone.html',
- },
- {
- title: 'Pseudo Cluster Deployment',
- link: '/en-us/docs/3.0.0/user_doc/guide/installation/pseudo-cluster.html',
- },
- {
- title: 'Cluster Deployment',
- link: '/en-us/docs/3.0.0/user_doc/guide/installation/cluster.html',
- },
- {
- title: 'Kubernetes Deployment',
- link: '/en-us/docs/3.0.0/user_doc/guide/installation/kubernetes.html',
- },
- ],
- },
{
title: 'Introduction to Functions',
children: [
@@ -255,10 +234,6 @@ export default {
},
],
},
- {
- title: 'Data Quality',
- link: '/en-us/docs/3.0.0/user_doc/guide/data-quality.html',
- },
{
title: 'Resource',
children: [
@@ -298,8 +273,8 @@ export default {
],
},
{
- title: 'Open API',
- link: '/en-us/docs/3.0.0/user_doc/guide/open-api.html',
+ title: 'Data Quality',
+ link: '/en-us/docs/3.0.0/user_doc/guide/data-quality.html',
},
{
title: 'Flink',
@@ -315,6 +290,27 @@ export default {
},
],
},
+ {
+ title: 'Installation',
+ children: [
+ {
+ title: 'Standalone Deployment',
+ link: '/en-us/docs/dev/user_doc/guide/installation/standalone.html',
+ },
+ {
+ title: 'Pseudo Cluster Deployment',
+ link: '/en-us/docs/dev/user_doc/guide/installation/pseudo-cluster.html',
+ },
+ {
+ title: 'Cluster Deployment',
+ link: '/en-us/docs/dev/user_doc/guide/installation/cluster.html',
+ },
+ {
+ title: 'Kubernetes Deployment',
+ link: '/en-us/docs/dev/user_doc/guide/installation/kubernetes.html',
+ },
+ ],
+ },
{
title: 'Architecture Guide',
children: [
@@ -344,6 +340,19 @@ export default {
},
],
},
+ {
+ title: 'API',
+ children: [
+ {
+ title: 'Open API',
+ link: '/en-us/docs/dev/user_doc/guide/open-api.html',
+ },
+ {
+ title: 'PyDolphinScheduler',
+ link: '/python/index.html',
+ },
+ ],
+ },
{
title: 'FAQ',
children: [
@@ -397,27 +406,6 @@ export default {
},
],
},
- {
- title: '部署指南',
- children: [
- {
- title: '单机部署(Standalone)',
- link: '/zh-cn/docs/3.0.0/user_doc/guide/installation/standalone.html',
- },
- {
- title: '伪集群部署(Pseudo-Cluster)',
- link: '/zh-cn/docs/3.0.0/user_doc/guide/installation/pseudo-cluster.html',
- },
- {
- title: '集群部署(Cluster)',
- link: '/zh-cn/docs/3.0.0/user_doc/guide/installation/cluster.html',
- },
- {
- title: 'Kubernetes部署(Kubernetes)',
- link: '/zh-cn/docs/3.0.0/user_doc/guide/installation/kubernetes.html',
- },
- ],
- },
{
title: '功能介绍',
children: [
@@ -602,10 +590,6 @@ export default {
},
],
},
- {
- title: '数据质量',
- link: '/zh-cn/docs/3.0.0/user_doc/guide/data-quality.html',
- },
{
title: '资源中心',
children: [
@@ -636,7 +620,7 @@ export default {
link: '/zh-cn/docs/3.0.0/user_doc/guide/security.html',
},
{
- title: '常见操作',
+ title: '常用指引',
children: [
{
title: '通用设置',
@@ -645,8 +629,8 @@ export default {
],
},
{
- title: 'API调用',
- link: '/zh-cn/docs/3.0.0/user_doc/guide/open-api.html',
+ title: '数据质量',
+ link: '/zh-cn/docs/3.0.0/user_doc/guide/data-quality.html',
},
{
title: 'Flink调用',
@@ -662,6 +646,27 @@ export default {
},
],
},
+ {
+ title: '部署指南',
+ children: [
+ {
+ title: '单机部署(Standalone)',
+ link: '/zh-cn/docs/dev/user_doc/guide/installation/standalone.html',
+ },
+ {
+ title: '伪集群部署(Pseudo-Cluster)',
+ link: '/zh-cn/docs/dev/user_doc/guide/installation/pseudo-cluster.html',
+ },
+ {
+ title: '集群部署(Cluster)',
+ link: '/zh-cn/docs/dev/user_doc/guide/installation/cluster.html',
+ },
+ {
+ title: 'Kubernetes部署(Kubernetes)',
+ link: '/zh-cn/docs/dev/user_doc/guide/installation/kubernetes.html',
+ },
+ ],
+ },
{
title: '架构设计',
children: [
@@ -691,6 +696,19 @@ export default {
},
],
},
+ {
+ title: 'API',
+ children: [
+ {
+ title: 'API调用',
+ link: '/zh-cn/docs/dev/user_doc/guide/open-api.html',
+ },
+ {
+ title: 'PyDolphinScheduler',
+ link: '/python/index.html',
+ },
+ ],
+ },
{
title: 'FAQ',
children: [
diff --git a/docs/configs/site.js b/docs/configs/site.js
index a279ac01fe..982a32aa04 100644
--- a/docs/configs/site.js
+++ b/docs/configs/site.js
@@ -58,16 +58,6 @@ export default {
text: 'Older Versions',
link: '/en-us/docs/release/history-versions.html',
},
- {
- key: 'pythonAPI',
- text: 'PyDolphinScheduler',
- link: '/python/index.html',
- },
- {
- key: 'docsdev',
- text: 'dev',
- link: '/en-us/docs/dev/user_doc/about/introduction.html',
- },
],
},
{