You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by zh...@apache.org on 2022/11/28 09:50:23 UTC
[dolphinscheduler] branch 3.0.3-prepare updated (8ea3e68f5b -> a28e206166)
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a change to branch 3.0.3-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
from 8ea3e68f5b fix-ldap-error (#12929)
new 9f14571713 [fix] Change the default unix shell executor from sh to bash (#12180)
new 31296afad4 Fix timing scheduling trigger master service report to get command parameter null pointer exception (#12419)
new f677d3311d make sure all failed task will save in errorTaskMap (#12424)
new 78ad0d0cd1 [Fix][UI] download resource return 401 (#12566)
new 1eb4f3a837 [doc][fix] Add new alert doc Slack (#12567)
new 6b1456985d [Feature-8030][docs] Add sqoop task doc (#12855)
new 589221d4f7 [Improvement][Document]Update architecture and process-start-flow pic (#12912)
new a28e206166 [Bug-12954] [Schedule] Fix that workflow-level configuration information does not take effect when timing triggers execution (#12955)
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/configs/docs3-0-2.js | 8 ++
docs/docs/en/architecture/design.md | 2 +-
docs/docs/en/guide/task/sqoop.md | 90 ++++++++++++++++++++
docs/docs/en/guide/upgrade/incompatible.md | 13 +++
docs/docs/zh/guide/task/sqoop.md | 91 +++++++++++++++++++++
docs/docs/zh/guide/upgrade/incompatible.md | 13 +++
docs/img/architecture-lastest.jpg | Bin 0 -> 57753 bytes
docs/img/process-start-flow-1.3.0.png | Bin 224425 -> 159291 bytes
docs/img/tasks/demo/sqoop_task01.png | Bin 0 -> 2530 bytes
docs/img/tasks/demo/sqoop_task02.png | Bin 0 -> 216945 bytes
docs/img/tasks/demo/sqoop_task03.png | Bin 0 -> 2455 bytes
docs/img/tasks/icons/sqoop.png | Bin 0 -> 815 bytes
.../master/runner/WorkflowExecuteRunnable.java | 19 ++---
.../service/process/ProcessServiceImpl.java | 22 ++---
.../service/quartz/ProcessScheduleJob.java | 3 +-
.../plugin/task/api/AbstractCommandExecutor.java | 9 +-
.../plugin/task/api/ShellCommandExecutor.java | 4 +-
dolphinscheduler-ui/src/views/login/use-login.ts | 2 +
18 files changed, 246 insertions(+), 30 deletions(-)
create mode 100644 docs/docs/en/guide/task/sqoop.md
create mode 100644 docs/docs/en/guide/upgrade/incompatible.md
create mode 100644 docs/docs/zh/guide/task/sqoop.md
create mode 100644 docs/docs/zh/guide/upgrade/incompatible.md
create mode 100644 docs/img/architecture-lastest.jpg
create mode 100644 docs/img/tasks/demo/sqoop_task01.png
create mode 100644 docs/img/tasks/demo/sqoop_task02.png
create mode 100644 docs/img/tasks/demo/sqoop_task03.png
create mode 100644 docs/img/tasks/icons/sqoop.png
[dolphinscheduler] 07/08: [Improvement][Document]Update architecture and process-start-flow pic (#12912)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.3-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 589221d4f703002b32006391f25fd286176816e2
Author: xuhhui <sz...@163.com>
AuthorDate: Wed Nov 16 16:03:00 2022 +0800
[Improvement][Document]Update architecture and process-start-flow pic (#12912)
Co-authored-by: xuhaihui <xu...@cmss.chinamobile.com>
(cherry picked from commit 9e6f4d2bc17579cdf9ca1f45fa006b01103efac8)
---
docs/docs/en/architecture/design.md | 2 +-
docs/img/architecture-lastest.jpg | Bin 0 -> 57753 bytes
docs/img/process-start-flow-1.3.0.png | Bin 224425 -> 159291 bytes
3 files changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/docs/en/architecture/design.md b/docs/docs/en/architecture/design.md
index 9e09e15948..c931bab7fd 100644
--- a/docs/docs/en/architecture/design.md
+++ b/docs/docs/en/architecture/design.md
@@ -5,7 +5,7 @@
### System Architecture Diagram
<p align="center">
- <img src="../../../img/architecture-1.3.0.jpg" alt="System architecture diagram" width="70%" />
+ <img src="../../../img/architecture-lastest.jpg" alt="System architecture diagram" width="70%" />
<p align="center">
<em>System architecture diagram</em>
</p>
diff --git a/docs/img/architecture-lastest.jpg b/docs/img/architecture-lastest.jpg
new file mode 100644
index 0000000000..71c58394e1
Binary files /dev/null and b/docs/img/architecture-lastest.jpg differ
diff --git a/docs/img/process-start-flow-1.3.0.png b/docs/img/process-start-flow-1.3.0.png
index fdd46724fd..5b863dfb99 100644
Binary files a/docs/img/process-start-flow-1.3.0.png and b/docs/img/process-start-flow-1.3.0.png differ
[dolphinscheduler] 03/08: make sure all failed task will save in errorTaskMap (#12424)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.3-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit f677d3311df8ce7200999ad1f13c1c3cfdf3b619
Author: Yann Ann <xi...@gmail.com>
AuthorDate: Wed Oct 19 09:36:47 2022 +0800
make sure all failed task will save in errorTaskMap (#12424)
(cherry picked from commit 38b643f69b65f4de9dd43809404470934bfadc7b)
---
.../server/master/runner/WorkflowExecuteRunnable.java | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java
index f3f84408a6..5c0fec018d 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java
@@ -375,13 +375,13 @@ public class WorkflowExecuteRunnable implements Callable<WorkflowSubmitStatue> {
retryTaskInstance(taskInstance);
} else if (taskInstance.getState().typeIsFailure()) {
completeTaskMap.put(taskInstance.getTaskCode(), taskInstance.getId());
+ errorTaskMap.put(taskInstance.getTaskCode(), taskInstance.getId());
// There are child nodes and the failure policy is: CONTINUE
if (processInstance.getFailureStrategy() == FailureStrategy.CONTINUE && DagHelper.haveAllNodeAfterNode(
Long.toString(taskInstance.getTaskCode()),
dag)) {
submitPostNode(Long.toString(taskInstance.getTaskCode()));
} else {
- errorTaskMap.put(taskInstance.getTaskCode(), taskInstance.getId());
if (processInstance.getFailureStrategy() == FailureStrategy.END) {
killAllTasks();
}
@@ -405,7 +405,6 @@ public class WorkflowExecuteRunnable implements Callable<WorkflowSubmitStatue> {
/**
* release task group
*
- * @param taskInstance
*/
public void releaseTaskGroup(TaskInstance taskInstance) {
logger.info("Release task group");
@@ -431,7 +430,6 @@ public class WorkflowExecuteRunnable implements Callable<WorkflowSubmitStatue> {
/**
* crate new task instance to retry, different objects from the original
*
- * @param taskInstance
*/
private void retryTaskInstance(TaskInstance taskInstance) throws StateEventHandleException {
if (!taskInstance.taskCanRetry()) {
@@ -646,10 +644,7 @@ public class WorkflowExecuteRunnable implements Callable<WorkflowSubmitStatue> {
}
private boolean needComplementProcess() {
- if (processInstance.isComplementData() && Flag.NO == processInstance.getIsSubProcess()) {
- return true;
- }
- return false;
+ return processInstance.isComplementData() && Flag.NO == processInstance.getIsSubProcess();
}
/**
@@ -1022,7 +1017,7 @@ public class WorkflowExecuteRunnable implements Callable<WorkflowSubmitStatue> {
/**
* clone a new taskInstance for retry and reset some logic fields
*
- * @return
+ * @return taskInstance
*/
public TaskInstance cloneRetryTaskInstance(TaskInstance taskInstance) {
TaskNode taskNode = dag.getNode(Long.toString(taskInstance.getTaskCode()));
@@ -1044,7 +1039,7 @@ public class WorkflowExecuteRunnable implements Callable<WorkflowSubmitStatue> {
/**
* clone a new taskInstance for tolerant and reset some logic fields
*
- * @return
+ * @return taskInstance
*/
public TaskInstance cloneTolerantTaskInstance(TaskInstance taskInstance) {
TaskNode taskNode = dag.getNode(Long.toString(taskInstance.getTaskCode()));
@@ -1064,9 +1059,9 @@ public class WorkflowExecuteRunnable implements Callable<WorkflowSubmitStatue> {
/**
* new a taskInstance
*
- * @param processInstance
- * @param taskNode
- * @return
+ * @param processInstance process instance
+ * @param taskNode task node
+ * @return task instance
*/
public TaskInstance newTaskInstance(ProcessInstance processInstance, TaskNode taskNode) {
TaskInstance taskInstance = new TaskInstance();
[dolphinscheduler] 02/08: Fix timing scheduling trigger master service report to get command parameter null pointer exception (#12419)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.3-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 31296afad4b1c6480487da9af6b53643273c0a89
Author: Kerwin <37...@users.noreply.github.com>
AuthorDate: Wed Oct 19 09:43:36 2022 +0800
Fix timing scheduling trigger master service report to get command parameter null pointer exception (#12419)
(cherry picked from commit a8e23008acdebd99811271611a15e83a9a7d8d92)
---
.../service/process/ProcessServiceImpl.java | 22 ++++++++++++----------
.../service/quartz/ProcessScheduleJob.java | 2 +-
2 files changed, 13 insertions(+), 11 deletions(-)
diff --git a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java
index c552740021..08f33e5619 100644
--- a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java
+++ b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java
@@ -30,6 +30,7 @@ import static org.apache.dolphinscheduler.plugin.task.api.enums.DataType.VARCHAR
import static org.apache.dolphinscheduler.plugin.task.api.enums.Direct.IN;
import static org.apache.dolphinscheduler.plugin.task.api.utils.DataQualityConstants.TASK_INSTANCE_ID;
+import org.apache.commons.lang3.StringUtils;
import static java.util.stream.Collectors.toSet;
import org.apache.dolphinscheduler.common.Constants;
@@ -133,7 +134,6 @@ import org.apache.dolphinscheduler.service.task.TaskPluginManager;
import org.apache.dolphinscheduler.spi.enums.ResourceType;
import org.apache.commons.collections.CollectionUtils;
-import org.apache.commons.lang.StringUtils;
import org.apache.commons.lang.math.NumberUtils;
import java.util.ArrayList;
@@ -390,16 +390,18 @@ public class ProcessServiceImpl implements ProcessService {
@Counted("dolphinscheduler_create_command_count")
public int createCommand(Command command) {
int result = 0;
- if (command != null) {
- // add command timezone
- Schedule schedule = scheduleMapper.queryByProcessDefinitionCode(command.getProcessDefinitionCode());
- Map<String, String> commandParams = JSONUtils.toMap(command.getCommandParam());
- if (commandParams != null && schedule != null) {
- commandParams.put(Constants.SCHEDULE_TIMEZONE, schedule.getTimezoneId());
- command.setCommandParam(JSONUtils.toJsonString(commandParams));
- }
- result = commandMapper.insert(command);
+ if (command == null) {
+ return result;
+ }
+ // add command timezone
+ Schedule schedule = scheduleMapper.queryByProcessDefinitionCode(command.getProcessDefinitionCode());
+ if (schedule != null) {
+ Map<String, String> commandParams = StringUtils.isNotBlank(command.getCommandParam()) ? JSONUtils.toMap(command.getCommandParam()) : new HashMap<>();
+ commandParams.put(Constants.SCHEDULE_TIMEZONE, schedule.getTimezoneId());
+ command.setCommandParam(JSONUtils.toJsonString(commandParams));
}
+ command.setId(null);
+ result = commandMapper.insert(command);
return result;
}
diff --git a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java
index ec5ed36f45..d726840fa1 100644
--- a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java
+++ b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java
@@ -17,6 +17,7 @@
package org.apache.dolphinscheduler.service.quartz;
+import org.apache.commons.lang3.StringUtils;
import org.apache.dolphinscheduler.common.Constants;
import org.apache.dolphinscheduler.common.enums.CommandType;
import org.apache.dolphinscheduler.common.enums.ReleaseState;
@@ -36,7 +37,6 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.quartz.QuartzJobBean;
-import org.springframework.util.StringUtils;
import io.micrometer.core.annotation.Counted;
import io.micrometer.core.annotation.Timed;
[dolphinscheduler] 08/08: [Bug-12954] [Schedule] Fix that workflow-level configuration information does not take effect when timing triggers execution (#12955)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.3-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit a28e206166a10010d36a5dc0d9e2fd54f799678b
Author: Kerwin <37...@users.noreply.github.com>
AuthorDate: Tue Nov 22 15:35:29 2022 +0800
[Bug-12954] [Schedule] Fix that workflow-level configuration information does not take effect when timing triggers execution (#12955)
(cherry picked from commit 2f8f0952fb6e45b84437e9f651096858d856bd96)
---
.../org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java | 1 +
1 file changed, 1 insertion(+)
diff --git a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java
index d726840fa1..0904d1b25e 100644
--- a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java
+++ b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java
@@ -91,6 +91,7 @@ public class ProcessScheduleJob extends QuartzJobBean {
command.setWarningGroupId(schedule.getWarningGroupId());
String workerGroup = StringUtils.isEmpty(schedule.getWorkerGroup()) ? Constants.DEFAULT_WORKER_GROUP : schedule.getWorkerGroup();
command.setWorkerGroup(workerGroup);
+ command.setEnvironmentCode(schedule.getEnvironmentCode());
command.setWarningType(schedule.getWarningType());
command.setProcessInstancePriority(schedule.getProcessInstancePriority());
command.setProcessDefinitionVersion(processDefinition.getVersion());
[dolphinscheduler] 06/08: [Feature-8030][docs] Add sqoop task doc (#12855)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.3-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 6b1456985d34b73c662599ece9a2170cd441115d
Author: baihongbin <48...@users.noreply.github.com>
AuthorDate: Sun Nov 13 00:09:05 2022 +0800
[Feature-8030][docs] Add sqoop task doc (#12855)
* [Feature-8030][docs] Add sqoop task doc
* Update docs/docs/zh/guide/task/sqoop.md
Co-authored-by: Eric Gao <er...@gmail.com>
* Update docs/docs/en/guide/task/sqoop.md
Co-authored-by: Eric Gao <er...@gmail.com>
* [Feature-8030][docs] Add sqoop task doc
Co-authored-by: Eric Gao <er...@gmail.com>
(cherry picked from commit 0373e0661586c48523cedaab8a80fc7298f50a4d)
---
docs/docs/en/guide/task/sqoop.md | 90 ++++++++++++++++++++++++++++++++++
docs/docs/zh/guide/task/sqoop.md | 91 +++++++++++++++++++++++++++++++++++
docs/img/tasks/demo/sqoop_task01.png | Bin 0 -> 2530 bytes
docs/img/tasks/demo/sqoop_task02.png | Bin 0 -> 216945 bytes
docs/img/tasks/demo/sqoop_task03.png | Bin 0 -> 2455 bytes
docs/img/tasks/icons/sqoop.png | Bin 0 -> 815 bytes
6 files changed, 181 insertions(+)
diff --git a/docs/docs/en/guide/task/sqoop.md b/docs/docs/en/guide/task/sqoop.md
new file mode 100644
index 0000000000..e2d65debbd
--- /dev/null
+++ b/docs/docs/en/guide/task/sqoop.md
@@ -0,0 +1,90 @@
+# Sqoop Node
+
+## Overview
+
+Sqoop task type for executing Sqoop application. The workers run `sqoop` to execute sqoop tasks.
+
+## Create Task
+
+- Click `Project Management -> Project Name -> Workflow Definition`, and click the `Create Workflow` button to enter the DAG editing page.
+- Drag from the toolbar <img src="../../../../img/tasks/icons/sqoop.png" width="15"/> to the canvas.
+
+## Task Parameters
+
+[//]: # (TODO: use the commented anchor below once our website template supports this syntax)
+[//]: # (- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md#default-task-parameters) `Default Task Parameters` section for default parameters.)
+
+- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md) `Default Task Parameters` section for default parameters.
+
+| **Parameter** | **Description** |
+|-------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Job Name | map-reduce job name |
+| Direct | (1) import:Imports an individual table from an RDBMS to HDFS or Hve. (2) export:Exports a set of files from HDFS or Hive back to an RDBMS. |
+| Hadoop Params | Hadoop custom param for sqoop job. |
+| Sqoop Advanced Parameters | Sqoop advanced param for sqoop job. |
+| Data Source - Type | Select the corresponding data source type. |
+| Data Source - Datasource | Select the corresponding DataSource. |
+| Data Source - ModelType | (1) Form:Synchronize data from a table, need to fill in the `Table` and `ColumnType`. (2) SQL:Synchronize data of SQL queries result, need to fill in the `SQL Statement`. |
+| Data Source - Table | Sets the table name to use when importing to Hive. |
+| Data Source - ColumnType | (1) All Columns:Import all fields in the selected table. (2) Some Columns:Import the specified fields in the selected table, need to fill in the `Column`. |
+| Data Source - Column | Fill in the field name, and separate with commas. |
+| Data Source - SQL Statement | Fill in SQL query statement. |
+| Data Source - Map Column Hive | Override mapping from SQL to Hive type for configured columns. |
+| Data Source - Map Column Java | Override mapping from SQL to Java type for configured columns. |
+| Data Target - Type | Select the corresponding data target type. |
+| Data Target - Database | Fill in the Hive database name. |
+| Data Target - Table | Fill in the Hive table name. |
+| Data Target - CreateHiveTable | Import a table definition into Hive. If set, then the job will fail if the target hive table exits. |
+| Data Target - DropDelimiter | Drops `\n`, `\r`, and `\01` from string fields when importing to Hive. |
+| Data Target - OverWriteSrc | Overwrite existing data in the Hive table. |
+| Data Target - Hive Target Dir | You can also explicitly choose the target directory. |
+| Data Target - ReplaceDelimiter | Replace `\n`, `\r`, and `\01` from string fields with user defined string when importing to Hive. |
+| Data Target - Hive partition Keys | Fill in the hive partition keys name, and separate with commas. |
+| Data Target - Hive partition Values | Fill in the hive partition Values name, and separate with commas. |
+| Data Target - Target Dir | Fill in the HDFS target directory. |
+| Data Target - DeleteTargetDir | Delete the target directory if it exists. |
+| Data Target - CompressionCodec | Choice the hadoop codec. |
+| Data Target - FileType | Choice the storage Type. |
+| Data Target - FieldsTerminated | Sets the field separator character. |
+| Data Target - LinesTerminated | Sets the end-of-line character. |
+
+## Task Example
+
+This example demonstrates importing data from MySQL into Hive. The MySQL database name is `test` and the table name is `example`. The following figure shows sample data.
+
+
+
+### Configuring the Sqoop environment
+
+If you are using the Sqoop task type in a production environment, you must ensure that the worker can execute the `sqoop` command.
+
+### Configuring Sqoop Task Node
+
+you can configure the node content by following the steps in the diagram below.
+
+
+
+The key configuration in this sample is shown in the following table.
+
+| **Parameter** | **Value** |
+|-------------------------------------|----------------------------------------------------------------------|
+| Job Name | sqoop_mysql_to_hive_test |
+| Data Source - Type | MYSQL |
+| Data Source - Datasource | MYSQL MyTestMySQL(You could change MyTestMySQL to the name you like) |
+| Data Source - ModelType | Form |
+| Data Source - Table | example |
+| Data Source - ColumnType | All Columns |
+| Data Target - Type | HIVE |
+| Data Target - Database | tmp |
+| Data Target - Table | example |
+| Data Target - CreateHiveTable | true |
+| Data Target - DropDelimiter | false |
+| Data Target - OverWriteSrc | true |
+| Data Target - Hive Target Dir | (No need to fill in) |
+| Data Target - ReplaceDelimiter | , |
+| Data Target - Hive partition Keys | (No need to fill in) |
+| Data Target - Hive partition Values | (No need to fill in) |
+
+### View run results
+
+
diff --git a/docs/docs/zh/guide/task/sqoop.md b/docs/docs/zh/guide/task/sqoop.md
new file mode 100644
index 0000000000..cfea038a31
--- /dev/null
+++ b/docs/docs/zh/guide/task/sqoop.md
@@ -0,0 +1,91 @@
+# SQOOP 节点
+
+## 综述
+
+SQOOP 任务类型，用于执行 SQOOP 程序。对于 SQOOP 节点，worker 会通过执行 `sqoop` 命令来执行 SQOOP 任务。
+
+## 创建任务
+
+- 点击项目管理 -> 项目名称 -> 工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面；
+- 拖动工具栏的 <img src="../../../../img/tasks/icons/sqoop.png" width="15"/> 任务节点到画板中。
+
+## 任务参数
+
+[//]: # (TODO: use the commented anchor below once our website template supports this syntax)
+[//]: # (- 默认参数说明请参考[DolphinScheduler任务参数附录](appendix.md#默认任务参数)`默认任务参数`一栏。)
+
+- 默认参数说明请参考[DolphinScheduler任务参数附录](appendix.md)`默认任务参数`一栏。
+
+| **任务参数** | **描述** |
+|------------------|----------------------------------------------------------------------|
+| 任务名称 | map-reduce 任务名称 |
+| 流向 | (1) import：从 RDBMS 导入 HDFS 或Hive (2) export：从 HDFS 或 Hive 导出到 RDBMS |
+| Hadoop 参数 | 添加自定义 Hadoop 参数 |
+| Sqoop 参数 | 添加自定义 Sqoop 参数 |
+| 数据来源 - 类型 | 选择数据源类型 |
+| 数据来源 - 数据源 | 选择数据源 |
+| 数据来源 - 模式 | (1) 单表：同步单张表的数据，需填写`表名`和`列类型` (2) SQL：同步 SQL 查询的结果，需填写`SQL语句` |
+| 数据来源 - 表名 | 设置需要导入 hive 的表名 |
+| 数据来源 - 列类型 | (1) 全表导入：导入表中的所有字段 (2) 选择列：导入表中的指定列，需填写`列`信息 |
+| 数据来源 - 列 | 填写字段名称，多个字段之间使用英文逗号分割 |
+| 数据来源 - SQL 语句 | 填写 SQL 查询语句 |
+| 数据来源 - Hive 类型映射 | 自定义 SQL 与 Hive 类型映射 |
+| 数据来源 - Java 类型映射 | 自定义 SQL 与 Java 类型映射 |
+| 数据目的 - 类型 | 选择数据目的类型 |
+| 数据目的 - 数据库 | 填写 Hive 数据库名称 |
+| 数据目的 - 表名 | 填写 Hive 表名 |
+| 数据目的 - 是否创建新表 | 选择是否自动根据导入数据类型创建数据目的表，如果目标表已经存在了，那么创建任务会失败 |
+| 数据目的 - 是否删除分隔符 | 自动删除字符串中的`\n`、`\r`和`\01`字符 |
+| 数据目的 - 是否覆盖数据源 | 覆盖 Hive 表中的现有数据 |
+| 数据目的 - Hive 目标路径 | 自定义 Hive 目标路径 |
+| 数据目的 - 替换分隔符 | 替换字符串中的`\n`、`\r`和`\01`字符 |
+| 数据目的 - Hive 分区键 | 填写 Hive 分区键，多个分区键之间使用英文逗号分割 |
+| 数据目的 - Hive 分区值 | 填写 Hive 分区值，多个分区值之间使用英文逗号分割 |
+| 数据目的 - 目标路径 | 填写 HDFS 的目标路径 |
+| 数据目的 - 是否删除目录 | 如果目录已经存在，则删除目录 |
+| 数据目的 - 压缩类型 | 选择 HDFS 文件压缩类型 |
+| 数据目的 - 保存格式 | 选择文件保存格式 |
+| 数据目的 - 列分隔符 | 自定义列分隔符 |
+| 数据目的 - 行分隔符 | 自定义行分隔符 |
+
+## 任务样例
+
+该样例演示为从 MySQL 数据导入到 Hive 中。 其中 MySQL 数据库名称为：`test`，表名称为`example`。下图为样例数据。
+
+
+
+### 配置 Sqoop 环境
+
+若生产环境中要是使用到 Sqoop 任务类型，则需要先配置好所需的环境。确保任务节点可以执行`sqoop`命令。
+
+### 配置 Sqoop 任务节点
+
+可按照下图步骤进行配置节点内容。
+
+
+
+本样例中的关键配置如下表。
+
+| **任务参数** | **参数值** |
+|------------------|------------------------------------------------|
+| 任务名称 | sqoop_mysql_to_hive_test |
+| 流向 | import |
+| 数据来源 - 类型 | MYSQL |
+| 数据来源 - 数据源 | MYSQL MyTestMySQL（您可以将MyTestMySQL改成您自己取的数据源名称） |
+| 数据来源 - 模式 | 表单 |
+| 数据来源 - 表名 | example |
+| 数据来源 - 列类型 | 全表导入 |
+| 数据目的 - 类型 | HIVE |
+| 数据目的 - 数据库 | tmp |
+| 数据目的 - 表名 | example |
+| 数据目的 - 是否创建新表 | true |
+| 数据目的 - 是否删除分隔符 | false |
+| 数据目的 - 是否覆盖数据源 | true |
+| 数据目的 - Hive 目标路径 | （无需填写） |
+| 数据目的 - 替换分隔符 | , |
+| 数据目的 - Hive 分区键 | （无需填写） |
+| 数据目的 - Hive 分区值 | （无需填写） |
+
+### 查看运行结果
+
+
diff --git a/docs/img/tasks/demo/sqoop_task01.png b/docs/img/tasks/demo/sqoop_task01.png
new file mode 100644
index 0000000000..ec63a52337
Binary files /dev/null and b/docs/img/tasks/demo/sqoop_task01.png differ
diff --git a/docs/img/tasks/demo/sqoop_task02.png b/docs/img/tasks/demo/sqoop_task02.png
new file mode 100644
index 0000000000..18215eb98b
Binary files /dev/null and b/docs/img/tasks/demo/sqoop_task02.png differ
diff --git a/docs/img/tasks/demo/sqoop_task03.png b/docs/img/tasks/demo/sqoop_task03.png
new file mode 100644
index 0000000000..1a197ea79a
Binary files /dev/null and b/docs/img/tasks/demo/sqoop_task03.png differ
diff --git a/docs/img/tasks/icons/sqoop.png b/docs/img/tasks/icons/sqoop.png
new file mode 100644
index 0000000000..6ff06de10e
Binary files /dev/null and b/docs/img/tasks/icons/sqoop.png differ
[dolphinscheduler] 01/08: [fix] Change the default unix shell executor from sh to bash (#12180)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.3-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 9f1457171374539b636439fc6d5d8f77d8882b91
Author: simsicon <si...@gmail.com>
AuthorDate: Tue Oct 18 12:57:37 2022 +0800
[fix] Change the default unix shell executor from sh to bash (#12180)
Co-authored-by: Kerwin <37...@users.noreply.github.com>
(cherry picked from commit ba538067f291c4fdb378ca84c02bb31e2fb2d295)
---
docs/docs/en/guide/upgrade/incompatible.md | 13 +++++++++++++
docs/docs/zh/guide/upgrade/incompatible.md | 13 +++++++++++++
.../plugin/task/api/AbstractCommandExecutor.java | 9 +++++----
.../plugin/task/api/ShellCommandExecutor.java | 4 ++--
4 files changed, 33 insertions(+), 6 deletions(-)
diff --git a/docs/docs/en/guide/upgrade/incompatible.md b/docs/docs/en/guide/upgrade/incompatible.md
new file mode 100644
index 0000000000..0fd2c409ea
--- /dev/null
+++ b/docs/docs/en/guide/upgrade/incompatible.md
@@ -0,0 +1,13 @@
+# Incompatible
+
+This document records the incompatible updates between each version. You need to check this document before you upgrade to related version.
+
+## dev
+
+* Remove the spark version of spark task ([#11860](https://github.com/apache/dolphinscheduler/pull/11860)).
+* Change the default unix shell executor from sh to bash ([#12180](https://github.com/apache/dolphinscheduler/pull/12180)).
+
+## 3.0.0
+
+* Copy and import workflow without 'copy' suffix [#10607](https://github.com/apache/dolphinscheduler/pull/10607)
+* Use semicolon as default sql segment separator [#10869](https://github.com/apache/dolphinscheduler/pull/10869)
diff --git a/docs/docs/zh/guide/upgrade/incompatible.md b/docs/docs/zh/guide/upgrade/incompatible.md
new file mode 100644
index 0000000000..0bc0a7e675
--- /dev/null
+++ b/docs/docs/zh/guide/upgrade/incompatible.md
@@ -0,0 +1,13 @@
+# 不向前兼容的更新
+
+本文档记录了各版本之间不兼容的更新内容。在升级到相关版本前，请检查本文档。
+
+## dev
+
+* Remove the spark version of spark task ([#11860](https://github.com/apache/dolphinscheduler/pull/11860)).
+* Change the default unix shell executor from sh to bash ([#12180](https://github.com/apache/dolphinscheduler/pull/12180)).
+
+## 3.0.0
+
+* Copy and import workflow without 'copy' suffix [#10607](https://github.com/apache/dolphinscheduler/pull/10607)
+* Use semicolon as default sql segment separator [#10869](https://github.com/apache/dolphinscheduler/pull/10869)
diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java
index 5faa07bba8..08cb1bda30 100644
--- a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java
+++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java
@@ -122,9 +122,10 @@ public abstract class AbstractCommandExecutor {
// if sudo.enable=true,setting up user to run commands
if (OSUtils.isSudoEnable()) {
- command.add("sudo");
- command.add("-u");
- command.add(taskRequest.getTenantCode());
+ command.add("sudo");
+ command.add("-u");
+ command.add(taskRequest.getTenantCode());
+ command.add("-E");
}
command.add(commandInterpreter());
command.addAll(Collections.emptyList());
@@ -398,7 +399,7 @@ public abstract class AbstractCommandExecutor {
return lineList;
}
-
+
/**
* find var pool
*
diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ShellCommandExecutor.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ShellCommandExecutor.java
index 09eeb53f3a..5761c29606 100644
--- a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ShellCommandExecutor.java
+++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ShellCommandExecutor.java
@@ -38,9 +38,9 @@ import org.slf4j.Logger;
public class ShellCommandExecutor extends AbstractCommandExecutor {
/**
- * For Unix-like, using sh
+ * For Unix-like, using bash
*/
- private static final String SH = "sh";
+ private static final String SH = "bash";
/**
* For Windows, using cmd.exe
[dolphinscheduler] 04/08: [Fix][UI] download resource return 401 (#12566)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.3-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 78ad0d0cd158f521bc548d1a717828d0985595d6
Author: hushuang <hu...@gmail.com>
AuthorDate: Fri Oct 28 11:52:22 2022 +0800
[Fix][UI] download resource return 401 (#12566)
resource download with form action, without header: sessionId: xxxx, set sessionId after login，support download resource success
(cherry picked from commit 0c519f7e5d3f6473564e2eb23fe3615670985de6)
---
dolphinscheduler-ui/src/views/login/use-login.ts | 2 ++
1 file changed, 2 insertions(+)
diff --git a/dolphinscheduler-ui/src/views/login/use-login.ts b/dolphinscheduler-ui/src/views/login/use-login.ts
index 6fd2080e55..acd450f946 100644
--- a/dolphinscheduler-ui/src/views/login/use-login.ts
+++ b/dolphinscheduler-ui/src/views/login/use-login.ts
@@ -24,6 +24,7 @@ import type { SessionIdRes } from '@/service/modules/login/types'
import type { UserInfoRes } from '@/service/modules/users/types'
import { useRouteStore } from '@/store/route/route'
import { useTimezoneStore } from '@/store/timezone/timezone'
+import cookies from 'js-cookie'
export function useLogin(state: any) {
const router: Router = useRouter()
@@ -36,6 +37,7 @@ export function useLogin(state: any) {
if (!valid) {
const loginRes: SessionIdRes = await login({ ...state.loginForm })
await userStore.setSessionId(loginRes.sessionId)
+ cookies.set('sessionId', loginRes.sessionId, { path: '/' })
const userInfoRes: UserInfoRes = await getUserInfo()
await userStore.setUserInfo(userInfoRes)
[dolphinscheduler] 05/08: [doc][fix] Add new alert doc Slack (#12567)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.3-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
commit 1eb4f3a837dc55a276986e21126c0c4db33e8ddc
Author: Tq <ti...@gmail.com>
AuthorDate: Tue Nov 1 09:49:05 2022 +0800
[doc][fix] Add new alert doc Slack (#12567)
(cherry picked from commit 4e2701b57ef55c93f1f14f348e96a2f7c5fd2177)
---
docs/configs/docs3-0-2.js | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/docs/configs/docs3-0-2.js b/docs/configs/docs3-0-2.js
index 392ac5ca44..a0d63e8bb6 100644
--- a/docs/configs/docs3-0-2.js
+++ b/docs/configs/docs3-0-2.js
@@ -268,6 +268,10 @@ export default {
title: 'Slack',
link: '/en-us/docs/dev/user_doc/guide/alert/slack.html',
},
+ {
+ title: 'Slack',
+ link: '/en-us/docs/dev/user_doc/guide/alert/slack.html',
+ },
],
},
{
@@ -806,6 +810,10 @@ export default {
title: 'Slack告警',
link: '/zh-cn/docs/dev/user_doc/guide/alert/slack.html',
},
+ {
+ title: 'Slack告警',
+ link: '/zh-cn/docs/dev/user_doc/guide/alert/slack.html',
+ },
],
},
{