You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by zh...@apache.org on 2022/04/14 07:53:57 UTC
[dolphinscheduler-website] branch master updated: add news (#773)
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/master by this push:
new 946c3e901 add news (#773)
946c3e901 is described below
commit 946c3e9019cf7e1cf01508f9a5277c3a202ce2c2
Author: lifeng <53...@users.noreply.github.com>
AuthorDate: Thu Apr 14 15:53:52 2022 +0800
add news (#773)
* add news Fast Task Type Expanding On Apache DolphinScheduler | Tutorial
add news Fast Task Type Expanding On Apache DolphinScheduler | Tutorial
* Update Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.md
---
...xpanding_On_Apache_DolphinScheduler_Tutorial.md | 387 +++++++++++++++++++
...xpanding_On_Apache_DolphinScheduler_Tutorial.md | 429 +++++++++++++++++++++
img/2022-03-29/1.png | Bin 0 -> 763914 bytes
img/2022-03-29/2.jpeg | Bin 0 -> 128091 bytes
img/2022-03-29/3.png | Bin 0 -> 1158268 bytes
img/2022-03-29/4.png | Bin 0 -> 55865 bytes
img/2022-03-29/5.png | Bin 0 -> 531651 bytes
img/2022-03-29/En/1.png | Bin 0 -> 423097 bytes
img/2022-03-29/En/2.png | Bin 0 -> 809636 bytes
img/2022-03-29/En/3.png | Bin 0 -> 299095 bytes
site_config/blog.js | 17 +-
site_config/home.jsx | 43 +--
12 files changed, 847 insertions(+), 29 deletions(-)
diff --git a/blog/en-us/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.md b/blog/en-us/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.md
new file mode 100644
index 000000000..fe28b2c13
--- /dev/null
+++ b/blog/en-us/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.md
@@ -0,0 +1,387 @@
+# Fast Task Type Expanding On Apache DolphinScheduler | Tutorial
+
+<div align=center>
+
+<img src="/img/2022-03-29/En/1.png"/>
+
+</div>
+
+## Background
+
+At present, the scheduler plays an indispensable role in big data ecology. The Apache DolphinScheduler, a top-tier Apache project, is one of the most stable and easy-to-use scheduling systems. With scheduling, distribution, high availability, and ease of use in place, it is only natural that users will want to quickly, easily, and concisely expand the Apache Dolphinscheduler task types as their business grows or as more components are used for various needs. This article shows you how to [...]
+
+## Author Bio
+
+<div align=center>
+
+<img src="/img/2022-03-29/En/2.png"/>
+
+</div>
+
+Baiqiang Zhang
+
+Baiqiang Zhang is a big data development engineer, who is interested in researching real-time computing, metadata governance, and big data basic components.
+
+## 1 What is SPI?
+
+SPI (Service Provider Interface) is a service delivery discovery mechanism built into the JDK. Most people will probably rarely use it, as it is positioned primarily for development vendors, and is described in more detail in the java.util.ServiceLoader files. The abstract concept of SPI refers to the dynamic loading of service implementation.
+
+## 2 Why did we introduce SPI?
+
+Different enterprises may have their components that need to be executed by tasks, for example, enterprises use Hive, the most commonly used tool in the big data ecosystem, in different ways. Some enterprises execute tasks through HiveServer2, and some use HiveClient to execute tasks. Considering the out-of-the-box Task provided by Apache DolphinScheduler does not support HiveClient’s Task, so most users will execute through the Shell. However, a Shell doesn’t work well compared with a T [...]
+
+First of all, we need to understand the history of the Task iteration of Apache DolphinScheduler. In DS 1.3.x, expanding a Task required recompiling the whole Apache DolphinScheduler, which was heavily coupled, so in Apache DolphinScheduler 2.0.x, we introduced SPI. As we mentioned earlier, the essence of SPI is to dynamically load the implementation of a service, so let’s make it more concrete and consider the Task of Apache DolphinScheduler as an execution service, and we need to execu [...]
+
+## 3 Who is using it?
+
+**a. Apache DolphinScheduler**
+
+i. task
+
+ii. datasource
+
+**b. Apache Flink**
+
+i. Flink sql connector, after the user has implemented a flink-connector, Flink is also dynamically loaded via SPI
+
+**c. Spring boot**
+
+i. spring boot spi
+
+**d. Jdbc**
+
+i. Before jdbc4.0, developers need to load the driver based on Class by forName(“xxx”), jdbc4 also based on the spi mechanism to discover the driver provider, you can expose the driver provider by specifying the implementation class in the META-INF/services/java. sql. Driver file
+
+**e. More**
+
+* **dubbo**
+* **common-logging**
+* **……**
+## 4 What’s the Apache DolphinScheduler SPI Process?
+
+<div align=center>
+
+<img src="/img/2022-03-29/En/3.png"/>
+
+</div>
+
+*Note: SPI Rules*
+
+*When compiling the specific implementation of the service into a JAR, we need to create the META-INF/services/ folder in the dir of the resource, and then create a fully qualified class name with the file name of the service, which is the fully qualified class name of the integrated interface. The content inside is the fully qualified class name of the implementing class.*
+
+To explain the above diagram, I have divided Apache DolphinScheduler into logical tasks and physical tasks, logical tasks refer to DependTask, SwitchTask, and physical tasks refer to ShellTask, SQLTask, which are the Task for executing tasks. In Apache DolphinScheduler, we generally expand the physical tasks, which are handed over to the Worker to execute, so what we need to understand is that when we have more than one Worker, we have to distribute the custom task to each machine with W [...]
+
+```plain
+public final class ServiceLoader<S> implements Iterable<S> {
+ //scanning dir prefix
+ private static final String PREFIX = "META-INF/services/";
+ //The class or interface representing the service being loaded
+ private final Class<S> service;
+ //The class loader used to locate, load, and instantiate providers
+ private final ClassLoader loader;
+ //Private inner class implementing fully-lazy provider lookup
+ private class LazyIterator implements Iterator<S> {
+ Class<S> service;
+ ClassLoader loader;
+ Enumeration<URL> configs = null;
+ String nextName = null;
+ //......
+ private boolean hasNextService() {
+ if (configs == null) {
+ try {
+ //get dir all class
+ String fullName = PREFIX + service.getName();
+ if (loader == null)
+ configs = ClassLoader.getSystemResources(fullName);
+ else
+ configs = loader.getResources(fullName);
+ } catch (IOException x) {
+ //......
+ }
+ //......
+ }
+ }
+ }
+}
+```
+## 5 How to extend a data source Task or DataSource ?
+
+### 5.1 Creating a Maven project
+
+```plain
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.dolphinscheduler \
+ -DarchetypeArtifactId=dolphinscheduler-hive-client-task \
+ -DarchetypeVersion=1.10.0 \
+ -DgroupId=org.apache.dolphinscheduler \
+ -DartifactId=dolphinscheduler-hive-client-task \
+ -Dversion=0.1 \
+ -Dpackage=org.apache.dolphinscheduler \
+ -DinteractiveMode=false
+```
+### 5.2 Maven dependencies
+
+```plain
+<! --dolphinscheduler spi basic core denpendence -->
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-spi</artifactId>
+ <version>${dolphinscheduler.lib.version}</version
+ <scope>${common.lib.scope}</scope>
+ </dependency
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-task-api</artifactId>
+ <version>${dolphinscheduler.lib.version}</version
+ <scope>${common.lib.scope}</scope>
+ </dependency
+```
+### 5.3 Creating a TaskChannelFactory
+
+First, we need to create the factory for the task service, which mainly targets to help build the TaskChannel and TaskPlugin parameters, and to give the unique identity of the task. The ChannelFactory connects the Task service group of Apache DolphinScheduler, and helps the front and back end interaction to build the TaskChannel.
+
+```plain
+package org.apache.dolphinscheduler.plugin.task.hive;
+import org.apache.dolphinscheduler.spi.params.base.PluginParams;
+import org.apache.dolphinscheduler.spi.task.TaskChannel;
+import org.apache.dolphinscheduler.spi.task.TaskChannelFactory;
+import java.util.List;
+public class HiveClientTaskChannelFactory implements TaskChannelFactory {
+ /**
+ * Create a task channel and execute tasks based on it
+ * @return Task Channel
+ */
+ @Override
+ public TaskChannel create() {
+ return new HiveClientTaskChannel();
+ }
+ /**
+ * Returns the globally unique identifier of the current task
+ * @return Task type name
+ */
+ @Override
+ public String getName() {
+ return "HIVE CLIENT";
+ }
+ /**
+ * The front-end pages need to be rendered, mainly into
+
+ * @return
+ */
+ @Override
+ public List<PluginParams> getParams() {
+ List<PluginParams> pluginParams = new ArrayList<>();
+ InputParam nodeName = InputParam.newBuilder("name", "$t('Node name')")
+ .addValidate(Validate.newBuilder()
+ .setRequired(true)
+ .build())
+ .build();
+ PluginParams runFlag = RadioParam.newBuilder("runFlag", "RUN_FLAG")
+ .addParamsOptions(new ParamsOptions("NORMAL", "NORMAL", false))
+ .addParamsOptions(new ParamsOptions("FORBIDDEN", "FORBIDDEN", false))
+ .build();
+ PluginParams build = CheckboxParam.newBuilder("Hive SQL", "Test HiveSQL")
+ .setDisplay(true)
+ .setValue("-- author: \n --desc:")
+ .build();
+ pluginParams.add(nodeName);
+ pluginParams.add(runFlag);
+ pluginParams.add(build);
+ return pluginParams;
+ }
+}
+```
+### 5.4 Creating a TaskChannel
+
+After we have a factory, we will create a TaskChannel based on it. The TaskChannel contains two methods, canceling and creating, currently, we only need to focus on creating tasks.
+
+```plain
+void cancelApplication(boolean status);
+ /**
+ * Build executable tasks
+ */
+ AbstractTask createTask(TaskRequest taskRequest);
+public class HiveClientTaskChannel implements TaskChannel {
+ @Override
+ public void cancelApplication(boolean b) {
+ //do nothing
+ }
+ @Override
+ public AbstractTask createTask(TaskRequest taskRequest) {
+ return new HiveClientTask(taskRequest);
+ }
+}
+```
+### 5.5 Building a Task Implementation
+
+With TaskChannel we get the physical task that can be executed, but we need to add the corresponding implementation to the current task to allow Apache DolphinScheduler to execute your task.
+
+We can see from the above figure that the tasks based on Yarn execution will inherit AbstractYarnTask, and those that do not need to be executed by Yarn will directly inherit AbstractTaskExecutor, which mainly contains an AppID, and CanalApplication setMainJar. As you can see above, our HiveClient needs to inherit AbstractYarnTask, and before building the task, we need to build the parameters object that fits the HiveClient to deserialize the JsonParam.
+
+```plain
+package org.apache.dolphinscheduler.plugin.task.hive;
+import org.apache.dolphinscheduler.spi.task.AbstractParameters;
+import org.apache.dolphinscheduler.spi.task.ResourceInfo;
+import java.util.List;
+public class HiveClientParameters extends AbstractParameters {
+ /**
+ * The easiest way to execute with HiveClient is to just paste in all the SQL, so we only need one SQL parameter
+ */
+ private String sql;
+ public String getSql() {
+ return sql;
+ }
+ public void setSql(String sql) {
+ this.sql = sql;
+ }
+ @Override
+ public boolean checkParameters() {
+ return sql ! = null;
+ }
+ @Override
+ public List<ResourceInfo> getResourceFilesList() {
+ return null;
+ }
+}
+```
+After implementing the parameters object, let’s implement the Task. The implementation in the example is relatively simple, which is to write the user’s parameters to a file and execute the task via Hive -f.
+```plain
+package org.apache.dolphinscheduler.plugin.task.hive;
+import org.apache.dolphinscheduler.plugin.task.api.AbstractYarnTask;
+import org.apache.dolphinscheduler.spi.task.AbstractParameters;
+import org.apache.dolphinscheduler.spi.task.request.TaskRequest;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import java.io;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+public class HiveClientTask extends AbstractYarnTask {
+ /**
+ * hive client parameters
+ */
+ private HiveClientParameters hiveClientParameters;
+ /**
+ * taskExecutionContext
+ */
+ private final TaskRequest taskExecutionContext;
+ public HiveClientTask(TaskRequest taskRequest) {
+ super(taskRequest);
+ this.taskExecutionContext = taskRequest;
+ }
+ /**
+ * task init method
+ */
+ @Override
+ public void init() {
+ logger.info("hive client task param is {}", JSONUtils.toJsonString(taskExecutionContext));
+ this.hiveClientParameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), HiveClientParameters.class);
+ if (this.hiveClientParameters ! = null && !hiveClientParameters.checkParameters()) {
+ throw new RuntimeException("hive client task params is not valid");
+ }
+ }
+ /**
+ * build task execution command
+ *
+ * @return task execution command or null
+ */
+ @Override
+ protected String buildCommand() {
+ String filePath = getFilePath();
+ if (writeExecutionContentToFile(filePath)) {
+ return "hive -f " + filePath;
+ }
+ return null;
+ }
+ /**
+ * get hive sql write path
+ *
+ * @return file write path
+ */
+ private String getFilePath() {
+ return String.format("%s/hive-%s-%s.sql", this.taskExecutionContext.getExecutePath(), this.taskExecutionContext.getTaskName(), this. taskExecutionContext.getTaskInstanceId());
+ }
+ @Override
+ protected void setMainJarName() {
+ //do nothing
+ }
+ /**
+ * write hive sql to filepath
+ *
+ * @param filePath file path
+ * @return write success?
+ */
+ private boolean writeExecutionContentToFile(String filePath) {
+ Path path = Paths.get(filePath);
+ try (BufferedWriter writer = Files.newBufferedWriter(path, StandardCharsets.UTF_8)) {
+ writer.write(this.hiveClientParameters.getSql());
+ logger.info("file:" + filePath + "write success.");
+ return true;
+ } catch (IOException e) {
+ logger.error("file:" + filePath + "write failed. please path auth.");
+ e.printStackTrace();
+ return false;
+ }
+ }
+ @Override
+ public AbstractParameters getParameters() {
+ return this.hiveClientParameters;
+ }
+}
+```
+### 5.6 Compliance with SPI Rules
+
+```plain
+# 1,Create META-INF/services folder under Resource, create the file with the same full class name of the interface
+zhang@xiaozhang resources % tree . /
+. /
+└── META-INF
+ └── services
+ └─ org.apache.dolphinscheduler.spi.task.TaskChannelFactory
+# 2, write the fully qualified class name of the implemented class in the file
+zhang@xiaozhang resources % more META-INF/services/org.apache.dolphinscheduler.spi.task.TaskChannelFactory
+org.apache.dolphinscheduler.plugin.task.hive.HiveClientTaskChannelFactory
+```
+### 5.7 Packaging and Deployment
+
+```plain
+## 1,Packing
+mvn clean install
+## 2, Deployment
+cp . /target/dolphinscheduler-task-hiveclient-1.0.jar $DOLPHINSCHEDULER_HOME/lib/
+## 3,restart dolphinscheduler server
+```
+After the above operation, we check the worker log tail -200f $Apache DolphinScheduler_HOME/log/Apache DolphinScheduler-worker.log.
+That’s all~ The front-end modifications involved above can be found in Apache DolphinScheduler-ui/src/js/conf/home/pages/dag/_source/formModel/
+
+## Join the Community
+
+There are many ways to participate and contribute to the DolphinScheduler community, including:
+
+**Documents, translation, Q&A, tests, codes, articles, keynote speeches, etc.**
+
+We assume the first PR (document, code) to contribute to be simple and should be used to familiarize yourself with the submission process and community collaboration style.
+
+So the community has compiled the following list of issues suitable for novices: [https://github.com/apache/dolphinscheduler/issues/5689](https://github.com/apache/dolphinscheduler/issues/5689)
+
+List of non-newbie issues: [https://github.com/apache/dolphinscheduler/issues?q=is%3Aopen+is%3Aissue+label%3A%22volunteer+wanted%22](https://github.com/apache/dolphinscheduler/issues?q=is%3Aopen+is%3Aissue+label%3A%22volunteer+wanted%22)
+
+How to participate in the contribution: [https://dolphinscheduler.apache.org/en-us/community/development/contribute.html](https://dolphinscheduler.apache.org/en-us/community/development/contribute.html)
+
+**GitHub Code Repository:** [https://github.com/apache/dolphinscheduler](https://github.com/apache/dolphinscheduler)
+
+**Official Website**：[https://dolphinscheduler.apache.org/](https://dolphinscheduler.apache.org/)
+
+**MailList**：dev@dolphinscheduler@apache.org
+
+**Twitter**：@DolphinSchedule
+
+**YouTube：**[https://www.youtube.com/channel/UCmrPmeE7dVqo8DYhSLHa0vA](https://www.youtube.com/channel/UCmrPmeE7dVqo8DYhSLHa0vA)
+
+**Slack：**[https://s.apache.org/dolphinscheduler-slack](https://s.apache.org/dolphinscheduler-slack)
+
+**Contributor Guide：**[https://dolphinscheduler.apache.org/en-us/community/index.html](https://dolphinscheduler.apache.org/en-us/community/index.html)
+
+Your Star for the project is important, don’t hesitate to lighten a Star for Apache DolphinScheduler ❤️
+
diff --git a/blog/zh-cn/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.md b/blog/zh-cn/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.md
new file mode 100644
index 000000000..f8c1f9dce
--- /dev/null
+++ b/blog/zh-cn/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.md
@@ -0,0 +1,429 @@
+# 极速开发扩充 Apache DolphinScheduler Task 类型 | 实用教程
+
+<div align=center>
+
+<img src="/img/2022-03-29/1.png"/>
+
+</div>
+
+## 背景简介
+
+目前在大数据生态中，调度系统是不可或缺的一个重要组件。Apache DolphinScheduler 作为一个顶级的 Apache 项目，其稳定性和易用性也可以说是名列前茅的。而对于一个调度系统来说，能够支持的可调度的任务类型同样是一个非常重要的因素，在调度、分布式、高可用、易用性解决了的情况下，随着业务的发展或者各种需求使用到的组件增多，用户自然而然会希望能够快速、方便、简洁地对 Apache Dolphinscheduler 可调度的任务类型进行扩充。本文便带大家了解如何方便、极速扩充一个 Apache DolphinScheduler Task。
+
+## 作者简介
+
+<div align=center>
+
+<img src="/img/2022-03-29/2.jpeg"/>
+
+</div>
+
+张柏强，大数据开发工程师，主要研究方向为实时计算、元数据治理、大数据基础组件。
+
+## **1 什么是 SPI 服务发现(What is SPI)？**
+
+SPI 全称为 (Service Provider Interface) ，是 JDK 内置的一种服务提供发现机制。大多数人可能会很少用到它，因为它的定位主要是面向开发厂商的，在 java.util.ServiceLoader 的文档里有比较详细的介绍，其抽象的概念是指动态加载某个服务实现。
+
+## **2 为什么要引入 SPI(Why did we introduce SPI)?**
+
+不同的企业可能会有自己的组件需要通过 task 去执行，大数据生态中最为常用数仓工具 Apache Hive 来举例，不同的企业使用 Hive 方法各有不同。有的企业通过 HiveServer2 执行任务，有的企业使用 HiveClient 执行任务，而 Apache DolphinScheduler 提供的开箱即用的 Task 中并没有支持 HiveClient 的 Task，所以大部分使用者都会通过 Shell 去执行。然而，Shell 哪有天然的TaskTemplate 好用呢？所以，Apache DolphinScheduler 为了使用户能够更好地根据企业需求定制不同的 Task，便支持了 TaskSPI 化。
+
+我们首先要了解一下 Apache DolphinScheduler 的 Task 改版历程，在 DS 1.3.x 时，扩充一个 Task 需要重新编译整个 Apache DolphinScheduler，耦合严重，所以在 Apache DolphinScheduler 2.0.x 引入了 SPI。前面我们提到了 SPI 的抽象概念是动态加载某个服务的实现，这里我们具象一点，将 Apache DolphinScheduler 的 Task 看成一个执行服务，而我们需要根据使用者的选择去执行不同的服务，如果没有的服务，则需要我们自己扩充，相比于 1.3.x 我们只需要完成我们的 Task 具体实现逻辑，然后遵守 SPI 的规则，编译成 Jar 并上传到指定目录，即可使用我们自己编写的 Task。
+
+## **3 谁在使用它(Who is using it)?**
+
+1、Apache DolphinScheduler
+
+* task
+* datasource
+2、Apache Flink
+
+* flink sql connector，用户实现了一个flink-connector后，Flink也是通过SPI来动态加载的
+3、Spring boot
+
+* spring boot spi
+4、Jdbc
+
+* jdbc4。0以前， 开发人员还需要基于Class。forName("xxx")的方式来装载驱动，jdbc4也基于spi的机制来发现驱动提供商了，可以通过META-INF/services/java。sql。Driver文件里指定实现类的方式来暴露驱动提供者
+5、更多
+
+* dubbo
+* common-logging
+## **4 Apache DolphinScheduler SPI Process?**
+
+
+<div align=center>
+
+<img src="/img/2022-03-29/3.png"/>
+
+</div>
+
+剖析一下上面这张图，我给 Apache DolphinScheduler 分为逻辑 Task 以及物理 Task，逻辑 Task 指 DependTask，SwitchTask 这种逻辑上的 Task；物理 Task 是指 ShellTask，SQLTask 这种执行任务的 Task。而在 Apache DolphinScheduler中，我们一般扩充的都是物理 Task，而物理 Task 都是交由 Worker 去执行，所以我们要明白的是，当我们在有多台 Worker 的情况下，要将自定义的 Task 分发到每一台有 Worker 的机器上，当我们启动 Worker 服务时，worker 会去启动一个 ClassLoader 来加载相应的实现了规则的 Task lib，可以看到 HiveClient 和 SeatunnelTask 都是用户自定义的，但是只有 HiveTask 被 Apache DolphinScheduler TaskPluginManage 加载了，原因是 SeatunnelTask 并没有去遵守 SPI 的规则。SPI 的规则图上也有赘述，也可以 [...]
+
+```plain
+public final class ServiceLoader<S> implements Iterable<S> {
+ //scanning dir prefix
+ private static final String PREFIX = "META-INF/services/";
+
+ //The class or interface representing the service being loaded
+ private final Class<S> service;
+
+ //The class loader used to locate, load, and instantiate providers
+ private final ClassLoader loader;
+
+ //Private inner class implementing fully-lazy provider lookup
+ private class LazyIterator implements Iterator<S> {
+ Class<S> service;
+ ClassLoader loader;
+ Enumeration<URL> configs = null;
+ String nextName = null;
+
+ //......
+ private boolean hasNextService() {
+ if (configs == null) {
+ try {
+ //get dir all class
+ String fullName = PREFIX + service.getName();
+ if (loader == null)
+ configs = ClassLoader.getSystemResources(fullName);
+ else
+ configs = loader.getResources(fullName);
+ } catch (IOException x) {
+ //......
+ }
+ //......
+ }
+ }
+ }
+}
+
+```
+
+
+
+## **5 如何扩展一个 data sourceTask or DataSource (How to extend a task or datasource)?**
+
+### **5.1 创建 Maven 项目**
+
+```plain
+
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.dolphinscheduler \
+ -DarchetypeArtifactId=dolphinscheduler-hive-client-task \
+ -DarchetypeVersion=1.10.0 \
+ -DgroupId=org.apache.dolphinscheduler \
+ -DartifactId=dolphinscheduler-hive-client-task \
+ -Dversion=0.1 \
+ -Dpackage=org.apache.dolphinscheduler \
+ -DinteractiveMode=false
+
+
+```
+
+### **5.2 Maven 依赖**
+
+```plain
+
+ <!--dolphinscheduler spi basic core denpendence-->
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-spi</artifactId>
+ <version>${dolphinscheduler.lib.version}</version>
+ <scope>${common.lib.scope}</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-task-api</artifactId>
+ <version>${dolphinscheduler.lib.version}</version>
+ <scope>${common.lib.scope}</scope>
+ </dependency>
+
+
+```
+
+### **5.3 创建 Task 通道工厂(TaskChannelFactory)**
+
+首先我们需要创建任务服务的工厂，其主要作用是帮助构建 TaskChannel 以及 TaskPlugin 参数，同时给出该任务的唯一标识，ChannelFactory 在 Apache DolphinScheduler 的 Task 服务组中，其作用属于是在任务组中的承上启下，交互前后端以及帮助 Worker 构建 TaskChannel。
+
+```plain
+package org.apache.dolphinscheduler.plugin.task.hive;
+
+import org.apache.dolphinscheduler.spi.params.base.PluginParams;
+import org.apache.dolphinscheduler.spi.task.TaskChannel;
+import org.apache.dolphinscheduler.spi.task.TaskChannelFactory;
+
+import java.util.List;
+
+public class HiveClientTaskChannelFactory implements TaskChannelFactory {
+ /**
+ * 创建任务通道,基于该通道执行任务
+ * @return 任务通道
+ */
+ @Override
+ public TaskChannel create() {
+ return new HiveClientTaskChannel();
+ }
+
+ /**
+ * 返回当前任务的全局唯一标识
+ * @return 任务类型名称
+ */
+ @Override
+ public String getName() {
+ return "HIVE CLIENT";
+ }
+
+ /**
+ * 前端页面需要用到的渲染,主要分为
+
+ * @return
+ */
+ @Override
+ public List<PluginParams> getParams() {
+ List<PluginParams> pluginParams = new ArrayList<>();
+ InputParam nodeName = InputParam.newBuilder("name", "$t('Node name')")
+ .addValidate(Validate.newBuilder()
+ .setRequired(true)
+ .build())
+ .build();
+ PluginParams runFlag = RadioParam.newBuilder("runFlag", "RUN_FLAG")
+ .addParamsOptions(new ParamsOptions("NORMAL", "NORMAL", false))
+ .addParamsOptions(new ParamsOptions("FORBIDDEN", "FORBIDDEN", false))
+ .build();
+
+ PluginParams build = CheckboxParam.newBuilder("Hive SQL", "Test HiveSQL")
+ .setDisplay(true)
+ .setValue("-- author: \n --desc:")
+ .build();
+
+ pluginParams.add(nodeName);
+ pluginParams.add(runFlag);
+ pluginParams.add(build);
+
+ return pluginParams;
+ }
+}
+```
+
+### **5.4 创建 TaskChannel**
+
+有了工厂之后，我们会根据工厂创建出 TaskChannel，TaskChannel 包含如下两个方法，一个是取消，一个是创建，目前不需要关注取消，主要关注创建任务。
+
+```plain
+ void cancelApplication(boolean status);
+
+ /**
+ * 构建可执行任务
+ */
+ AbstractTask createTask(TaskRequest taskRequest);
+ public class HiveClientTaskChannel implements TaskChannel {
+ @Override
+ public void cancelApplication(boolean b) {
+ //do nothing
+ }
+
+ @Override
+ public AbstractTask createTask(TaskRequest taskRequest) {
+ return new HiveClientTask(taskRequest);
+ }
+}
+
+```
+
+
+### **5.5 构建 Task 实现**
+
+通过 TaskChannel 我们得到了可执行的物理 Task，但是我们需要给当前 Task 添加相应的实现，才能够让Apache DolphinScheduler 去执行你的任务，首先在编写 Task 之前我们需要先了解一下 Task 之间的关系：
+
+<div align=center>
+
+<img src="/img/2022-03-29/4.png"/>
+
+</div>
+
+通过上图我们可以看到，基于 Yarn 执行任务的 Task 都会去继承 AbstractYarnTask，不需要经过 Yarn 执行的都会去直接继承 AbstractTaskExecutor，主要是包含一个 AppID，以及 CanalApplication setMainJar 之类的方法，想知道的小伙伴可以自己去深入研究一下，如上可知我们实现的 HiveClient 就需要继承 AbstractYarnTask，在构建 Task 之前，我们需要构建一下适配 HiveClient 的 Parameters 对象用来反序列化JsonParam。
+
+```plain
+ package com.jegger.dolphinscheduler.plugin.task.hive;
+
+import org.apache.dolphinscheduler.spi.task.AbstractParameters;
+import org.apache.dolphinscheduler.spi.task.ResourceInfo;
+
+import java.util.List;
+
+public class HiveClientParameters extends AbstractParameters {
+ /**
+ * 用HiveClient执行,最简单的方式就是将所有SQL全部贴进去即可,所以我们只需要一个SQL参数
+ */
+ private String sql;
+
+ public String getSql() {
+ return sql;
+ }
+
+ public void setSql(String sql) {
+ this.sql = sql;
+ }
+
+ @Override
+ public boolean checkParameters() {
+ return sql != null;
+ }
+
+ @Override
+ public List<ResourceInfo> getResourceFilesList() {
+ return null;
+ }
+}
+
+```
+
+
+实现了 Parameters 对象之后，我们具体实现 Task，例子中的实现比较简单，就是将用户的参数写入到文件中，通过 Hive -f 去执行任务。
+
+```plain
+ package org.apache.dolphinscheduler.plugin.task.hive;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractYarnTask;
+import org.apache.dolphinscheduler.spi.task.AbstractParameters;
+import org.apache.dolphinscheduler.spi.task.request.TaskRequest;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+
+
+public class HiveClientTask extends AbstractYarnTask {
+
+ /**
+ * hive client parameters
+ */
+ private HiveClientParameters hiveClientParameters;
+
+ /**
+ * taskExecutionContext
+ */
+ private final TaskRequest taskExecutionContext;
+
+
+
+ public HiveClientTask(TaskRequest taskRequest) {
+ super(taskRequest);
+ this.taskExecutionContext = taskRequest;
+ }
+
+ /**
+ * task init method
+ */
+ @Override
+ public void init() {
+ logger.info("hive client task param is {}", JSONUtils.toJsonString(taskExecutionContext));
+ this.hiveClientParameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), HiveClientParameters.class);
+
+ if (this.hiveClientParameters != null && !hiveClientParameters.checkParameters()) {
+ throw new RuntimeException("hive client task params is not valid");
+ }
+ }
+
+ /**
+ * build task execution command
+ *
+ * @return task execution command or null
+ */
+ @Override
+ protected String buildCommand() {
+ String filePath = getFilePath();
+ if (writeExecutionContentToFile(filePath)) {
+ return "hive -f " + filePath;
+ }
+ return null;
+ }
+
+ /**
+ * get hive sql write path
+ *
+ * @return file write path
+ */
+ private String getFilePath() {
+ return String.format("%s/hive-%s-%s.sql", this.taskExecutionContext.getExecutePath(), this.taskExecutionContext.getTaskName(), this.taskExecutionContext.getTaskInstanceId());
+ }
+
+ @Override
+ protected void setMainJarName() {
+ //do nothing
+ }
+
+ /**
+ * write hive sql to filepath
+ *
+ * @param filePath file path
+ * @return write success?
+ */
+ private boolean writeExecutionContentToFile(String filePath) {
+ Path path = Paths.get(filePath);
+ try (BufferedWriter writer = Files.newBufferedWriter(path, StandardCharsets.UTF_8)) {
+ writer.write(this.hiveClientParameters.getSql());
+ logger.info("file:" + filePath + "write success.");
+ return true;
+ } catch (IOException e) {
+ logger.error("file:" + filePath + "write failed.please path auth.");
+ e.printStackTrace();
+ return false;
+ }
+
+ }
+
+ @Override
+ public AbstractParameters getParameters() {
+ return this.hiveClientParameters;
+ }
+}
+
+```
+
+
+#### **5.6 遵守 SPI 规则**
+
+```plain
+ # 1,Resource下创建META-INF/services文件夹,创建接口全类名相同的文件
+zhang@xiaozhang resources % tree ./
+./
+└── META-INF
+ └── services
+ └── org.apache.dolphinscheduler.spi.task.TaskChannelFactory
+# 2,在文件中写入实现类的全限定类名
+zhang@xiaozhang resources % more META-INF/services/org.apache.dolphinscheduler.spi.task.TaskChannelFactory
+org.apache.dolphinscheduler.plugin.task.hive.HiveClientTaskChannelFactory
+
+```
+
+
+#### **5.7 打包和部署**
+
+```plain
+## 1,打包
+mvn clean install
+## 2,部署
+cp ./target/dolphinscheduler-task-hiveclient-1.0.jar $DOLPHINSCHEDULER_HOME/lib/
+## 3,restart dolphinscheduler server
+
+```
+
+
+以上操作完成后，我们查看 worker 日志 tail -200f $Apache DolphinScheduler_HOME/log/Apache DolphinScheduler-worker.log
+
+<div align=center>
+
+<img src="/img/2022-03-29/5.png"/>
+
+</div>
+
+Apache DolphinScheduler 的插件开发就到此完成~涉及到前端的修改可以参考：
+
+Apache DolphinScheduler-ui/src/js/conf/home/pages/dag/_source/formModel/
+
diff --git a/img/2022-03-29/1.png b/img/2022-03-29/1.png
new file mode 100644
index 000000000..17a7301c3
Binary files /dev/null and b/img/2022-03-29/1.png differ
diff --git a/img/2022-03-29/2.jpeg b/img/2022-03-29/2.jpeg
new file mode 100644
index 000000000..52752db71
Binary files /dev/null and b/img/2022-03-29/2.jpeg differ
diff --git a/img/2022-03-29/3.png b/img/2022-03-29/3.png
new file mode 100644
index 000000000..669723ab0
Binary files /dev/null and b/img/2022-03-29/3.png differ
diff --git a/img/2022-03-29/4.png b/img/2022-03-29/4.png
new file mode 100644
index 000000000..ec415f094
Binary files /dev/null and b/img/2022-03-29/4.png differ
diff --git a/img/2022-03-29/5.png b/img/2022-03-29/5.png
new file mode 100644
index 000000000..fca46c75f
Binary files /dev/null and b/img/2022-03-29/5.png differ
diff --git a/img/2022-03-29/En/1.png b/img/2022-03-29/En/1.png
new file mode 100644
index 000000000..64614bd95
Binary files /dev/null and b/img/2022-03-29/En/1.png differ
diff --git a/img/2022-03-29/En/2.png b/img/2022-03-29/En/2.png
new file mode 100644
index 000000000..22330da3f
Binary files /dev/null and b/img/2022-03-29/En/2.png differ
diff --git a/img/2022-03-29/En/3.png b/img/2022-03-29/En/3.png
new file mode 100644
index 000000000..4bc0b5418
Binary files /dev/null and b/img/2022-03-29/En/3.png differ
diff --git a/site_config/blog.js b/site_config/blog.js
index da28fd0f4..f784edab5 100644
--- a/site_config/blog.js
+++ b/site_config/blog.js
@@ -3,6 +3,14 @@ export default {
barText: 'Blog',
postsTitle: 'All posts',
list: [
+ {
+
+ title: 'Fast Task Type Expanding On Apache DolphinScheduler | Tutorial',
+ author: 'Debra Chen',
+ dateStr: '2022-4-14',
+ desc: 'At present, the scheduler plays an indispensable role in big data ecology... ',
+ link: '/en-us/blog/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.html',
+ },
{
title: 'First Anniversary Celebration of Apache DolphinScheduler\'s Graduation From ASF Incubator! Here Comes the Project Status Report!',
@@ -183,10 +191,17 @@ export default {
barText: '博客',
postsTitle: '所有文章',
list: [
+ {
+ title: '极速开发扩充 Apache DolphinScheduler Task 类型 | 实用教程',
+ author: 'Debra Chen',
+ dateStr: '2022-4-14',
+ desc: '目前在大数据生态中，调度系统是不可或缺的一个重要组件......',
+ link: '/zh-cn/blog/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.html',
+ },
{
title: 'Apache DolphinScheduler ASF 孵化器毕业一周年，汇报来了！',
author: 'Debra Chen',
- dateStr: '2022-3-21',
+ dateStr: '2022-4-14',
desc: '不知不觉，Apache DolphinScheduler 已经从......',
link: '/zh-cn/blog/Apache_DolphinScheduler_s_Graduation_From_ASF_Incubator.html',
},
diff --git a/site_config/home.jsx b/site_config/home.jsx
index e51aba84d..51e0c9132 100644
--- a/site_config/home.jsx
+++ b/site_config/home.jsx
@@ -54,6 +54,13 @@ export default {
events: {
title: '事件 & 新闻',
list: [
+ {
+ img: '/img/3-16/1.png',
+ title: '极速开发扩充 Apache DolphinScheduler Task 类型 | 实用教程',
+ content: '目前在大数据生态中，调度系统是不可或缺的一个重要组件..',
+ dateStr: '2022-4-14',
+ link: '/zh-cn/blog/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.html',
+ },
{
img: '/img/2022-4-14/1.png',
title: 'Apache DolphinScheduler ASF 孵化器毕业一周年，汇报来了！',
@@ -68,20 +75,6 @@ export default {
dateStr: '2022-3-21',
link: '/zh-cn/blog/K8s_Cisco_Hangzhou.html',
},
- {
- img: '/img/3-16/1.png',
- title: '杭州思科对 Apache DolphinScheduler Alert 模块的改造',
- content: '杭州思科已经将 Apache DolphinScheduler 引入公司自建的大数据平台..',
- dateStr: '2022-3-7',
- link: '/zh-cn/blog/Hangzhou_cisco.html',
- },
- {
- img: '/img/2022-3-11/1.jpeg',
- title: '日均处理 10000+ 工作流实例，Apache DolphinScheduler 在 360 数科的实践',
- content: '从 2020 年起，360 数科全面将调度系统从 Azkaban 迁移到 Apache DolphinScheduler...',
- dateStr: '2022-3-15',
- link: '/zh-cn/blog/How_Does_360_DIGITECH_process_10_000+_workflow_instances_per_day.html',
- },
],
},
ourusers: {
@@ -553,6 +546,14 @@ export default {
events: {
title: 'Events & News',
list: [
+ {
+
+ img: '/img/2022-03-29/En/1.png"',
+ title: 'Fast Task Type Expanding On Apache DolphinScheduler | Tutorial',
+ content: 'At present, the scheduler plays an indispensable role in big data ecology....',
+ dateStr: '2022-4-14',
+ link: '/en-us/blog/Fast_Task_Type_Expanding_On_Apache_DolphinScheduler_Tutorial.html',
+ },
{
img: '/img/2022-4-14/3.png',
title: 'First Anniversary Celebration of Apache DolphinScheduler\'s Graduation From ASF Incubator! Here Comes the Project Status Report!\n',
@@ -567,21 +568,7 @@ export default {
dateStr: '2022-3-10',
link: '/en-us/blog/K8s_Cisco_Hangzhou.html',
},
- {
- img: '/img/3-16/1.png',
- title: 'Cisco Hangzhou\'s Travel Through Apache DolphinScheduler Alert Module Refactor',
- content: 'Cisco Hangzhou has introduced Apache DolphinScheduler....',
- dateStr: '2022-3-7',
- link: '/en-us/blog/Hangzhou_cisco.html',
- },
- {
- img: '/img/2022-3-11/1.jpeg',
- title: 'How Does 360 DIGITECH process 10,000+ workflow instances per day by Apache DolphinScheduler',
- content: 'ince 2020, 360 DIGITECH has fully migrated its scheduling system from Azkaban to Apache DolphinScheduler....',
- dateStr: '2022-2-24',
- link: '/en-us/blog/How_Does_360_DIGITECH_process_10_000+_workflow_instances_per_day.html',
- },
],
},
userreview: {