You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2022/06/21 07:31:38 UTC
[inlong-website] branch master updated: [INLONG-4716][Doc] Update the plugin file name (#435)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 884ba148a [INLONG-4716][Doc] Update the plugin file name (#435)
884ba148a is described below
commit 884ba148ae75e4b612ef48ac177876bebe1b72ed
Author: Charles Zhang <do...@apache.org>
AuthorDate: Tue Jun 21 15:31:33 2022 +0800
[INLONG-4716][Doc] Update the plugin file name (#435)
---
...de_plugin.md => how_to_extend_data_node_for_manager.md} | 8 ++++----
...load_node_en.md => how_to_extend_data_node_for_sort.md} | 8 ++++----
...de_plugin.md => how_to_extend_data_node_for_manager.md} | 14 +++++---------
...d_node_ch.md => how_to_extend_data_node_for_sort.md.md} | 12 ++++++------
4 files changed, 19 insertions(+), 23 deletions(-)
diff --git a/docs/design_and_concept/inlong_data_node_plugin.md b/docs/design_and_concept/how_to_extend_data_node_for_manager.md
similarity index 78%
rename from docs/design_and_concept/inlong_data_node_plugin.md
rename to docs/design_and_concept/how_to_extend_data_node_for_manager.md
index e005be1ca..a06cabbbe 100644
--- a/docs/design_and_concept/inlong_data_node_plugin.md
+++ b/docs/design_and_concept/how_to_extend_data_node_for_manager.md
@@ -1,19 +1,19 @@
---
-title: Data Node Plugin
+title: Manager Plugin for Data Node
sidebar_position: 5
---
## Overview
Inlong is aimed at create dataflow between different data sources, now Inlong has support several universal data sources such as **MySQL**, **Apache Kafka**, **ClickHouse** on Input/Output respectively,
-You can refer to [data_node](https://inlong.apache.org/docs/next/data_node/extract_node/auto_push) for specific information.
+You can refer to [Data Node](data_node/extract_node/overview.md) for specific information.
We Plan to support more data sources in the future, and this article is a development manual to extend data nodes.
## Extend Extract Node
In order to extend an input data sources , also refered to **extract node** in Inlong. We take **MySQL_BINLOG** for example.
-- Develop extract node plugin in sort, refer to [how_to_write_plugin_sort](https://inlong.apache.org/docs/next/design_and_concept/how_to_write_plugin_sort)
+- Develop extract node plugin in sort, refer to [Sort Plugin](design_and_concept/how_to_extend_data_node_for_sort.md)
- Add **TaskType** in `org.apache.inlong.common.enums.TaskTypeEnum`
- Add **SourceType** in `org.apache.inlong.manager.common.enums.SourceType`
- Create new package under package path: `org.apache.inlong.manager.common.pojo.source`, develop every entity class needed.
@@ -25,7 +25,7 @@ In order to extend an input data sources , also refered to **extract node** in I
The way to extend data sink is nearly the same with data source
-- Develop load node plugin in sort, refer to [how_to_write_plugin_sort](https://inlong.apache.org/docs/next/design_and_concept/how_to_write_plugin_sort)
+- Develop load node plugin in sort, refer to [Sort Plugin](design_and_concept/how_to_extend_data_node_for_sort.md)
- Add **SinkType** in `org.apache.inlong.manager.common.enums.SinkType`
- Create new package under package path: `org.apache.inlong.manager.common.pojo.sink`, develop every entity class needed.
- Create Operation class for new data source under package path: `org.apache.inlong.manager.service.sink`.
diff --git a/docs/design_and_concept/how_to_extend_extract_or_load_node_en.md b/docs/design_and_concept/how_to_extend_data_node_for_sort.md
similarity index 94%
rename from docs/design_and_concept/how_to_extend_extract_or_load_node_en.md
rename to docs/design_and_concept/how_to_extend_data_node_for_sort.md
index 69e404eac..af6e6d944 100644
--- a/docs/design_and_concept/how_to_extend_extract_or_load_node_en.md
+++ b/docs/design_and_concept/how_to_extend_data_node_for_sort.md
@@ -5,9 +5,9 @@ sidebar_position: 3
## Overview
-InLong-Sort is known as a real-time ETL system. Currently, supported extract or load includes FileSystemExtractNode, KafkaExtractNode, MongoExtractNode, MySqlExtractNode, OracleExtractNode , PostgresExtractNode , PulsarExtractNode, SqlServerExtractNode, ClickHouseLoadNode, ElasticsearchLoadNode, FileSystemLoadNode, GreenplumLoadNode, HbaseLoadNode, HiveLoadNode, IcebergLoadNode, KafkaLoadNode, MySqlLoadNode, OracleLoadNode, PostgresLoadNode, SqlServerLoadNode, TDSQLPostgresLoadNode, etc. [...]
+InLong Sort is known as a real-time ETL system. Currently, supported extract or load includes FileSystemExtractNode, KafkaExtractNode, MongoExtractNode, MySqlExtractNode, OracleExtractNode , PostgresExtractNode , PulsarExtractNode, SqlServerExtractNode, ClickHouseLoadNode, ElasticsearchLoadNode, FileSystemLoadNode, GreenplumLoadNode, HbaseLoadNode, HiveLoadNode, IcebergLoadNode, KafkaLoadNode, MySqlLoadNode, OracleLoadNode, PostgresLoadNode, SqlServerLoadNode, TDSQLPostgresLoadNode, etc. [...]
-This article describes how to extend a new source (abstracted as extract node in inlong) or a new sink (abstracted as load node in inlong) in InLong-Sort. After understanding the InLong-Sort architecture, you can understand how the source corresponds to the extract node, and how the sink corresponds to the load node. The architecture of inlong sort can be represented by UML object relation diagram as:
+This article describes how to extend a new source (abstracted as extract node in inlong) or a new sink (abstracted as load node in inlong) in InLong Sort. After understanding the InLong Sort architecture, you can understand how the source corresponds to the extract node, and how the sink corresponds to the load node. The architecture of inlong sort can be represented by UML object relation diagram as:
@@ -156,9 +156,9 @@ public interface Node {...}
**Step 3**：Extend the Sort connector, Kafka's sort connector is in inlong/inlong-sort/sort-connectors/kafka.
-## Bundle extract node and load node into InLong-Sort
+## Bundle extract node and load node into InLong Sort
-To integrate extract and load into the InLong-Sort mainstream, you need to implement the semantics mentioned in the overview section: group, stream, node, etc. The entry class of InLong-sort is in `inlong-sort/sort-core/src/main/java/org/apache/inlong/sort/Entrance.java`. How to integrate extract and load into InLong-Sort can refer to the following ut. First, build the corresponding extractnode and loadnode, then build noderelation, streaminfo and groupinfo, and finally use FlinkSqlParse [...]
+To integrate extract and load into the InLong Sort mainstream, you need to implement the semantics mentioned in the overview section: group, stream, node, etc. The entry class of InLong Sort is in `inlong-sort/sort-core/src/main/java/org/apache/inlong/sort/Entrance.java`. How to integrate extract and load into InLong Sort can refer to the following ut. First, build the corresponding extractnode and loadnode, then build noderelation, streaminfo and groupinfo, and finally use FlinkSqlParse [...]
```java
public class MongoExtractToKafkaLoad extends AbstractTestBase {
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/inlong_data_node_plugin.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_manager.md
similarity index 72%
rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/inlong_data_node_plugin.md
rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_manager.md
index c29e57355..f06171a7e 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/inlong_data_node_plugin.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_manager.md
@@ -1,19 +1,19 @@
---
-title: 数据节点插件
+title: Manager 数据节点插件
sidebar_position: 5
---
## 总览
Inlong 设计初衷即是为了在不同数据源之间创建数据流，截止目前，Inlong 已经支持多种常用数据源的读取和写入，如 **MySQL**, **Apache Kafka**, **ClickHouse** 等，
-详细内容可参考 [数据节点](https://inlong.apache.org/zh-CN/docs/next/data_node/extract_node/auto_push)。
+详细内容可参考 [数据节点](data_node/extract_node/overview.md)。
我们会在未来支持更多的常用数据源，故本文会简短介绍如何在现有框架下扩展数据源。
## 扩展读取节点
以 **MySQL_BINLOG** 为例，介绍如何在 Inlong 框架下扩展读取节点。
-- 首先需要在 Sort 组件内支持该数据源，详情参考 [Sort 插件](https://inlong.apache.org/zh-CN/docs/next/design_and_concept/how_to_write_plugin_sort)
+- 首先需要在 Sort 组件内支持该数据源，详情参考 [Sort 插件](design_and_concept/how_to_extend_data_node_for_sort.md)
- 在枚举类`org.apache.inlong.common.enums.TaskTypeEnum`中增加对应的枚举
- 在枚举类`org.apache.inlong.manager.common.enums.SourceType`中同样增加对应枚举
- 在`org.apache.inlong.manager.common.pojo.source`路径下创建文件夹，创建对应实体类
@@ -24,12 +24,8 @@ Inlong 设计初衷即是为了在不同数据源之间创建数据流，截止
扩展写入节点的方式与读取节点类似
-- 首先需要在 Sort 组件内支持该数据源，详情参考 [Sort 插件](https://inlong.apache.org/zh-CN/docs/next/design_and_concept/how_to_write_plugin_sort)
+- 首先需要在 Sort 组件内支持该数据源，详情参考 [Sort 插件](design_and_concept/how_to_extend_data_node_for_sort.md)
- 在枚举类`org.apache.inlong.manager.common.enums.SinkType`中增加对应枚举
- 在`org.apache.inlong.manager.common.pojo.sink`路径下创建文件夹, 创建对应实体类
- 在`org.apache.inlong.manager.service.sink`路径下创建对应工具类
-- 支持数据源到**LoadNode**的转换函数，参考代码 `org.apache.inlong.manager.service.sort.util.LoadNodeUtils`
-
-## 写在最后
-
-如果你有任何优化的建议，请通过邮件或以创建 [ISSUE](https://github.com/apache/inlong/issues/new/choose) 的形式告诉我们。
+- 支持数据源到**LoadNode**的转换函数，参考代码 `org.apache.inlong.manager.service.sort.util.LoadNodeUtils`
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_extract_or_load_node_ch.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_sort.md.md
similarity index 94%
rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_extract_or_load_node_ch.md
rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_sort.md.md
index 66df2cb6c..b26fe5179 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_extract_or_load_node_ch.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_sort.md.md
@@ -5,9 +5,9 @@ sidebar_position: 3
## 总览
-InLong-Sort 是一个 ETL 系统，当前支持的 extract 或 load 包括 FileSystemExtractNode，KafkaExtractNode，MongoExtractNode，MySqlExtractNode，OracleExtractNode，PostgresExtractNode，PulsarExtractNode，SqlServerExtractNode，ClickHouseLoadNode ，ElasticsearchLoadNode，FileSystemLoadNode，GreenplumLoadNode，HbaseLoadNode，HiveLoadNode，IcebergLoadNode，KafkaLoadNode，MySqlLoadNode，OracleLoadNode，PostgresLoadNode，SqlServerLoadNode，TDSQLPostgresLoadNode 等。InLong-Sort是基于Flink SQL的ETL方案，Flink SQL强大的表达能力带来的高可扩展性、灵活性，基本上 Flink SQ [...]
+InLong Sort 是一个 ETL 系统，当前支持的 extract 或 load 包括 FileSystemExtractNode，KafkaExtractNode，MongoExtractNode，MySqlExtractNode，OracleExtractNode，PostgresExtractNode，PulsarExtractNode，SqlServerExtractNode，ClickHouseLoadNode ，ElasticsearchLoadNode，FileSystemLoadNode，GreenplumLoadNode，HbaseLoadNode，HiveLoadNode，IcebergLoadNode，KafkaLoadNode，MySqlLoadNode，OracleLoadNode，PostgresLoadNode，SqlServerLoadNode，TDSQLPostgresLoadNode 等。InLong Sort是基于Flink SQL的ETL方案，Flink SQL强大的表达能力带来的高可扩展性、灵活性，基本上 Flink SQ [...]
-本文介绍如何在 InLong-Sort 中扩展一个新的 source（在 InLong 中抽象为 Extract Node）或一个新的 sink（在InLong中抽象为 Load Node ）。在弄清楚 InLong 的架构之后，就可以明白 Source 与 Extract Node 如何对应，Sink 与 Load Node 如何对应。InLong-Sort 架构的 UML 对象关系图如下：
+本文介绍如何在 InLong Sort 中扩展一个新的 source（在 InLong 中抽象为 Extract Node）或一个新的 sink（在InLong中抽象为 Load Node ）。在弄清楚 InLong 的架构之后，就可以明白 Source 与 Extract Node 如何对应，Sink 与 Load Node 如何对应。InLong Sort 架构的 UML 对象关系图如下：
@@ -157,12 +157,12 @@ public interface Node {...}
**第三步**：扩展 Sort Connector，Kafka 的 sort connector 在 `inlong/inlong-sort/sort-connectors/kafka` 目录下。
-## 集成 Extract 和 Load 到 InLong-Sort 主流程
+## 集成 Extract 和 Load 到 InLong Sort 主流程
-将 Extract 和 Load 集成到 InLong-Sort 主流程中，需要构建总览小节中提到的语意：Group、Stream、Node 等。
-InLong-Sort 的入口类在`inlong-sort/sort-core/src/main/java/org/apache/inlong/sort/Entrance.java`。
+将 Extract 和 Load 集成到 InLong Sort 主流程中，需要构建总览小节中提到的语意：Group、Stream、Node 等。
+InLong Sort 的入口类在`inlong-sort/sort-core/src/main/java/org/apache/inlong/sort/Entrance.java`。
-Extract 和 Load 如何集成至 InLong-Sort，可参考下面的 UT，首先构建对应的 ExtractNode、LoadNode，再构建 NodeRelation、StreamInfo、GroupInfo，最后通过 FlinkSqlParser 执行。
+Extract 和 Load 如何集成至 InLong Sort，可参考下面的 UT，首先构建对应的 ExtractNode、LoadNode，再构建 NodeRelation、StreamInfo、GroupInfo，最后通过 FlinkSqlParser 执行。
```java
public class MongoExtractToKafkaLoad extends AbstractTestBase {