You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2022/04/15 12:15:31 UTC
[incubator-inlong-website] 01/01: [INLONG-3740][Doc] Rerank the plugin guide documents
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch fix-3740
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
commit 70fe33feb4f7eddd98de361b718df9d7153cbb23
Author: dockerzhang <do...@apache.org>
AuthorDate: Fri Apr 15 20:15:16 2022 +0800
[INLONG-3740][Doc] Rerank the plugin guide documents
---
...ow_to_write_plugin_sort.md => basic_concept.md} | 0
docs/design_and_concept/extend_sink_in_sort.md | 107 ------------------
.../how_to_write_plugin_agent.md | 2 +-
.../how_to_write_plugin_dashboard.md | 2 +-
.../how_to_write_plugin_manager.md | 2 +-
.../design_and_concept/how_to_write_plugin_sort.md | 119 ++++++++++++++++++---
docs/user_guide/dashboard_usage.md | 2 +-
.../how_to_write_plugin_agent.md | 2 +-
.../how_to_write_plugin_dashboard.md | 2 +-
.../how_to_write_plugin_manager.md | 2 +-
.../current/user_guide/dashboard_usage.md | 2 +-
11 files changed, 112 insertions(+), 130 deletions(-)
diff --git a/docs/design_and_concept/how_to_write_plugin_sort.md b/docs/design_and_concept/basic_concept.md
similarity index 100%
copy from docs/design_and_concept/how_to_write_plugin_sort.md
copy to docs/design_and_concept/basic_concept.md
diff --git a/docs/design_and_concept/extend_sink_in_sort.md b/docs/design_and_concept/extend_sink_in_sort.md
deleted file mode 100644
index ca589372a..000000000
--- a/docs/design_and_concept/extend_sink_in_sort.md
+++ /dev/null
@@ -1,107 +0,0 @@
----
-title: Sort Plugin
-sidebar_position: 3
----
-
-# Overview
-InLong-Sort is known as a real-time ETL system. Currently, supported sinks are hive, kafka, clickhouse and iceberg.
-This article introduces how to extend a new type of sink in InLong-Sort.
-
-# Extend a new sink function
-InLong-Sort is based on flink, when extending a new type of sink in InLong-Sort, either a new type of flink sink
-or a predefined sink in flink is required.
-refer to [DataStream Connectors ](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/overview/#datastream-connectors).
-
-# Extend a new sink protocol
-Firstly, implement a new sink protocol which extends
-`inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/sink/SinkInfo.java`
-
-All necessary attributes used by the corresponding flink sink which extended before should be placed in the protocol
-
-Example
-```java
-public class DemoSinkInfo extends SinkInfo {
-
- // Place necessary attributes here
-
- @JsonCreator
- public DemoSinkInfo(FieldInfo[] fields) {
- super(fields);
- }
-}
-```
-
-Secondly, mark the new sink protocol as subtype of SinkInfo and give it a name
-
-Example : A new sink protocol DemoSinkInfo whose subtype name is `Constants.SINK_TYPE_DEMO`
-```java
-/**
- * The base class of the data sink in the metadata.
- */
-@JsonTypeInfo(
- use = JsonTypeInfo.Id.NAME,
- include = JsonTypeInfo.As.PROPERTY,
- property = "type")
-@JsonSubTypes({
- @Type(value = ClickHouseSinkInfo.class, name = Constants.SINK_TYPE_CLICKHOUSE),
- @Type(value = HiveSinkInfo.class, name = Constants.SINK_TYPE_HIVE),
- @Type(value = KafkaSinkInfo.class, name = Constants.SINK_TYPE_KAFKA),
- @Type(value = IcebergSinkInfo.class, name = Constants.SINK_TYPE_ICEBERG),
-
- // The new sink protocol
- @Type(value = DemoSinkInfo.class, name = Constants.SINK_TYPE_DEMO)}
-)
-public abstract class SinkInfo implements Serializable {
-
- private static final long serialVersionUID = 1485856855405721745L;
-
- @JsonProperty("fields")
- private final FieldInfo[] fields;
-
- public SinkInfo(@JsonProperty("fields") FieldInfo[] fields) {
- this.fields = checkNotNull(fields);
- }
-
- @JsonProperty("fields")
- public FieldInfo[] getFields() {
- return fields;
- }
-}
-```
-
-# Bundle the new sink, make it an alternative sink in Inlong_sort
-After extending a new flink sink and a new sink protocol in InLong-Sort, we can bundle the new sink into the flink job
-(the entrance of InLong-Sort)
-
-The entrance of the flink job is `inlong-sort/sort-single-tenant/src/main/java/org/apache/inlong/sort/singletenant/flink/Entrance.java`
-
-Example
-
-``` java
-private static void buildSinkStream(
- DataStream<Row> sourceStream,
- Configuration config,
- SinkInfo sinkInfo,
- Map<String, Object> properties,
- long dataflowId) throws IOException, ClassNotFoundException {
- final String sinkType = checkNotNull(config.getString(Constants.SINK_TYPE));
-
- switch (sinkType) {
- case Constants.SINK_TYPE_CLICKHOUSE:
- break;
- case Constants.SINK_TYPE_HIVE:
- break;
- case Constants.SINK_TYPE_ICEBERG:
- break;
- case Constants.SINK_TYPE_KAFKA:
- break;
- case Constants.SINK_TYPE_DEMO:
- // Add the extended sink function here
- break;
- default:
- throw new IllegalArgumentException("Unsupported sink type " + sinkType);
- }
-
-}
-
-```
\ No newline at end of file
diff --git a/docs/design_and_concept/how_to_write_plugin_agent.md b/docs/design_and_concept/how_to_write_plugin_agent.md
index 9e1315208..26ad07da7 100644
--- a/docs/design_and_concept/how_to_write_plugin_agent.md
+++ b/docs/design_and_concept/how_to_write_plugin_agent.md
@@ -1,6 +1,6 @@
---
title: Agent Plugin
-sidebar_position: 1
+sidebar_position: 2
---
# Overview
diff --git a/docs/design_and_concept/how_to_write_plugin_dashboard.md b/docs/design_and_concept/how_to_write_plugin_dashboard.md
index fe7ab6b3f..4743e77c0 100644
--- a/docs/design_and_concept/how_to_write_plugin_dashboard.md
+++ b/docs/design_and_concept/how_to_write_plugin_dashboard.md
@@ -1,6 +1,6 @@
---
title: Dashboard Plugin
-sidebar_position: 4
+sidebar_position: 5
---
# Overview
diff --git a/docs/design_and_concept/how_to_write_plugin_manager.md b/docs/design_and_concept/how_to_write_plugin_manager.md
index 9cbf48c66..8f2e1da6c 100644
--- a/docs/design_and_concept/how_to_write_plugin_manager.md
+++ b/docs/design_and_concept/how_to_write_plugin_manager.md
@@ -1,6 +1,6 @@
---
title: Manager Plugin
-sidebar_position: 2
+sidebar_position: 4
---
# Overview
diff --git a/docs/design_and_concept/how_to_write_plugin_sort.md b/docs/design_and_concept/how_to_write_plugin_sort.md
index 7f7e9a7bf..ca589372a 100644
--- a/docs/design_and_concept/how_to_write_plugin_sort.md
+++ b/docs/design_and_concept/how_to_write_plugin_sort.md
@@ -1,18 +1,107 @@
---
-title: Basic Concept
-sidebar_position: 1
+title: Sort Plugin
+sidebar_position: 3
---
-| Name | Description | Other |
-|------------|----------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|
-| Group | Data Streams Group, it contains multiple data streams, and one Group represents one data ingestion. | Group has attributes such as ID and Name. |
-| Stream | Data Stream, a stream has a specific flow direction. | Stream has attributes such as ID, Name, and data fields. |
-| Node | Data Node, including `Extract Node` and `Load Node`, stands for the data source and sink types separately. | |
-| InLongMsg | InLong data format, if you consume message directly from the message queue, you need to perform `InLongMsg` parsing first. | |
-| Agent | Represents various collection capabilities. | It contains File Agent, SQL Agent, Binlog Agent, etc. |
-| DataProxy | Forward received data to different message queues. | Supports data transmission blocking, placing retransmission. |
-| Sort | Data stream sorting | Sort-flink based on Flink, sort-standalone for local sorting. |
-| TubeMQ | InLong's self-developed message queuing service | It can also be called Tube, with low-cost, high-performance features. |
-| Pulsar | [Apache Pulsar](https://pulsar.apache.org/), a high-performance, high-consistency message queue service | |
-| Hive | [Apache Hive](https://hive.apache.org/), a data warehouse built on the Hadoop architecture | |
-| ClickHouse | [ClickHouse](https://clickhouse.com/), a high performance columnar OLAP database | |
\ No newline at end of file
+# Overview
+InLong-Sort is known as a real-time ETL system. Currently, supported sinks are hive, kafka, clickhouse and iceberg.
+This article introduces how to extend a new type of sink in InLong-Sort.
+
+# Extend a new sink function
+InLong-Sort is based on flink, when extending a new type of sink in InLong-Sort, either a new type of flink sink
+or a predefined sink in flink is required.
+refer to [DataStream Connectors ](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/overview/#datastream-connectors).
+
+# Extend a new sink protocol
+Firstly, implement a new sink protocol which extends
+`inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/sink/SinkInfo.java`
+
+All necessary attributes used by the corresponding flink sink which extended before should be placed in the protocol
+
+Example
+```java
+public class DemoSinkInfo extends SinkInfo {
+
+ // Place necessary attributes here
+
+ @JsonCreator
+ public DemoSinkInfo(FieldInfo[] fields) {
+ super(fields);
+ }
+}
+```
+
+Secondly, mark the new sink protocol as subtype of SinkInfo and give it a name
+
+Example : A new sink protocol DemoSinkInfo whose subtype name is `Constants.SINK_TYPE_DEMO`
+```java
+/**
+ * The base class of the data sink in the metadata.
+ */
+@JsonTypeInfo(
+ use = JsonTypeInfo.Id.NAME,
+ include = JsonTypeInfo.As.PROPERTY,
+ property = "type")
+@JsonSubTypes({
+ @Type(value = ClickHouseSinkInfo.class, name = Constants.SINK_TYPE_CLICKHOUSE),
+ @Type(value = HiveSinkInfo.class, name = Constants.SINK_TYPE_HIVE),
+ @Type(value = KafkaSinkInfo.class, name = Constants.SINK_TYPE_KAFKA),
+ @Type(value = IcebergSinkInfo.class, name = Constants.SINK_TYPE_ICEBERG),
+
+ // The new sink protocol
+ @Type(value = DemoSinkInfo.class, name = Constants.SINK_TYPE_DEMO)}
+)
+public abstract class SinkInfo implements Serializable {
+
+ private static final long serialVersionUID = 1485856855405721745L;
+
+ @JsonProperty("fields")
+ private final FieldInfo[] fields;
+
+ public SinkInfo(@JsonProperty("fields") FieldInfo[] fields) {
+ this.fields = checkNotNull(fields);
+ }
+
+ @JsonProperty("fields")
+ public FieldInfo[] getFields() {
+ return fields;
+ }
+}
+```
+
+# Bundle the new sink, make it an alternative sink in Inlong_sort
+After extending a new flink sink and a new sink protocol in InLong-Sort, we can bundle the new sink into the flink job
+(the entrance of InLong-Sort)
+
+The entrance of the flink job is `inlong-sort/sort-single-tenant/src/main/java/org/apache/inlong/sort/singletenant/flink/Entrance.java`
+
+Example
+
+``` java
+private static void buildSinkStream(
+ DataStream<Row> sourceStream,
+ Configuration config,
+ SinkInfo sinkInfo,
+ Map<String, Object> properties,
+ long dataflowId) throws IOException, ClassNotFoundException {
+ final String sinkType = checkNotNull(config.getString(Constants.SINK_TYPE));
+
+ switch (sinkType) {
+ case Constants.SINK_TYPE_CLICKHOUSE:
+ break;
+ case Constants.SINK_TYPE_HIVE:
+ break;
+ case Constants.SINK_TYPE_ICEBERG:
+ break;
+ case Constants.SINK_TYPE_KAFKA:
+ break;
+ case Constants.SINK_TYPE_DEMO:
+ // Add the extended sink function here
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported sink type " + sinkType);
+ }
+
+}
+
+```
\ No newline at end of file
diff --git a/docs/user_guide/dashboard_usage.md b/docs/user_guide/dashboard_usage.md
index 02e087ea7..935b484e8 100644
--- a/docs/user_guide/dashboard_usage.md
+++ b/docs/user_guide/dashboard_usage.md
@@ -1,5 +1,5 @@
---
-title: Dashboard Usage Guide
+title: Dashboard Guide
sidebar_position: 1
---
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_agent.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_agent.md
index 27c5a6836..22f8e5d39 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_agent.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_agent.md
@@ -1,6 +1,6 @@
---
title: Agent 插件
-sidebar_position: 1
+sidebar_position: 2
---
# 总览
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
index 34e023ca8..2a2b0514c 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
@@ -1,6 +1,6 @@
---
title: Dashboard 插件
-sidebar_position: 4
+sidebar_position: 5
---
# 总览
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_manager.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_manager.md
index f790d6d65..e2f4f738a 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_manager.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_manager.md
@@ -1,6 +1,6 @@
---
title: Manager 插件
-sidebar_position: 2
+sidebar_position: 4
---
# 总览
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/user_guide/dashboard_usage.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/user_guide/dashboard_usage.md
index 25f509c63..49a90127f 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/user_guide/dashboard_usage.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/user_guide/dashboard_usage.md
@@ -1,5 +1,5 @@
---
-title: Dashboard 使用指引
+title: Dashboard 指引
sidebar_position: 1
---