You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2022/04/15 12:15:30 UTC
[incubator-inlong-website] branch fix-3740 created (now 70fe33feb)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a change to branch fix-3740
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
at 70fe33feb [INLONG-3740][Doc] Rerank the plugin guide documents
This branch includes the following new commits:
new 70fe33feb [INLONG-3740][Doc] Rerank the plugin guide documents
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[incubator-inlong-website] 01/01: [INLONG-3740][Doc] Rerank the plugin guide documents
Posted by do...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch fix-3740
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
commit 70fe33feb4f7eddd98de361b718df9d7153cbb23
Author: dockerzhang <do...@apache.org>
AuthorDate: Fri Apr 15 20:15:16 2022 +0800
[INLONG-3740][Doc] Rerank the plugin guide documents
---
...ow_to_write_plugin_sort.md => basic_concept.md} | 0
docs/design_and_concept/extend_sink_in_sort.md | 107 ------------------
.../how_to_write_plugin_agent.md | 2 +-
.../how_to_write_plugin_dashboard.md | 2 +-
.../how_to_write_plugin_manager.md | 2 +-
.../design_and_concept/how_to_write_plugin_sort.md | 119 ++++++++++++++++++---
docs/user_guide/dashboard_usage.md | 2 +-
.../how_to_write_plugin_agent.md | 2 +-
.../how_to_write_plugin_dashboard.md | 2 +-
.../how_to_write_plugin_manager.md | 2 +-
.../current/user_guide/dashboard_usage.md | 2 +-
11 files changed, 112 insertions(+), 130 deletions(-)
diff --git a/docs/design_and_concept/how_to_write_plugin_sort.md b/docs/design_and_concept/basic_concept.md
similarity index 100%
copy from docs/design_and_concept/how_to_write_plugin_sort.md
copy to docs/design_and_concept/basic_concept.md
diff --git a/docs/design_and_concept/extend_sink_in_sort.md b/docs/design_and_concept/extend_sink_in_sort.md
deleted file mode 100644
index ca589372a..000000000
--- a/docs/design_and_concept/extend_sink_in_sort.md
+++ /dev/null
@@ -1,107 +0,0 @@
----
-title: Sort Plugin
-sidebar_position: 3
----
-
-# Overview
-InLong-Sort is known as a real-time ETL system. Currently, supported sinks are hive, kafka, clickhouse and iceberg.
-This article introduces how to extend a new type of sink in InLong-Sort.
-
-# Extend a new sink function
-InLong-Sort is based on flink, when extending a new type of sink in InLong-Sort, either a new type of flink sink
-or a predefined sink in flink is required.
-refer to [DataStream Connectors ](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/overview/#datastream-connectors).
-
-# Extend a new sink protocol
-Firstly, implement a new sink protocol which extends
-`inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/sink/SinkInfo.java`
-
-All necessary attributes used by the corresponding flink sink which extended before should be placed in the protocol
-
-Example
-```java
-public class DemoSinkInfo extends SinkInfo {
-
- // Place necessary attributes here
-
- @JsonCreator
- public DemoSinkInfo(FieldInfo[] fields) {
- super(fields);
- }
-}
-```
-
-Secondly, mark the new sink protocol as subtype of SinkInfo and give it a name
-
-Example : A new sink protocol DemoSinkInfo whose subtype name is `Constants.SINK_TYPE_DEMO`
-```java
-/**
- * The base class of the data sink in the metadata.
- */
-@JsonTypeInfo(
- use = JsonTypeInfo.Id.NAME,
- include = JsonTypeInfo.As.PROPERTY,
- property = "type")
-@JsonSubTypes({
- @Type(value = ClickHouseSinkInfo.class, name = Constants.SINK_TYPE_CLICKHOUSE),
- @Type(value = HiveSinkInfo.class, name = Constants.SINK_TYPE_HIVE),
- @Type(value = KafkaSinkInfo.class, name = Constants.SINK_TYPE_KAFKA),
- @Type(value = IcebergSinkInfo.class, name = Constants.SINK_TYPE_ICEBERG),
-
- // The new sink protocol
- @Type(value = DemoSinkInfo.class, name = Constants.SINK_TYPE_DEMO)}
-)
-public abstract class SinkInfo implements Serializable {
-
- private static final long serialVersionUID = 1485856855405721745L;
-
- @JsonProperty("fields")
- private final FieldInfo[] fields;
-
- public SinkInfo(@JsonProperty("fields") FieldInfo[] fields) {
- this.fields = checkNotNull(fields);
- }
-
- @JsonProperty("fields")
- public FieldInfo[] getFields() {
- return fields;
- }
-}
-```
-
-# Bundle the new sink, make it an alternative sink in Inlong_sort
-After extending a new flink sink and a new sink protocol in InLong-Sort, we can bundle the new sink into the flink job
-(the entrance of InLong-Sort)
-
-The entrance of the flink job is `inlong-sort/sort-single-tenant/src/main/java/org/apache/inlong/sort/singletenant/flink/Entrance.java`
-
-Example
-
-``` java
-private static void buildSinkStream(
- DataStream<Row> sourceStream,
- Configuration config,
- SinkInfo sinkInfo,
- Map<String, Object> properties,
- long dataflowId) throws IOException, ClassNotFoundException {
- final String sinkType = checkNotNull(config.getString(Constants.SINK_TYPE));
-
- switch (sinkType) {
- case Constants.SINK_TYPE_CLICKHOUSE:
- break;
- case Constants.SINK_TYPE_HIVE:
- break;
- case Constants.SINK_TYPE_ICEBERG:
- break;
- case Constants.SINK_TYPE_KAFKA:
- break;
- case Constants.SINK_TYPE_DEMO:
- // Add the extended sink function here
- break;
- default:
- throw new IllegalArgumentException("Unsupported sink type " + sinkType);
- }
-
-}
-
-```
\ No newline at end of file
diff --git a/docs/design_and_concept/how_to_write_plugin_agent.md b/docs/design_and_concept/how_to_write_plugin_agent.md
index 9e1315208..26ad07da7 100644
--- a/docs/design_and_concept/how_to_write_plugin_agent.md
+++ b/docs/design_and_concept/how_to_write_plugin_agent.md
@@ -1,6 +1,6 @@
---
title: Agent Plugin
-sidebar_position: 1
+sidebar_position: 2
---
# Overview
diff --git a/docs/design_and_concept/how_to_write_plugin_dashboard.md b/docs/design_and_concept/how_to_write_plugin_dashboard.md
index fe7ab6b3f..4743e77c0 100644
--- a/docs/design_and_concept/how_to_write_plugin_dashboard.md
+++ b/docs/design_and_concept/how_to_write_plugin_dashboard.md
@@ -1,6 +1,6 @@
---
title: Dashboard Plugin
-sidebar_position: 4
+sidebar_position: 5
---
# Overview
diff --git a/docs/design_and_concept/how_to_write_plugin_manager.md b/docs/design_and_concept/how_to_write_plugin_manager.md
index 9cbf48c66..8f2e1da6c 100644
--- a/docs/design_and_concept/how_to_write_plugin_manager.md
+++ b/docs/design_and_concept/how_to_write_plugin_manager.md
@@ -1,6 +1,6 @@
---
title: Manager Plugin
-sidebar_position: 2
+sidebar_position: 4
---
# Overview
diff --git a/docs/design_and_concept/how_to_write_plugin_sort.md b/docs/design_and_concept/how_to_write_plugin_sort.md
index 7f7e9a7bf..ca589372a 100644
--- a/docs/design_and_concept/how_to_write_plugin_sort.md
+++ b/docs/design_and_concept/how_to_write_plugin_sort.md
@@ -1,18 +1,107 @@
---
-title: Basic Concept
-sidebar_position: 1
+title: Sort Plugin
+sidebar_position: 3
---
-| Name | Description | Other |
-|------------|----------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|
-| Group | Data Streams Group, it contains multiple data streams, and one Group represents one data ingestion. | Group has attributes such as ID and Name. |
-| Stream | Data Stream, a stream has a specific flow direction. | Stream has attributes such as ID, Name, and data fields. |
-| Node | Data Node, including `Extract Node` and `Load Node`, stands for the data source and sink types separately. | |
-| InLongMsg | InLong data format, if you consume message directly from the message queue, you need to perform `InLongMsg` parsing first. | |
-| Agent | Represents various collection capabilities. | It contains File Agent, SQL Agent, Binlog Agent, etc. |
-| DataProxy | Forward received data to different message queues. | Supports data transmission blocking, placing retransmission. |
-| Sort | Data stream sorting | Sort-flink based on Flink, sort-standalone for local sorting. |
-| TubeMQ | InLong's self-developed message queuing service | It can also be called Tube, with low-cost, high-performance features. |
-| Pulsar | [Apache Pulsar](https://pulsar.apache.org/), a high-performance, high-consistency message queue service | |
-| Hive | [Apache Hive](https://hive.apache.org/), a data warehouse built on the Hadoop architecture | |
-| ClickHouse | [ClickHouse](https://clickhouse.com/), a high performance columnar OLAP database | |
\ No newline at end of file
+# Overview
+InLong-Sort is known as a real-time ETL system. Currently, supported sinks are hive, kafka, clickhouse and iceberg.
+This article introduces how to extend a new type of sink in InLong-Sort.
+
+# Extend a new sink function
+InLong-Sort is based on flink, when extending a new type of sink in InLong-Sort, either a new type of flink sink
+or a predefined sink in flink is required.
+refer to [DataStream Connectors ](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/overview/#datastream-connectors).
+
+# Extend a new sink protocol
+Firstly, implement a new sink protocol which extends
+`inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/sink/SinkInfo.java`
+
+All necessary attributes used by the corresponding flink sink which extended before should be placed in the protocol
+
+Example
+```java
+public class DemoSinkInfo extends SinkInfo {
+
+ // Place necessary attributes here
+
+ @JsonCreator
+ public DemoSinkInfo(FieldInfo[] fields) {
+ super(fields);
+ }
+}
+```
+
+Secondly, mark the new sink protocol as subtype of SinkInfo and give it a name
+
+Example : A new sink protocol DemoSinkInfo whose subtype name is `Constants.SINK_TYPE_DEMO`
+```java
+/**
+ * The base class of the data sink in the metadata.
+ */
+@JsonTypeInfo(
+ use = JsonTypeInfo.Id.NAME,
+ include = JsonTypeInfo.As.PROPERTY,
+ property = "type")
+@JsonSubTypes({
+ @Type(value = ClickHouseSinkInfo.class, name = Constants.SINK_TYPE_CLICKHOUSE),
+ @Type(value = HiveSinkInfo.class, name = Constants.SINK_TYPE_HIVE),
+ @Type(value = KafkaSinkInfo.class, name = Constants.SINK_TYPE_KAFKA),
+ @Type(value = IcebergSinkInfo.class, name = Constants.SINK_TYPE_ICEBERG),
+
+ // The new sink protocol
+ @Type(value = DemoSinkInfo.class, name = Constants.SINK_TYPE_DEMO)}
+)
+public abstract class SinkInfo implements Serializable {
+
+ private static final long serialVersionUID = 1485856855405721745L;
+
+ @JsonProperty("fields")
+ private final FieldInfo[] fields;
+
+ public SinkInfo(@JsonProperty("fields") FieldInfo[] fields) {
+ this.fields = checkNotNull(fields);
+ }
+
+ @JsonProperty("fields")
+ public FieldInfo[] getFields() {
+ return fields;
+ }
+}
+```
+
+# Bundle the new sink, make it an alternative sink in Inlong_sort
+After extending a new flink sink and a new sink protocol in InLong-Sort, we can bundle the new sink into the flink job
+(the entrance of InLong-Sort)
+
+The entrance of the flink job is `inlong-sort/sort-single-tenant/src/main/java/org/apache/inlong/sort/singletenant/flink/Entrance.java`
+
+Example
+
+``` java
+private static void buildSinkStream(
+ DataStream<Row> sourceStream,
+ Configuration config,
+ SinkInfo sinkInfo,
+ Map<String, Object> properties,
+ long dataflowId) throws IOException, ClassNotFoundException {
+ final String sinkType = checkNotNull(config.getString(Constants.SINK_TYPE));
+
+ switch (sinkType) {
+ case Constants.SINK_TYPE_CLICKHOUSE:
+ break;
+ case Constants.SINK_TYPE_HIVE:
+ break;
+ case Constants.SINK_TYPE_ICEBERG:
+ break;
+ case Constants.SINK_TYPE_KAFKA:
+ break;
+ case Constants.SINK_TYPE_DEMO:
+ // Add the extended sink function here
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported sink type " + sinkType);
+ }
+
+}
+
+```
\ No newline at end of file
diff --git a/docs/user_guide/dashboard_usage.md b/docs/user_guide/dashboard_usage.md
index 02e087ea7..935b484e8 100644
--- a/docs/user_guide/dashboard_usage.md
+++ b/docs/user_guide/dashboard_usage.md
@@ -1,5 +1,5 @@
---
-title: Dashboard Usage Guide
+title: Dashboard Guide
sidebar_position: 1
---
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_agent.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_agent.md
index 27c5a6836..22f8e5d39 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_agent.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_agent.md
@@ -1,6 +1,6 @@
---
title: Agent 插件
-sidebar_position: 1
+sidebar_position: 2
---
# 总览
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
index 34e023ca8..2a2b0514c 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
@@ -1,6 +1,6 @@
---
title: Dashboard 插件
-sidebar_position: 4
+sidebar_position: 5
---
# 总览
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_manager.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_manager.md
index f790d6d65..e2f4f738a 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_manager.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_manager.md
@@ -1,6 +1,6 @@
---
title: Manager 插件
-sidebar_position: 2
+sidebar_position: 4
---
# 总览
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/user_guide/dashboard_usage.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/user_guide/dashboard_usage.md
index 25f509c63..49a90127f 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/user_guide/dashboard_usage.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/user_guide/dashboard_usage.md
@@ -1,5 +1,5 @@
---
-title: Dashboard 使用指引
+title: Dashboard 指引
sidebar_position: 1
---