You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by go...@apache.org on 2021/12/09 04:32:32 UTC
[incubator-inlong-website] branch master updated: [INLONG-1939] add basic concepts for InLong (#221)
This is an automated email from the ASF dual-hosted git repository.
gosonzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 74e006e [INLONG-1939] add basic concepts for InLong (#221)
74e006e is described below
commit 74e006e65ff172c0813eefed3c726b61635ff143
Author: dockerzhang <do...@apache.org>
AuthorDate: Thu Dec 9 12:32:26 2021 +0800
[INLONG-1939] add basic concepts for InLong (#221)
Co-authored-by: dockerzhang <do...@tencent.com>
---
docs/design_and_concept/basic_concept.md | 13 +++++++++++--
docs/modules/agent/{file_collect.md => file.md} | 4 ++--
docs/modules/agent/overview.md | 2 +-
docs/quick_start/hive_example.md | 8 ++++----
.../current/design_and_concept/basic_concept.md | 12 +++++++++++-
.../current/modules/agent/{file_collect.md => file.md} | 4 ++--
.../current/modules/agent/overview.md | 2 +-
.../current/quick_start/hive_example.md | 8 ++++----
static/img/inlong_architecture.png | Bin 117953 -> 0 bytes
9 files changed, 36 insertions(+), 17 deletions(-)
diff --git a/docs/design_and_concept/basic_concept.md b/docs/design_and_concept/basic_concept.md
index 1fd2f02..871c9c1 100644
--- a/docs/design_and_concept/basic_concept.md
+++ b/docs/design_and_concept/basic_concept.md
@@ -3,5 +3,14 @@ title: Basic Concept
sidebar_position: 1
---
-Will be added soon
-
+| Name | Description | Other |
+| ---- | ---- | ---- |
+| Group | Data Streams Group, it contains multiple data streams, and one Group represents one data access. | Group has attributes such as ID and Name. |
+| Stream | Data Stream, a stream has a specific flow direction. | Stream has attributes such as ID, Name, and data fields. |
+| Agent | Represents various collection capabilities. | It contains File Agent, SQL Agent, Binlog Agent, etc. |
+| DataProxy | Forward received data to different message queues. | Supports data transmission blocking, placing retransmission. |
+| Sort | Data stream sorting | Sort-flink based on Flink, sort-standalone for local sorting. |
+| TubeMQ | InLong's self-developed message queuing service | It can also be called Tube, with low-cost, high-performance features. |
+| Pulsar | [Apache Pulsar](https://pulsar.apache.org/), a high-performance, high-consistency message queue service |
+| Hive | [Apache Hive](https://hive.apache.org/), a data warehouse built on the Hadoop architecture |
+| ClickHouse | [ClickHouse](https://clickhouse.com/), a high performance columnar OLAP database | |
\ No newline at end of file
diff --git a/docs/modules/agent/file_collect.md b/docs/modules/agent/file.md
similarity index 98%
rename from docs/modules/agent/file_collect.md
rename to docs/modules/agent/file.md
index 36f5591..219e1df 100644
--- a/docs/modules/agent/file_collect.md
+++ b/docs/modules/agent/file.md
@@ -1,9 +1,9 @@
---
-title: File Collect
+title: File
sidebar_position: 3
---
-## File Collect Configuration
+## File Agent Configuration
```
/data/inlong-agent/test.log //Represents reading the new file test.log in the inlong-agent folder
/data/inlong-agent/test[0-9]{1} // means to read the new file test in the inlong-agent folder followed by a number at the end
diff --git a/docs/modules/agent/overview.md b/docs/modules/agent/overview.md
index 1f071e2..7e1ab15 100644
--- a/docs/modules/agent/overview.md
+++ b/docs/modules/agent/overview.md
@@ -3,7 +3,7 @@ title: Overview
sidebar_position: 1
---
-InLong-Agent is a collection tool that supports multiple types of data sources, and is committed to achieving stable and efficient data collection functions between multiple heterogeneous data sources including file, sql, Binlog, metrics, etc.
+InLong-Agent is a collection tool that supports multiple types of data sources, and is committed to achieving stable and efficient data collection functions between multiple heterogeneous data sources including File, SQL, Binlog, Metrics, etc.
## Design Concept
In order to solve the problem of data source diversity, InLong-agent abstracts multiple data sources into a unified source concept, and abstracts sinks to write data. When you need to access a new data source, you only need to configure the format and reading parameters of the data source to achieve efficient reading.
diff --git a/docs/quick_start/hive_example.md b/docs/quick_start/hive_example.md
index b75cfca..7be6dbc 100644
--- a/docs/quick_start/hive_example.md
+++ b/docs/quick_start/hive_example.md
@@ -18,21 +18,21 @@ Before we begin, we need to install InLong. Here we provide two ways:
## 3 Create a data access
After deployment, we first enter the "Data Access" interface, click "Create an Access" in the upper right corner to create a new date access, and fill in the data streams group information as shown in the figure below.
-<img src="img/create-group.png" align="center" alt="Create Group"/>
+
Then we click the next button, and fill in the stream information as shown in the figure below.
-<img src="img/create-stream.png" align="center" alt="Create Stream"/>
+
Note that the message source is "File", and we don't need to create a message source manually.
Then we fill in the following information in the "data information" column below.
-<img src="img/data-information.png" align="center" alt="Data Information"/>
+
Then we select Hive in the data flow and click "Add" to add Hive configuration
-<img src="img/hive-config.png" align="center" alt="Hive Config"/>
+
Note that the target table does not need to be created in advance, as InLong Manager will automatically create the table for us after the access is approved. Also, please use connection test to ensure that InLong Manager can connect to your Hive.
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/basic_concept.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/basic_concept.md
index dab4399..f63b643 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/basic_concept.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/basic_concept.md
@@ -3,4 +3,14 @@ title: 基本概念
sidebar_position: 1
---
-Will be added soon
+| Name | Description | Other |
+| ---- | ---- | ---- |
+| Group | 数据流组，包含多个数据流，一个Group 代表一个数据接入 | Group 有ID、Name 等属性 |
+| Stream | 数据流，一个数据流有具体的流向 | Stream 有ID、Name、数据字段等属性 |
+| Agent | 代表各种采集能力 | 包含文件Agent、SQL Agent、Binlog Agent 等 |
+| DataProxy | 将接收到的数据转发到不同的消息队列 | 支持数据发送阻塞和落盘重发 |
+| Sort | 数据流分拣 | 主要有基于Flink的sort-flink，sort-standalone 本地分拣 |
+| TubeMQ | InLong自带的消息队列服务 | 也可以叫Tube，拥有低成本、高性能特性 |
+| Pulsar | 即[Apache Pulsar](https://pulsar.apache.org/), 高性能、高一致性消息队列服务 |
+| Hive | 即[Apache Hive](https://hive.apache.org/)，一个建立在Hadoop架构之上的数据仓库 |
+| ClickHouse | [ClickHouse](https://clickhouse.com/)，高性能列式OLAP 数据库 | |
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/file_collect.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/file.md
similarity index 98%
rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/file_collect.md
rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/file.md
index 49d8e7a..740fbab 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/file_collect.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/file.md
@@ -1,9 +1,9 @@
---
-title: 文件采集
+title: 文件
sidebar_position: 3
---
-## 文件采集配置
+## 文件Agent配置
```
/data/inlong-agent/test.log //代表读取inlong-agent文件夹下的的新增文件test.log
/data/inlong-agent/test[0-9]{1} //代表读取inlong-agent文件夹下的新增文件test后接一个数字结尾
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/overview.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/overview.md
index ec0fc14..609a63f 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/overview.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/overview.md
@@ -3,7 +3,7 @@ title: 总览
sidebar_position: 1
---
-InLong-Agent是一个支持多种数据源类型的收集工具，致力于实现包括file、sql、Binlog、metrics等多种异构数据源之间稳定高效的数据采集功能。
+InLong-Agent是一个支持多种数据源类型的收集工具，致力于实现包括File、Sql、Binlog、Metrics等多种异构数据源之间稳定高效的数据采集功能。
## 设计理念
为了解决数据源多样性问题，InLong-agent 将多种数据源抽象成统一的source概念，并抽象出sink来对数据进行写入。当需要接入一个新的数据源的时候，只需要配置好数据源的格式与读取参数便能跟做到高效读取。
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/hive_example.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/hive_example.md
index 572634b..f3f8c93 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/hive_example.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/hive_example.md
@@ -20,21 +20,21 @@ Hive 是运行的必备组件。如果您的机器上没有 Hive，这里推荐
## 3 新建接入
部署完毕后，首先我们进入 “数据接入” 界面，点击右上角的 “新建接入”，新建一条接入，按下图所示填入数据流 Group 信息
-<img src="img/create-group.png" align="center" alt="Create Group"/>
+
然后点击下一步，按下图所示填入数据流信息
-<img src="img/create-stream.png" align="center" alt="Create Stream"/>
+
注意其中消息来源选择“文件”，暂时不用新建数据源。
然后我们在下面的“数据信息”一栏中填入以下信息
-<img src="img/data-information.png" align="center" alt="Data Information"/>
+
然后在数据流向中选择 Hive，并点击 “添加”，添加 Hive 配置
-<img src="img/hive-config.png" align="center" alt="Hive Config"/>
+
注意这里目标表无需提前创建，InLong Manager 会在接入通过之后自动为我们创建表。另外，请使用 “连接测试” 保证 InLong Manager 可以连接到你的 Hive。
diff --git a/static/img/inlong_architecture.png b/static/img/inlong_architecture.png
deleted file mode 100644
index 644939a..0000000
Binary files a/static/img/inlong_architecture.png and /dev/null differ