You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "haifxu (via GitHub)" <gi...@apache.org> on 2023/09/12 07:09:48 UTC
[GitHub] [inlong-website] haifxu commented on a diff in pull request #850: [INLONG-849][Doc] Add Iceberg extract node documentation
haifxu commented on code in PR #850:
URL: https://github.com/apache/inlong-website/pull/850#discussion_r1322549304
##########
docs/data_node/extract_node/iceberg.md:
##########
@@ -0,0 +1,158 @@
+---
+title: Iceberg
+sidebar_position: 14
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+[Apache Iceberg](https://iceberg.apache.org/) is a high-performance format for huge analytic tables.
+
+## Version
+
+| Extract Node | Version |
+|-------------------------|--------------------------------------------------------------|
+| [Iceberg](./iceberg.md) | [Iceberg](https://iceberg.apache.org/): 0.12.x, 0.13.x <br/> |
+
+## Dependencies
+
+<pre><code parentName="pre">
+{`<dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>sort-connector-iceberg</artifactId>
+ <version>${siteVariables.inLongVersion}</version>
+</dependency>
+`}
+</code></pre>
+
+## Usage
+
+Before creating the Iceberg task, we need a Flink environment integrated with Hadoop
+
+1. Download `Hadoop` on the host.
Review Comment:
Done.
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/iceberg.md:
##########
@@ -0,0 +1,153 @@
+---
+title: Iceberg
+sidebar_position: 14
+---
+
+import {siteVariables} from '../../version';
+
+## 概览
+
+[Apache Iceberg](https://iceberg.apache.org/) 是一种用于大型分析表的高性能格式。
+
+## 版本
+
+| 提取节点 | 版本 |
+|-------------------------|------------------------------------------------------|
+| [Iceberg](./iceberg.md) | [Iceberg](https://iceberg.apache.org/)：0.12.x，0.13.x |
+
+## 依赖项
+
+<pre><code parentName="pre">
+{`<dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>sort-connector-iceberg</artifactId>
+ <version>${siteVariables.inLongVersion}</version>
+</dependency>
+`}
+</code></pre>
+
+## 如何配置 Iceberg 数据抽取节点
+
+在创建 Iceberg 任务之前，我们需要一个集成 Hadoop 的 Flink 环境
+
+1. 在宿主机中下载 `Hadoop`
+2. 在宿主机中修改 `jobmanager.sh` 和 `taskmanager.sh`，加入 `Hadoop` 环境变量。启动命令可以参考 [Apache Flink](https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/bin)
+
+```shell
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+```
+
+3. 修改 `docker/docker-compose` 目录下的 `docker-compose.yml`，将 `Hadoop` 和 `Flink 启动命令` 挂载至容器中
+
+```shell
+# flink jobmanager
+ jobmanager:
+ image: apache/flink:1.13-scala_2.11
+ container_name: jobmanager
+ user: root
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - HADOOP_HOME:HADOOP_HOME
+ - /jobmanager.sh:/opt/flink/bin/jobmanager.sh
+ ports:
+ - "8081:8081"
+ command: jobmanager
+
+ # flink taskmanager
+ taskmanager:
+ image: apache/flink:1.13-scala_2.11
+ container_name: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 2
+ volumes:
+ - HADOOP_HOME:HADOOP_HOME
+ - /taskmanager.sh:/opt/flink/bin/taskmanager.sh
+ command: taskmanager
+```
+
+### SQL API 的使用
+
+使用 Flink sql client 之前，`sql-client.sh` 启动命令也需要添加 Hadoop 环境变量，并挂载至容器。
+启动命令可以参考 [Apache Flink](https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/bin/sql-client.sh)
+
+```shell
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+```
+
+使用 `Flink sql cli`：
+
+```
+CREATE TABLE `iceberg_table_source`(
+ PRIMARY KEY (`_id`) NOT ENFORCED,
+ `_id` STRING,
+ `id` INT,
+ `name` STRING,
+ `age` INT)
+ WITH (
+ 'metrics.audit.key' = '13&14',
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org