You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2024/02/29 08:45:01 UTC
(doris) 04/13: [fix](paimon) auto deplay paimon oss/s3 jar file (#31568)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 450422556ad05c68a05f3ba8631eae1550b91ff6
Author: Mingyu Chen <mo...@163.com>
AuthorDate: Thu Feb 29 15:12:23 2024 +0800
[fix](paimon) auto deplay paimon oss/s3 jar file (#31568)
No need to deploy paimon oss/s3 jar files manually.
Include them in preload-extensions-jar-with-dependencies.jar
---
docs/en/docs/lakehouse/multi-catalog/paimon.md | 38 +++++------------------
docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md | 38 +++++------------------
fe/be-java-extensions/preload-extensions/pom.xml | 12 +++++++
3 files changed, 27 insertions(+), 61 deletions(-)
diff --git a/docs/en/docs/lakehouse/multi-catalog/paimon.md b/docs/en/docs/lakehouse/multi-catalog/paimon.md
index 49012cf7318..63ec1f0a3f6 100644
--- a/docs/en/docs/lakehouse/multi-catalog/paimon.md
+++ b/docs/en/docs/lakehouse/multi-catalog/paimon.md
@@ -75,14 +75,6 @@ CREATE CATALOG `paimon_kerberos` PROPERTIES (
#### MINIO
-> Note that.
->
-> user need download [paimon-s3-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.6.0-incubating/paimon-s3-0.6.0-incubating.jar)
->
-> Place it in directory `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` and restart be
->
-> Starting from version 2.0.2, this file can be placed in BE's `custom_lib/` directory (if it does not exist, just create it manually) to prevent the file from being lost due to the replacement of the lib directory when upgrading the cluster.
-
```sql
CREATE CATALOG `paimon_s3` PROPERTIES (
"type" = "paimon",
@@ -95,14 +87,6 @@ CREATE CATALOG `paimon_s3` PROPERTIES (
#### OBS
-> Note that.
->
-> user need download [paimon-s3-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.6.0-incubating/paimon-s3-0.6.0-incubating.jar)
->
-> Place it in directory `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` and restart be
->
-> Starting from version 2.0.2, this file can be placed in BE's `custom_lib/` directory (if it does not exist, just create it manually) to prevent the file from being lost due to the replacement of the lib directory when upgrading the cluster.
-
```sql
CREATE CATALOG `paimon_obs` PROPERTIES (
"type" = "paimon",
@@ -115,14 +99,6 @@ CREATE CATALOG `paimon_obs` PROPERTIES (
#### COS
-> Note that.
->
-> user need download [paimon-s3-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.6.0-incubating/paimon-s3-0.6.0-incubating.jar)
->
-> Place it in directory `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` and restart be
->
-> Starting from version 2.0.2, this file can be placed in BE's `custom_lib/` directory (if it does not exist, just create it manually) to prevent the file from being lost due to the replacement of the lib directory when upgrading the cluster.
-
```sql
CREATE CATALOG `paimon_cos` PROPERTIES (
"type" = "paimon",
@@ -135,13 +111,6 @@ CREATE CATALOG `paimon_cos` PROPERTIES (
#### OSS
->Note that.
->
-> user need download [paimon-oss-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-oss/0.6.0-incubating/paimon-oss-0.6.0-incubating.jar)
->
-> Place it in directory `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` and restart be
-
-
```sql
CREATE CATALOG `paimon_oss` PROPERTIES (
"type" = "paimon",
@@ -217,3 +186,10 @@ CREATE CATALOG `paimon_kerberos` PROPERTIES (
This is a compatible issue exist in 2.0.2 with Paimon 0.5, you need to upgrade to 2.0.3 or higher to solve this problem. Or [patch](https://github.com/apache/doris/pull/24985) yourself.
+3. When accessing object storage (OSS, S3, etc.), encounter "file system does not support".
+
+ In versions before 2.0.5 (inclusive), users need to manually download the following jar package and place it in the `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` directory, and restart BE.
+
+ - OSS: [paimon-oss-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-oss/0.6.0-incubating/paimon-oss-0.6.0-incubating.jar)
+ - Other Object Storage: [paimon-s3-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.6.0-incubating/paimon-s3-0.6.0-incubating.jar)
+
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md b/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
index d196df7a87e..00f36bacebe 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
@@ -75,14 +75,6 @@ CREATE CATALOG `paimon_kerberos` PROPERTIES (
#### MINIO
-> 注意：
->
-> 用户需要手动下载[paimon-s3-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.6.0-incubating/paimon-s3-0.6.0-incubating.jar)
-
-> 放在 `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` 目录下并重启be。
->
-> 从 2.0.2 版本起，可以将这个文件放置在BE的 `custom_lib/` 目录下（如不存在，手动创建即可），以防止升级集群时因为 lib 目录被替换而导致文件丢失。
-
```sql
CREATE CATALOG `paimon_s3` PROPERTIES (
"type" = "paimon",
@@ -95,14 +87,6 @@ CREATE CATALOG `paimon_s3` PROPERTIES (
#### OBS
-> 注意：
->
-> 用户需要手动下载[paimon-s3-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.6.0-incubating/paimon-s3-0.6.0-incubating.jar)
-
-> 放在 `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` 目录下并重启be。
->
-> 从 2.0.2 版本起，可以将这个文件放置在BE的 `custom_lib/` 目录下（如不存在，手动创建即可），以防止升级集群时因为 lib 目录被替换而导致文件丢失。
-
```sql
CREATE CATALOG `paimon_obs` PROPERTIES (
"type" = "paimon",
@@ -115,14 +99,6 @@ CREATE CATALOG `paimon_obs` PROPERTIES (
#### COS
-> 注意：
->
-> 用户需要手动下载[paimon-s3-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.6.0-incubating/paimon-s3-0.6.0-incubating.jar)
-
-> 放在 `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` 目录下并重启be。
->
-> 从 2.0.2 版本起，可以将这个文件放置在BE的 `custom_lib/` 目录下（如不存在，手动创建即可），以防止升级集群时因为 lib 目录被替换而导致文件丢失。
-
```sql
CREATE CATALOG `paimon_s3` PROPERTIES (
"type" = "paimon",
@@ -135,11 +111,6 @@ CREATE CATALOG `paimon_s3` PROPERTIES (
#### OSS
->注意：
->
-> 用户需要手动下载[paimon-oss-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-oss/0.6.0-incubating/paimon-oss-0.6.0-incubating.jar)
-> 放在 `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` 目录下并重启be
-
```sql
CREATE CATALOG `paimon_oss` PROPERTIES (
"type" = "paimon",
@@ -185,7 +156,6 @@ CREATE CATALOG `paimon_kerberos` PROPERTIES (
);
```
-
## 列类型映射
| Paimon Data Type | Doris Data Type | Comment |
@@ -217,3 +187,11 @@ CREATE CATALOG `paimon_kerberos` PROPERTIES (
这是 Doris 2.0.2 版本和 Paimon 0.5 版本的一个兼容性问题，需要升级到 2.0.3 或更高版本解决，或自行 [patch](https://github.com/apache/doris/pull/24985)
+3. 访问对象存储（OSS、S3 等）报错文件系统不支持
+
+ 在 2.0.5（含）之前的版本，用户需手动下载以下 jar 包并放置在 `${DORIS_HOME}/be/lib/java_extensions/preload-extensions` 目录下，重启 BE。
+
+ - 访问 OSS：[paimon-oss-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-oss/0.6.0-incubating/paimon-oss-0.6.0-incubating.jar)
+ - 访问其他对象存储：[paimon-s3-0.6.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.6.0-incubating/paimon-s3-0.6.0-incubating.jar)
+
+
diff --git a/fe/be-java-extensions/preload-extensions/pom.xml b/fe/be-java-extensions/preload-extensions/pom.xml
index 8c6f57f746c..6732fbc75a5 100644
--- a/fe/be-java-extensions/preload-extensions/pom.xml
+++ b/fe/be-java-extensions/preload-extensions/pom.xml
@@ -228,6 +228,18 @@ under the License.
<artifactId>hadoop-cos</artifactId>
<version>3.3.5</version>
</dependency>
+ <!-- For BE Paimon OSS/S3 Access -->
+ <dependency>
+ <groupId>org.apache.paimon</groupId>
+ <artifactId>paimon-s3</artifactId>
+ <version>${paimon.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.paimon</groupId>
+ <artifactId>paimon-oss</artifactId>
+ <version>${paimon.version}</version>
+ </dependency>
</dependencies>
<build>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org