You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by le...@apache.org on 2020/04/12 12:19:41 UTC
[incubator-hudi] branch asf-site updated: [HUDI-782] Add doc for
OSS filesystem (#1508)
This is an automated email from the ASF dual-hosted git repository.
leesf pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new fbb83fe [HUDI-782] Add doc for OSS filesystem (#1508)
fbb83fe is described below
commit fbb83fe36e830d6d6a6fc5b9836be43f30995114
Author: Shen Hong <sh...@126.com>
AuthorDate: Sun Apr 12 20:19:33 2020 +0800
[HUDI-782] Add doc for OSS filesystem (#1508)
---
docs/_docs/0_5_oss_filesystem.cn.md | 62 +++++++++++++++++++++++++++++++++++++
docs/_docs/0_5_oss_filesystem.md | 61 ++++++++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
diff --git a/docs/_docs/0_5_oss_filesystem.cn.md b/docs/_docs/0_5_oss_filesystem.cn.md
new file mode 100644
index 0000000..5083b1d
--- /dev/null
+++ b/docs/_docs/0_5_oss_filesystem.cn.md
@@ -0,0 +1,62 @@
+---
+title: OSS Filesystem
+keywords: hudi, hive, aliyun, oss, spark, presto
+permalink: /docs/oss_hoodie.html
+summary: In this page, we go over how to configure Hudi with OSS filesystem.
+last_modified_at: 2020-04-12T10:50:50-10:00
+language: cn
+---
+这个页面描述了如何让你的Hudi spark任务使用Aliyun OSS存储。
+
+## Aliyun OSS 部署
+
+为了让Hudi使用OSS，需要增加两部分的配置:
+
+- 为Hidi增加Aliyun OSS的相关配置
+- 增加Jar包的MVN依赖
+
+### Aliyun OSS 相关的配置
+
+新增下面的配置到你的Hudi能访问的core-site.xml文件。使用你的OSS bucket name替换掉`fs.defaultFS`，使用OSS endpoint地址替换`fs.oss.endpoint`，使用OSS的key和secret分别替换`fs.oss.accessKeyId`和`fs.oss.accessKeySecret`。主要Hudi就能读写相应的bucket。
+
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>oss://bucketname/</value>
+ </property>
+
+ <property>
+ <name>fs.oss.endpoint</name>
+ <value>oss-endpoint-address</value>
+ <description>Aliyun OSS endpoint to connect to.</description>
+ </property>
+
+ <property>
+ <name>fs.oss.accessKeyId</name>
+ <value>oss_key</value>
+ <description>Aliyun access key ID</description>
+ </property>
+
+ <property>
+ <name>fs.oss.accessKeySecret</name>
+ <value>oss-secret</value>
+ <description>Aliyun access key secret</description>
+ </property>
+
+ <property>
+ <name>fs.oss.impl</name>
+ <value>org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem</value>
+ </property>
+```
+
+### Aliyun OSS Libs
+
+新增Aliyun hadoop的jar包的MVN依赖到pom.xml文件。
+
+```xml
+<dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-aliyun</artifactId>
+ <version>3.2.1</version>
+</dependency>
+```
diff --git a/docs/_docs/0_5_oss_filesystem.md b/docs/_docs/0_5_oss_filesystem.md
new file mode 100644
index 0000000..5ecd155
--- /dev/null
+++ b/docs/_docs/0_5_oss_filesystem.md
@@ -0,0 +1,61 @@
+---
+title: OSS Filesystem
+keywords: hudi, hive, aliyun, oss, spark, presto
+permalink: /docs/oss_hoodie.html
+summary: In this page, we go over how to configure Hudi with OSS filesystem.
+last_modified_at: 2020-04-12T11:23:24-10:00
+---
+In this page, we explain how to get your Hudi spark job to store into Aliyun OSS.
+
+## Aliyun OSS configs
+
+There are two configurations required for Hudi-OSS compatibility:
+
+- Adding Aliyun OSS Credentials for Hudi
+- Adding required Jars to classpath
+
+### Aliyun OSS Credentials
+
+Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the `fs.defaultFS` with your OSS bucket name, replace `fs.oss.endpoint` with your OSS endpoint, replace `fs.oss.accessKeyId` with your OSS key, replace `fs.oss.accessKeySecret` with your OSS secret. Hudi should be able to read/write from the bucket.
+
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>oss://bucketname/</value>
+ </property>
+
+ <property>
+ <name>fs.oss.endpoint</name>
+ <value>oss-endpoint-address</value>
+ <description>Aliyun OSS endpoint to connect to.</description>
+ </property>
+
+ <property>
+ <name>fs.oss.accessKeyId</name>
+ <value>oss_key</value>
+ <description>Aliyun access key ID</description>
+ </property>
+
+ <property>
+ <name>fs.oss.accessKeySecret</name>
+ <value>oss-secret</value>
+ <description>Aliyun access key secret</description>
+ </property>
+
+ <property>
+ <name>fs.oss.impl</name>
+ <value>org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem</value>
+ </property>
+```
+
+### Aliyun OSS Libs
+
+Aliyun hadoop libraries jars to add to our pom.xml.
+
+```xml
+<dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-aliyun</artifactId>
+ <version>3.2.1</version>
+</dependency>
+```