You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by xu...@apache.org on 2021/08/11 19:48:58 UTC

[hudi] branch asf-site updated: [HUDI-2289] Add ks3 support doc for hudi (#3436)

This is an automated email from the ASF dual-hosted git repository.

xushiyan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 9e14388  [HUDI-2289] Add ks3 support doc for hudi (#3436)
9e14388 is described below

commit 9e143884644f6d04321ac7d80fcd716e676fb3ef
Author: xuzifu666 <12...@qq.com>
AuthorDate: Thu Aug 12 03:48:43 2021 +0800

    [HUDI-2289] Add ks3 support doc for hudi (#3436)
    
    * add ks3 support doc for hudi
    
    * add md file for ks3 storage on hudi
    
    Co-authored-by: xuzifu <xuzifu.com>
---
 docs/_docs/3_3_ks3_filesystem.md | 50 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/docs/_docs/3_3_ks3_filesystem.md b/docs/_docs/3_3_ks3_filesystem.md
new file mode 100644
index 0000000..1445636
--- /dev/null
+++ b/docs/_docs/3_3_ks3_filesystem.md
@@ -0,0 +1,50 @@
+---
+title: KS3 Filesystem
+keywords: hudi, hive, aws, s3, spark, presto, ks3
+permalink: /docs/ks3_hoodie.html
+summary: In this page, we go over how to configure Hudi with KS3 filesystem.
+last_modified_at: 2021-08-09T15:59:57-04:00
+---
+In this page, we explain how to get your Hudi spark job to store into KS3.
+
+## KS3 configs
+
+There are two configurations required for Hudi-KS3 compatibility:
+
+- Adding KS3 Credentials for Hudi
+- Adding required Jars to classpath
+
+### KS3 Credentials
+
+Simplest way to use Hudi with KS3, is to configure your `SparkSession` or `SparkContext` with KS3 credentials. Hudi will automatically pick this up and talk to KS3.
+
+Alternatively, add the required configs in your core-site.xml from where Hudi can fetch them. Replace the `fs.defaultFS` with your KS3 bucket name and Hudi should be able to read/write from the bucket.
+
+```xml
+  <property>
+      <name>fs.defaultFS</name>
+      <value>hdfs://ks3node</value>
+  </property>
+
+  <property>
+      <name>fs.ks3.impl</name>
+      <value>com.ksyun.kmr.hadoop.fs.Ks3FileSystem</value>
+  </property>
+
+  <property>
+      <name>fs.ks3.AccessKey</name>
+      <value>KS3_KEY</value>
+  </property>
+
+  <property>
+       <name>fs.ks3.AccessSecret</name>
+       <value>KS3_SECRET</value>
+  </property>
+
+```
+
+### KS3 Libs
+
+KS3 hadoop libraries to add to our classpath
+
+ - com.ksyun:ks3-kss-java-sdk:1.0.2