You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by co...@apache.org on 2024/12/13 12:00:52 UTC
(hudi) branch asf-site updated: [DOCS] Cut new docs version 1.0.0 (#12484)
This is an automated email from the ASF dual-hosted git repository.
codope pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ec3bedb6c4f [DOCS] Cut new docs version 1.0.0 (#12484)
ec3bedb6c4f is described below
commit ec3bedb6c4faa006a35d3c60fcd6c10bf22ed0c8
Author: Sagar Sumit <sa...@gmail.com>
AuthorDate: Fri Dec 13 17:30:45 2024 +0530
[DOCS] Cut new docs version 1.0.0 (#12484)
* [DOCS] Cut new docs version 1.0.0
* fix broken links
---
website/docusaurus.config.js | 10 +-
.../version-1.0.0/azure_hoodie.md | 52 +
.../version-1.0.0/bos_hoodie.md | 58 +
.../version-1.0.0/cloud.md | 28 +
.../version-1.0.0/comparison.md | 48 +
.../version-1.0.0/concepts.md | 154 ++
.../version-1.0.0/configurations.md | 604 ++++++
.../version-1.0.0/cos_hoodie.md | 73 +
.../version-1.0.0/deployment.md | 433 ++++
.../version-1.0.0/docker_demo.md | 1122 ++++++++++
.../version-1.0.0/docs-versions.md | 12 +
.../version-1.0.0/flink-quick-start-guide.md | 530 +++++
.../version-1.0.0/gcs_hoodie.md | 61 +
.../version-1.0.0/ibm_cos_hoodie.md | 78 +
.../version-1.0.0/migration_guide.md | 60 +
.../version-1.0.0/oss_hoodie.md | 71 +
.../version-1.0.0/overview.md | 155 ++
.../version-1.0.0/performance.md | 62 +
.../version-1.0.0/powered_by.md | 88 +
.../version-1.0.0/privacy.md | 23 +
.../version-1.0.0/querying_data.md | 224 ++
.../version-1.0.0/s3_hoodie.md | 81 +
.../version-1.0.0/spark_quick-start-guide.md | 449 ++++
.../version-1.0.0/use_cases.md | 67 +
.../version-1.0.0/writing_data.md | 222 ++
website/releases/download.md | 4 +
website/src/components/HomepageFeatures/index.js | 2 +-
website/src/components/HomepageHeader/index.js | 2 +-
.../versioned_docs/version-0.10.0/file_layouts.md | 4 +-
.../versioned_docs/version-0.10.0/file_sizing.md | 2 +-
.../version-0.10.0/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.10.0/overview.mdx | 4 +-
website/versioned_docs/version-0.10.0/use_cases.md | 4 +-
.../version-0.10.0/write_operations.md | 10 +-
.../versioned_docs/version-0.10.1/file_layouts.md | 2 +-
.../versioned_docs/version-0.10.1/file_sizing.md | 2 +-
.../version-0.10.1/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.10.1/overview.mdx | 4 +-
website/versioned_docs/version-0.10.1/use_cases.md | 6 +-
.../version-0.10.1/write_operations.md | 10 +-
.../versioned_docs/version-0.11.0/file_layouts.md | 2 +-
.../versioned_docs/version-0.11.0/file_sizing.md | 2 +-
.../version-0.11.0/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.11.0/overview.mdx | 4 +-
website/versioned_docs/version-0.11.0/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.11.0/use_cases.md | 4 +-
.../version-0.11.0/write_operations.md | 8 +-
.../versioned_docs/version-0.11.1/file_layouts.md | 2 +-
.../versioned_docs/version-0.11.1/file_sizing.md | 2 +-
.../version-0.11.1/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.11.1/overview.mdx | 4 +-
website/versioned_docs/version-0.11.1/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.11.1/use_cases.md | 4 +-
.../version-0.11.1/write_operations.md | 8 +-
.../versioned_docs/version-0.12.0/file_layouts.md | 2 +-
.../versioned_docs/version-0.12.0/file_sizing.md | 2 +-
.../version-0.12.0/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.12.0/overview.mdx | 4 +-
website/versioned_docs/version-0.12.0/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.12.0/use_cases.md | 4 +-
.../version-0.12.0/write_operations.md | 8 +-
.../versioned_docs/version-0.12.1/file_layouts.md | 2 +-
.../versioned_docs/version-0.12.1/file_sizing.md | 2 +-
.../version-0.12.1/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.12.1/overview.mdx | 4 +-
website/versioned_docs/version-0.12.1/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.12.1/use_cases.md | 4 +-
.../version-0.12.1/write_operations.md | 8 +-
.../versioned_docs/version-0.12.2/file_layouts.md | 2 +-
.../versioned_docs/version-0.12.2/file_sizing.md | 2 +-
.../version-0.12.2/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.12.2/overview.mdx | 4 +-
website/versioned_docs/version-0.12.2/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.12.2/use_cases.md | 4 +-
.../version-0.12.2/write_operations.md | 8 +-
.../versioned_docs/version-0.12.3/file_layouts.md | 2 +-
.../versioned_docs/version-0.12.3/file_sizing.md | 2 +-
.../version-0.12.3/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.12.3/overview.mdx | 4 +-
website/versioned_docs/version-0.12.3/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.12.3/use_cases.md | 4 +-
.../version-0.12.3/write_operations.md | 8 +-
.../versioned_docs/version-0.13.0/file_layouts.md | 2 +-
.../versioned_docs/version-0.13.0/file_sizing.md | 2 +-
.../version-0.13.0/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.13.0/overview.mdx | 4 +-
website/versioned_docs/version-0.13.0/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.13.0/use_cases.md | 4 +-
.../version-0.13.0/write_operations.md | 8 +-
.../versioned_docs/version-0.13.1/file_layouts.md | 2 +-
.../versioned_docs/version-0.13.1/file_sizing.md | 2 +-
.../version-0.13.1/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.13.1/overview.mdx | 4 +-
website/versioned_docs/version-0.13.1/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.13.1/use_cases.md | 4 +-
.../version-0.13.1/write_operations.md | 8 +-
.../versioned_docs/version-0.14.0/file_layouts.md | 2 +-
.../versioned_docs/version-0.14.0/file_sizing.md | 2 +-
.../version-0.14.0/flink-quick-start-guide.md | 2 +-
website/versioned_docs/version-0.14.0/overview.mdx | 4 +-
website/versioned_docs/version-0.14.0/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.14.0/use_cases.md | 4 +-
.../version-0.14.0/write_operations.md | 8 +-
website/versioned_docs/version-0.14.1/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.15.0/s3_hoodie.md | 2 +-
.../versioned_docs/version-1.0.0/azure_hoodie.md | 50 +
.../version-1.0.0/basic_configurations.md | 797 +++++++
website/versioned_docs/version-1.0.0/bos_hoodie.md | 57 +
website/versioned_docs/version-1.0.0/cleaning.md | 155 ++
website/versioned_docs/version-1.0.0/cli.md | 755 +++++++
website/versioned_docs/version-1.0.0/cloud.md | 38 +
website/versioned_docs/version-1.0.0/clustering.md | 346 +++
website/versioned_docs/version-1.0.0/compaction.md | 228 ++
website/versioned_docs/version-1.0.0/comparison.md | 56 +
website/versioned_docs/version-1.0.0/concepts.md | 172 ++
.../version-1.0.0/concurrency_control.md | 339 +++
.../versioned_docs/version-1.0.0/configurations.md | 2287 ++++++++++++++++++++
website/versioned_docs/version-1.0.0/cos_hoodie.md | 71 +
website/versioned_docs/version-1.0.0/deployment.md | 284 +++
.../version-1.0.0/disaster_recovery.md | 309 +++
.../versioned_docs/version-1.0.0/docker_demo.md | 1526 +++++++++++++
website/versioned_docs/version-1.0.0/encryption.md | 73 +
website/versioned_docs/version-1.0.0/faq.md | 15 +
.../version-1.0.0/faq_design_and_concepts.md | 61 +
.../versioned_docs/version-1.0.0/faq_general.md | 98 +
.../version-1.0.0/faq_integrations.md | 68 +
.../version-1.0.0/faq_reading_tables.md | 31 +
.../versioned_docs/version-1.0.0/faq_storage.md | 193 ++
.../version-1.0.0/faq_table_services.md | 55 +
.../version-1.0.0/faq_writing_tables.md | 194 ++
.../file_sizing.md | 7 +-
.../version-1.0.0/flink-quick-start-guide.md | 474 ++++
.../versioned_docs/version-1.0.0/flink_tuning.md | 117 +
.../versioned_docs/version-1.0.0/gcp_bigquery.md | 102 +
website/versioned_docs/version-1.0.0/gcs_hoodie.md | 60 +
.../version-1.0.0/hoodie_streaming_ingestion.md | 638 ++++++
website/versioned_docs/version-1.0.0/hudi_stack.md | 174 ++
.../versioned_docs/version-1.0.0/ibm_cos_hoodie.md | 77 +
website/versioned_docs/version-1.0.0/indexes.md | 226 ++
.../version-1.0.0/ingestion_flink.md | 179 ++
.../version-1.0.0/ingestion_kafka_connect.md | 51 +
website/versioned_docs/version-1.0.0/intro.md | 47 +
website/versioned_docs/version-1.0.0/jfs_hoodie.md | 96 +
.../versioned_docs/version-1.0.0/key_generation.md | 215 ++
website/versioned_docs/version-1.0.0/markers.md | 91 +
website/versioned_docs/version-1.0.0/metadata.md | 134 ++
.../version-1.0.0/metadata_indexing.md | 318 +++
website/versioned_docs/version-1.0.0/metrics.md | 232 ++
.../version-1.0.0/migration_guide.md | 118 +
website/versioned_docs/version-1.0.0/oci_hoodie.md | 80 +
website/versioned_docs/version-1.0.0/oss_hoodie.md | 70 +
.../{version-0.14.0 => version-1.0.0}/overview.mdx | 47 +-
.../versioned_docs/version-1.0.0/performance.md | 133 ++
.../platform_services_post_commit_callback.md | 58 +
.../version-1.0.0/precommit_validator.md | 101 +
website/versioned_docs/version-1.0.0/privacy.md | 22 +
website/versioned_docs/version-1.0.0/procedures.md | 2001 +++++++++++++++++
.../version-1.0.0/python-rust-quick-start-guide.md | 119 +
.../versioned_docs/version-1.0.0/querying_data.md | 99 +
.../version-1.0.0/quick-start-guide.md | 1315 +++++++++++
.../version-1.0.0/reading_tables_batch_reads.md | 35 +
.../reading_tables_streaming_reads.md | 99 +
.../versioned_docs/version-1.0.0/record_merger.md | 253 +++
website/versioned_docs/version-1.0.0/rollbacks.md | 72 +
.../{version-0.14.0 => version-1.0.0}/s3_hoodie.md | 2 +-
.../version-1.0.0/schema_evolution.md | 319 +++
.../version-1.0.0/snapshot_exporter.md | 135 ++
website/versioned_docs/version-1.0.0/sql_ddl.md | 969 +++++++++
website/versioned_docs/version-1.0.0/sql_dml.md | 539 +++++
.../versioned_docs/version-1.0.0/sql_queries.md | 714 ++++++
.../version-1.0.0/storage_layouts.md | 69 +
website/versioned_docs/version-1.0.0/structure.md | 20 +
.../version-1.0.0/syncing_aws_glue_data_catalog.md | 72 +
.../version-1.0.0/syncing_datahub.md | 50 +
.../version-1.0.0/syncing_metastore.md | 299 +++
.../versioned_docs/version-1.0.0/syncing_xtable.md | 54 +
.../versioned_docs/version-1.0.0/table_types.md | 207 ++
website/versioned_docs/version-1.0.0/timeline.md | 153 ++
.../version-1.0.0/troubleshooting.md | 222 ++
.../versioned_docs/version-1.0.0/tuning-guide.md | 106 +
website/versioned_docs/version-1.0.0/use_cases.md | 103 +
.../write_operations.md | 96 +-
.../versioned_docs/version-1.0.0/writing_data.md | 444 ++++
.../writing_tables_streaming_writes.md | 95 +
.../versioned_sidebars/version-1.0.0-sidebars.json | 163 ++
website/versions.json | 1 +
186 files changed, 25799 insertions(+), 196 deletions(-)
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index 874a8fb1f80..1d5c45afc02 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -140,11 +140,11 @@ module.exports = {
},
{
from: ["/docs/releases", "/docs/next/releases"],
- to: "/releases/release-0.15.0",
+ to: "/releases/release-1.0.0",
},
{
from: ["/releases"],
- to: "/releases/release-0.15.0",
+ to: "/releases/release-1.0.0",
},
],
},
@@ -327,7 +327,7 @@ module.exports = {
},
{
label: "Releases",
- to: "/releases/release-0.15.0",
+ to: "/releases/release-1.0.0",
},
{
label: "Download",
@@ -509,8 +509,8 @@ module.exports = {
path: "next",
banner: "unreleased",
},
- "0.15.0": {
- label: "0.15.0",
+ "1.0.0": {
+ label: "1.0.0",
path: "",
},
},
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/azure_hoodie.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/azure_hoodie.md
new file mode 100644
index 00000000000..f7ccb8424a1
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/azure_hoodie.md
@@ -0,0 +1,52 @@
+---
+title: Azure 文件系统
+keywords: [ hudi, hive, azure, spark, presto]
+summary: 在本页中，我们讨论如何在 Azure 文件系统中配置 Hudi 。
+last_modified_at: 2020-05-25T19:00:57-04:00
+language: cn
+---
+在本页中，我们解释如何在 Microsoft Azure 上使用 Hudi 。
+
+## 声明
+
+本页面由 Hudi 社区维护。
+如果信息不准确，或者你有信息要补充，请尽管创建 JIRA ticket。
+对此贡献高度赞赏。
+
+## 支持的存储系统
+
+Hudi 支持两种存储系统。
+
+- Azure Blob 存储
+- Azure Data Lake Gen 2
+
+## 经过验证的 Spark 与存储系统的组合
+
+#### Azure Data Lake Storage Gen 2 上的 HDInsight Spark 2.4
+This combination works out of the box. No extra config needed.
+这种组合开箱即用，不需要额外的配置。
+
+#### Azure Data Lake Storage Gen 2 上的 Databricks Spark 2.4
+- 将 Hudi jar 包导入到 databricks 工作区 。
+
+- 将文件系统挂载到 dbutils 。
+ ```scala
+ dbutils.fs.mount(
+ source = "abfss://xxx@xxx.dfs.core.windows.net",
+ mountPoint = "/mountpoint",
+ extraConfigs = configs)
+ ```
+- 当写入 Hudi 数据集时，使用 abfss URL
+ ```scala
+ inputDF.write
+ .format("org.apache.hudi")
+ .options(opts)
+ .mode(SaveMode.Append)
+ .save("abfss://<<storage-account>>.dfs.core.windows.net/hudi-tables/customer")
+ ```
+- 当读取 Hudi 数据集时，使用挂载点
+ ```scala
+ spark.read
+ .format("org.apache.hudi")
+ .load("/mountpoint/hudi-tables/customer")
+ ```
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/bos_hoodie.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/bos_hoodie.md
new file mode 100644
index 00000000000..6cb9f582c25
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/bos_hoodie.md
@@ -0,0 +1,58 @@
+---
+title: BOS Filesystem
+keywords: [ hudi, hive, baidu, bos, spark, presto]
+summary: In this page, we go over how to configure Hudi with BOS filesystem.
+last_modified_at: 2021-06-09T11:38:24-10:00
+language: cn
+---
+这个页面描述了如何让你的Hudi任务使用Baidu BOS存储。
+
+## Baidu BOS 部署
+
+为了让Hudi使用BOS，需要增加两部分的配置:
+
+- 为Hudi增加Baidu BOS的相关配置
+- 增加Jar包到classpath
+
+### Baidu BOS 相关的配置
+
+新增下面的配置到你的Hudi能访问的core-site.xml文件。使用你的BOS bucket name替换掉`fs.defaultFS`，使用BOS endpoint地址替换`fs.bos.endpoint`，使用BOS的key和secret分别替换`fs.bos.access.key`和`fs.bos.secret.access.key`，这样Hudi就能读写相应的bucket。
+
+```xml
+<property>
+ <name>fs.defaultFS</name>
+ <value>bos://bucketname/</value>
+</property>
+
+<property>
+ <name>fs.bos.endpoint</name>
+ <value>bos-endpoint-address</value>
+ <description>Baidu bos endpoint to connect to,for example : http://bj.bcebos.com</description>
+</property>
+
+<property>
+ <name>fs.bos.access.key</name>
+ <value>bos-key</value>
+ <description>Baidu access key</description>
+</property>
+
+<property>
+ <name>fs.bos.secret.access.key</name>
+ <value>bos-secret-key</value>
+ <description>Baidu secret key.</description>
+</property>
+
+<property>
+ <name>fs.bos.impl</name>
+ <value>org.apache.hadoop.fs.bos.BaiduBosFileSystem</value>
+</property>
+```
+
+### Baidu BOS Libs
+
+新增Baidu hadoop的jar包添加到classpath.
+
+- com.baidubce:bce-java-sdk:0.10.165
+- bos-hdfs-sdk-1.0.2-community.jar
+
+可以从[这里](https://sdk.bce.baidu.com/console-sdk/bos-hdfs-sdk-1.0.2-community.jar.zip) 下载bos-hdfs-sdk jar包，然后解压。
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/cloud.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/cloud.md
new file mode 100644
index 00000000000..ae4d2728f98
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/cloud.md
@@ -0,0 +1,28 @@
+---
+title: 云储存
+keywords: [ hudi, aws, gcp, oss, azure, cloud]
+summary: "In this page, we introduce how Hudi work with different Cloud providers."
+toc: true
+last_modified_at: 2019-06-16T21:59:57-04:00
+language: cn
+---
+
+## 与云存储连接
+
+无论使用RDD/WriteClient API还是数据源，以下信息都有助于配置对云存储的访问。
+
+ * [AWS S3](/cn/docs/s3_hoodie) <br/>
+ S3和Hudi协同工作所需的配置。
+ * [Google Cloud Storage](/cn/docs/gcs_hoodie) <br/>
+ GCS和Hudi协同工作所需的配置。
+ * [Alibaba Cloud OSS](/cn/docs/oss_hoodie) <br/>
+ 阿里云和Hudi协同工作所需的配置。
+ * [Microsoft Azure](/cn/docs/azure_hoodie) <br/>
+ Azure和Hudi协同工作所需的配置。
+ * [Tencent Cloud Object Storage](/cn/docs/cos_hoodie) <br/>
+ COS和Hudi协同工作所需的配置。
+ * [IBM Cloud Object Storage](/cn/docs/ibm_cos_hoodie) <br/>
+ IBM Cloud Object Storage和Hudi协同工作所需的配置。
+* [Baidu Cloud Object Storage](bos_hoodie) <br/>
+ 百度BOS和Hudi协同工作所需的配置。
+
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/comparison.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/comparison.md
new file mode 100644
index 00000000000..050dd530a94
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/comparison.md
@@ -0,0 +1,48 @@
+---
+title: 对比
+keywords: [ apache, hudi, kafka, kudu, hive, hbase, stream processing]
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+Apache Hudi填补了在DFS上处理数据的巨大空白，并可以和这些技术很好地共存。然而，
+通过将Hudi与一些相关系统进行对比，来了解Hudi如何适应当前的大数据生态系统，并知晓这些系统在设计中做的不同权衡仍将非常有用。
+
+## Kudu
+
+[Apache Kudu](https://kudu.apache.org)是一个与Hudi具有相似目标的存储系统，该系统通过对`upserts`支持来对PB级数据进行实时分析。
+一个关键的区别是Kudu还试图充当OLTP工作负载的数据存储，而Hudi并不希望这样做。
+因此，Kudu不支持增量拉取(截至2017年初)，而Hudi支持以便进行增量处理。
+
+Kudu与分布式文件系统抽象和HDFS完全不同，它自己的一组存储服务器通过RAFT相互通信。
+与之不同的是，Hudi旨在与底层Hadoop兼容的文件系统(HDFS，S3或Ceph)一起使用，并且没有自己的存储服务器群，而是依靠Apache Spark来完成繁重的工作。
+因此，Hudi可以像其他Spark作业一样轻松扩展，而Kudu则需要硬件和运营支持，特别是HBase或Vertica等数据存储系统。
+到目前为止，我们还没有做任何直接的基准测试来比较Kudu和Hudi(鉴于RTTable正在进行中)。
+但是，如果我们要使用[CERN](https://db-blog.web.cern.ch/blog/zbigniew-baranowski/2017-01-performance-comparison-different-file-formats-and-storage-engines)，
+我们预期Hudi在摄取parquet上有更卓越的性能。
+
+## Hive事务
+
+[Hive事务/ACID](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)是另一项类似的工作，它试图实现在ORC文件格式之上的存储`读取时合并`。
+可以理解，此功能与Hive以及[LLAP](https://cwiki.apache.org/confluence/display/Hive/LLAP)之类的其他工作紧密相关。
+Hive事务不提供Hudi提供的读取优化存储选项或增量拉取。
+在实现选择方面，Hudi充分利用了类似Spark的处理框架的功能，而Hive事务特性则在用户或Hive Metastore启动的Hive任务/查询的下实现。
+根据我们的生产经验，与其他方法相比，将Hudi作为库嵌入到现有的Spark管道中要容易得多，并且操作不会太繁琐。
+Hudi还设计用于与Presto/Spark等非Hive引擎合作，并计划引入除parquet以外的文件格式。
+
+## HBase
+
+尽管[HBase](https://hbase.apache.org)最终是OLTP工作负载的键值存储层，但由于与Hadoop的相似性，用户通常倾向于将HBase与分析相关联。
+鉴于HBase经过严格的写优化，它支持开箱即用的亚秒级更新，Hive-on-HBase允许用户查询该数据。 但是，就分析工作负载的实际性能而言，Parquet/ORC之类的混合列式存储格式可以轻松击败HBase，因为这些工作负载主要是读取繁重的工作。
+Hudi弥补了更快的数据与分析存储格式之间的差距。从运营的角度来看，与管理分析使用的HBase region服务器集群相比，为用户提供可更快给出数据的库更具可扩展性。
+最终，HBase不像Hudi这样重点支持`提交时间`、`增量拉取`之类的增量处理原语。
+
+## 流式处理
+
+一个普遍的问题："Hudi与流处理系统有何关系？"，我们将在这里尝试回答。简而言之，Hudi可以与当今的批处理(`写时复制存储`)和流处理(`读时合并存储`)作业集成，以将计算结果存储在Hadoop中。
+对于Spark应用程序，这可以通过将Hudi库与Spark/Spark流式DAG直接集成来实现。在非Spark处理系统(例如Flink、Hive)情况下，可以在相应的系统中进行处理，然后通过Kafka主题/DFS中间文件将其发送到Hudi表中。从概念上讲，数据处理
+管道仅由三个部分组成：`输入`，`处理`，`输出`，用户最终针对输出运行查询以便使用管道的结果。Hudi可以充当将数据存储在DFS上的输入或输出。Hudi在给定流处理管道上的适用性最终归结为你的查询在Presto/SparkSQL/Hive的适用性。
+
+更高级的用例围绕[增量处理](https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop)的概念展开，
+甚至在`处理`引擎内部也使用Hudi来加速典型的批处理管道。例如：Hudi可用作DAG内的状态存储(类似Flink使用的[rocksDB(https://ci.apache.org/projects/flink/flink-docs-release-1.2/ops/state_backends#the-rocksdbstatebackend))。
+这是路线图上的一个项目并将最终以[Beam Runner](https://issues.apache.org/jira/browse/HUDI-60)的形式呈现。
\ No newline at end of file
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/concepts.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/concepts.md
new file mode 100644
index 00000000000..822b1ccb369
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/concepts.md
@@ -0,0 +1,154 @@
+---
+title: 概念
+keywords: [ hudi, design, storage, views, timeline]
+summary: "Here we introduce some basic concepts & give a broad technical overview of Hudi"
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+Apache Hudi(发音为“Hudi”)在DFS的数据集上提供以下流原语
+
+ * 插入更新 (如何改变数据集?)
+ * 增量拉取 (如何获取变更的数据?)
+
+在本节中，我们将讨论重要的概念和术语，这些概念和术语有助于理解并有效使用这些原语。
+
+## 时间轴
+在它的核心，Hudi维护一条包含在不同的`即时`时间所有对数据集操作的`时间轴`，从而提供，从不同时间点出发得到不同的视图下的数据集。Hudi即时包含以下组件
+
+ * `操作类型` : 对数据集执行的操作类型
+ * `即时时间` : 即时时间通常是一个时间戳(例如：20190117010349)，该时间戳按操作开始时间的顺序单调增加。
+ * `状态` : 即时的状态
+
+Hudi保证在时间轴上执行的操作的原子性和基于即时时间的时间轴一致性。
+
+执行的关键操作包括
+
+ * `COMMITS` - 一次提交表示将一组记录**原子写入**到数据集中。
+ * `CLEANS` - 删除数据集中不再需要的旧文件版本的后台活动。
+ * `DELTA_COMMIT` - 增量提交是指将一批记录**原子写入**到MergeOnRead存储类型的数据集中，其中一些/所有数据都可以只写到增量日志中。
+ * `COMPACTION` - 协调Hudi中差异数据结构的后台活动，例如：将更新从基于行的日志文件变成列格式。在内部，压缩表现为时间轴上的特殊提交。
+ * `ROLLBACK` - 表示提交/增量提交不成功且已回滚，删除在写入过程中产生的所有部分文件。
+ * `SAVEPOINT` - 将某些文件组标记为"已保存"，以便清理程序不会将其删除。在发生灾难/数据恢复的情况下，它有助于将数据集还原到时间轴上的某个点。
+
+任何给定的即时都可以处于以下状态之一
+
+ * `REQUESTED` - 表示已调度但尚未启动的操作。
+ * `INFLIGHT` - 表示当前正在执行该操作。
+ * `COMPLETED` - 表示在时间轴上完成了该操作。
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_timeline.png").default} alt="hudi_timeline.png" />
+</figure>
+
+上面的示例显示了在Hudi数据集上大约10:00到10:20之间发生的更新事件，大约每5分钟一次，将提交元数据以及其他后台清理/压缩保留在Hudi时间轴上。
+观察的关键点是：提交时间指示数据的`到达时间`（上午10:20），而实际数据组织则反映了实际时间或`事件时间`，即数据所反映的（从07:00开始的每小时时段）。在权衡数据延迟和完整性时，这是两个关键概念。
+
+如果有延迟到达的数据（事件时间为9:00的数据在10:20达到，延迟 >1 小时），我们可以看到upsert将新数据生成到更旧的时间段/文件夹中。
+在时间轴的帮助下，增量查询可以只提取10:00以后成功提交的新数据，并非常高效地只消费更改过的文件，且无需扫描更大的文件范围，例如07:00后的所有时间段。
+
+## 文件组织
+Hudi将DFS上的数据集组织到`基本路径`下的目录结构中。数据集分为多个分区，这些分区是包含该分区的数据文件的文件夹，这与Hive表非常相似。
+每个分区被相对于基本路径的特定`分区路径`区分开来。
+
+在每个分区内，文件被组织为`文件组`，由`文件id`唯一标识。
+每个文件组包含多个`文件切片`，其中每个切片包含在某个提交/压缩即时时间生成的基本列文件（`*.parquet`）以及一组日志文件（`*.log*`），该文件包含自生成基本文件以来对基本文件的插入/更新。
+Hudi采用MVCC设计，其中压缩操作将日志和基本文件合并以产生新的文件片，而清理操作则将未使用的/较旧的文件片删除以回收DFS上的空间。
+
+Hudi通过索引机制将给定的hoodie键（记录键+分区路径）映射到文件组，从而提供了高效的Upsert。
+一旦将记录的第一个版本写入文件，记录键和文件组/文件id之间的映射就永远不会改变。 简而言之，映射的文件组包含一组记录的所有版本。
+
+## 存储类型和视图
+Hudi存储类型定义了如何在DFS上对数据进行索引和布局以及如何在这种组织之上实现上述原语和时间轴活动（即如何写入数据）。
+反过来，`视图`定义了基础数据如何暴露给查询（即如何读取数据）。
+
+| 存储类型 | 支持的视图 |
+|-------------- |----------------|
+| 写时复制 | 近实时 + 增量 |
+| 读时合并 | 近实时 + 增量 + 读优化 |
+
+### 存储类型
+Hudi支持以下存储类型。
+
+ - [写时复制](#copy-on-write-storage) : 仅使用列文件格式（例如parquet）存储数据。通过在写入过程中执行同步合并以更新版本并重写文件。
+
+ - [读时合并](#merge-on-read-storage) : 使用列式（例如parquet）+ 基于行（例如avro）的文件格式组合来存储数据。 更新记录到增量文件中，然后进行同步或异步压缩以生成列文件的新版本。
+
+下表总结了这两种存储类型之间的权衡
+
+| 权衡 | 写时复制 | 读时合并 |
+|-------------- |------------------| ------------------|
+| 数据延迟 | 更高 | 更低 |
+| 更新代价(I/O) | 更高（重写整个parquet文件） | 更低（追加到增量日志） |
+| Parquet文件大小 | 更小（高更新代价（I/o）） | 更大（低更新代价） |
+| 写放大 | 更高 | 更低（取决于压缩策略） |
+
+
+### 视图
+Hudi支持以下存储数据的视图
+
+ - **读优化视图** : 在此视图上的查询将查看给定提交或压缩操作中数据集的最新快照。
+ 该视图仅将最新文件切片中的基本/列文件暴露给查询，并保证与非Hudi列式数据集相比，具有相同的列式查询性能。
+ - **增量视图** : 对该视图的查询只能看到从某个提交/压缩后写入数据集的新数据。该视图有效地提供了更改流，来支持增量数据管道。
+ - **实时视图** : 在此视图上的查询将查看某个增量提交操作中数据集的最新快照。该视图通过动态合并最新的基本文件(例如parquet)和增量文件(例如avro)来提供近实时数据集（几分钟的延迟）。
+
+
+下表总结了不同视图之间的权衡。
+
+| 权衡 | 读优化 | 实时 |
+|-------------- |------------------| ------------------|
+| 数据延迟 | 更高 | 更低 |
+| 查询延迟 | 更低（原始列式性能）| 更高（合并列式 + 基于行的增量） |
+
+
+## 写时复制存储 {#copy-on-write-storage}
+
+写时复制存储中的文件片仅包含基本/列文件，并且每次提交都会生成新版本的基本文件。
+换句话说，我们压缩每个提交，从而所有的数据都是以列数据的形式储存。在这种情况下，写入数据非常昂贵（我们需要重写整个列数据文件，即使只有一个字节的新数据被提交），而读取数据的成本则没有增加。
+这种视图有利于读取繁重的分析工作。
+
+以下内容说明了将数据写入写时复制存储并在其上运行两个查询时，它是如何工作的。
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_cow.png").default} alt="hudi_cow.png" />
+</figure>
+
+
+随着数据的写入，对现有文件组的更新将为该文件组生成一个带有提交即时时间标记的新切片，而插入分配一个新文件组并写入该文件组的第一个切片。
+这些文件切片及其提交即时时间在上面用颜色编码。
+针对这样的数据集运行SQL查询（例如：`select count(*)`统计该分区中的记录数目），首先检查时间轴上的最新提交并过滤每个文件组中除最新文件片以外的所有文件片。
+如您所见，旧查询不会看到以粉红色标记的当前进行中的提交的文件，但是在该提交后的新查询会获取新数据。因此，查询不受任何写入失败/部分写入的影响，仅运行在已提交数据上。
+
+写时复制存储的目的是从根本上改善当前管理数据集的方式，通过以下方法来实现
+
+ - 优先支持在文件级原子更新数据，而无需重写整个表/分区
+ - 能够只读取更新的部分，而不是进行低效的扫描或搜索
+ - 严格控制文件大小来保持出色的查询性能（小的文件会严重损害查询性能）。
+
+## 读时合并存储 {#merge-on-read-storage}
+
+读时合并存储是写时复制的升级版，从某种意义上说，它仍然可以通过读优化表提供数据集的读取优化视图（写时复制的功能）。
+此外，它将每个文件组的更新插入存储到基于行的增量日志中，通过文件id，将增量日志和最新版本的基本文件进行合并，从而提供近实时的数据查询。因此，此存储类型智能地平衡了读和写的成本，以提供近乎实时的查询。
+这里最重要的一点是压缩器，它现在可以仔细挑选需要压缩到其列式基础文件中的增量日志（根据增量日志的文件大小），以保持查询性能（较大的增量日志将会提升近实时的查询时间，并同时需要更长的合并时间）。
+
+以下内容说明了存储的工作方式，并显示了对近实时表和读优化表的查询。
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_mor.png").default} alt="hudi_mor.png" />
+</figure>
+
+此示例中发生了很多有趣的事情，这些带出了该方法的微妙之处。
+
+ - 现在，我们每1分钟左右就有一次提交，这是其他存储类型无法做到的。
+ - 现在，在每个文件id组中，都有一个增量日志，其中包含对基础列文件中记录的更新。
+ 在示例中，增量日志包含10:05至10:10的所有数据。与以前一样，基本列式文件仍使用提交进行版本控制。
+ 因此，如果只看一眼基本文件，那么存储布局看起来就像是写时复制表的副本。
+ - 定期压缩过程会从增量日志中合并这些更改，并生成基础文件的新版本，就像示例中10:05发生的情况一样。
+ - 有两种查询同一存储的方式：读优化（RO）表和近实时（RT）表，具体取决于我们选择查询性能还是数据新鲜度。
+ - 对于RO表来说，提交数据在何时可用于查询将有些许不同。 请注意，以10:10运行的（在RO表上的）此类查询将不会看到10:05之后的数据，而在RT表上的查询总会看到最新的数据。
+ - 何时触发压缩以及压缩什么是解决这些难题的关键。
+ 通过实施压缩策略，在该策略中，与较旧的分区相比，我们会积极地压缩最新的分区，从而确保RO表能够以一致的方式看到几分钟内发布的数据。
+
+读时合并存储上的目的是直接在DFS上启用近实时处理，而不是将数据复制到专用系统，后者可能无法处理大数据量。
+该存储还有一些其他方面的好处，例如通过避免数据的同步合并来减少写放大，即批量数据中每1字节数据需要的写入数据量。
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/configurations.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/configurations.md
new file mode 100644
index 00000000000..4c3a3415d3e
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/configurations.md
@@ -0,0 +1,604 @@
+---
+title: 配置
+keywords: [ garbage collection, hudi, jvm, configs, tuning]
+summary: "Here we list all possible configurations and what they mean"
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+该页面介绍了几种配置写入或读取Hudi数据集的作业的方法。
+简而言之，您可以在几个级别上控制行为。
+
+- **[Spark数据源配置](#spark-datasource)** : 这些配置控制Hudi Spark数据源，提供如下功能：
+ 定义键和分区、选择写操作、指定如何合并记录或选择要读取的视图类型。
+- **[WriteClient 配置](#writeclient-configs)** : 在内部，Hudi数据源使用基于RDD的`HoodieWriteClient` API
+ 真正执行对存储的写入。 这些配置可对文件大小、压缩（compression）、并行度、压缩（compaction）、写入模式、清理等底层方面进行完全控制。
+ 尽管Hudi提供了合理的默认设置，但在不同情形下，可能需要对这些配置进行调整以针对特定的工作负载进行优化。
+- **[RecordPayload 配置](#PAYLOAD_CLASS_OPT_KEY)** : 这是Hudi提供的最底层的定制。
+ RecordPayload定义了如何根据传入的新记录和存储的旧记录来产生新值以进行插入更新。
+ Hudi提供了诸如`OverwriteWithLatestAvroPayload`的默认实现，该实现仅使用最新或最后写入的记录来更新存储。
+ 在数据源和WriteClient级别，都可以将其重写为扩展`HoodieRecordPayload`类的自定义类。
+
+
+## Spark数据源配置
+
+可以通过将以下选项传递到`option(k,v)`方法中来配置使用数据源的Spark作业。
+实际的数据源级别配置在下面列出。
+
+### 写选项
+
+另外，您可以使用`options()`或`option(k,v)`方法直接传递任何WriteClient级别的配置。
+
+```java
+inputDF.write()
+.format("org.apache.hudi")
+.options(clientOpts) // 任何Hudi客户端选项都可以传入
+.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")
+.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")
+.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")
+.option(HoodieWriteConfig.TABLE_NAME, tableName)
+.mode(SaveMode.Append)
+.save(basePath);
+```
+
+用于通过`write.format.option(...)`写入数据集的选项
+
+#### TABLE_NAME_OPT_KEY
+ 属性：`hoodie.datasource.write.table.name` [必须]<br/>
+ <span >Hive表名，用于将数据集注册到其中。</span>
+
+#### OPERATION_OPT_KEY
+ 属性：`hoodie.datasource.write.operation`, 默认值：`upsert`<br/>
+ <span >是否为写操作进行插入更新、插入或批量插入。使用`bulkinsert`将新数据加载到表中，之后使用`upsert`或`insert`。
+ 批量插入使用基于磁盘的写入路径来扩展以加载大量输入，而无需对其进行缓存。</span>
+
+#### STORAGE_TYPE_OPT_KEY
+ 属性：`hoodie.datasource.write.storage.type`, 默认值：`COPY_ON_WRITE` <br/>
+ <span >此写入的基础数据的存储类型。两次写入之间不能改变。</span>
+
+#### PRECOMBINE_FIELD_OPT_KEY
+ 属性：`hoodie.datasource.write.precombine.field`, 默认值：`ts` <br/>
+ <span >实际写入之前在preCombining中使用的字段。
+ 当两个记录具有相同的键值时，我们将使用Object.compareTo(..)从precombine字段中选择一个值最大的记录。</span>
+
+#### PAYLOAD_CLASS_OPT_KEY
+ 属性：`hoodie.datasource.write.payload.class`, 默认值：`org.apache.hudi.OverwriteWithLatestAvroPayload` <br/>
+ <span >使用的有效载荷类。如果您想在插入更新或插入时使用自己的合并逻辑，请重写此方法。
+ 这将使得`PRECOMBINE_FIELD_OPT_VAL`设置的任何值无效</span>
+
+#### RECORDKEY_FIELD_OPT_KEY
+ 属性：`hoodie.datasource.write.recordkey.field`, 默认值：`uuid` <br/>
+ <span >记录键字段。用作`HoodieKey`中`recordKey`部分的值。
+ 实际值将通过在字段值上调用.toString()来获得。可以使用点符号指定嵌套字段，例如：`a.b.c`</span>
+
+#### PARTITIONPATH_FIELD_OPT_KEY
+ 属性：`hoodie.datasource.write.partitionpath.field`, 默认值：`partitionpath` <br/>
+ <span >分区路径字段。用作`HoodieKey`中`partitionPath`部分的值。
+ 通过调用.toString()获得实际的值</span>
+
+#### HIVE_STYLE_PARTITIONING_OPT_KEY
+ 属性：`hoodie.datasource.write.hive_style_partitioning`, 默认值：`false` <br/>
+ <span >如果设置为true，则生成基于Hive格式的partition目录：`partition_column_name=partition_value`</span>
+
+#### KEYGENERATOR_CLASS_OPT_KEY
+ 属性：`hoodie.datasource.write.keygenerator.class` <br/>
+ <span >键生成器类，实现从输入的`Row`对象中提取键。该配置优先级大于 `hoodie.datasource.write.keygenerator.type`, 用于使用用户自定义键生成器</span>
+
+#### KEYGENERATOR_TYPE_OPT_KEY
+ 属性: `hoodie.datasource.write.keygenerator.type`, 默认值: `SIMPLE` <br/>
+<span >键生成器类型，默认 `SIMPLE` 类型，该配置优先级低于 `hoodie.datasource.write.keygenerator.class`, 是推荐使用的配置方式</span>
+
+#### COMMIT_METADATA_KEYPREFIX_OPT_KEY
+ 属性：`hoodie.datasource.write.commitmeta.key.prefix`, 默认值：`_` <br/>
+ <span >以该前缀开头的选项键会自动添加到提交/增量提交的元数据中。
+ 这对于与hudi时间轴一致的方式存储检查点信息很有用</span>
+
+#### INSERT_DROP_DUPS_OPT_KEY
+ 属性：`hoodie.datasource.write.insert.drop.duplicates`, 默认值：`false` <br/>
+ <span >如果设置为true，则在插入操作期间从传入DataFrame中过滤掉所有重复记录。</span>
+
+#### HIVE_SYNC_ENABLED_OPT_KEY
+ 属性：`hoodie.datasource.hive_sync.enable`, 默认值：`false` <br/>
+ <span >设置为true时，将数据集注册并同步到Apache Hive Metastore</span>
+
+#### HIVE_DATABASE_OPT_KEY
+ 属性：`hoodie.datasource.hive_sync.database`, 默认值：`default` <br/>
+ <span >要同步到的数据库</span>
+
+#### HIVE_TABLE_OPT_KEY
+ 属性：`hoodie.datasource.hive_sync.table`, [Required] <br/>
+ <span >要同步到的表</span>
+
+#### HIVE_USER_OPT_KEY
+ 属性：`hoodie.datasource.hive_sync.username`, 默认值：`hive` <br/>
+ <span >要使用的Hive用户名</span>
+
+#### HIVE_PASS_OPT_KEY
+ 属性：`hoodie.datasource.hive_sync.password`, 默认值：`hive` <br/>
+ <span >要使用的Hive密码</span>
+
+#### HIVE_URL_OPT_KEY
+ 属性：`hoodie.datasource.hive_sync.jdbcurl`, 默认值：`jdbc:hive2://localhost:10000` <br/>
+ <span >Hive metastore url</span>
+
+#### HIVE_PARTITION_FIELDS_OPT_KEY
+ 属性：`hoodie.datasource.hive_sync.partition_fields`, 默认值：` ` <br/>
+ <span >数据集中用于确定Hive分区的字段。</span>
+
+#### HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY
+ 属性：`hoodie.datasource.hive_sync.partition_extractor_class`, 默认值：`org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor` <br/>
+ <span >用于将分区字段值提取到Hive分区列中的类。</span>
+
+#### HIVE_ASSUME_DATE_PARTITION_OPT_KEY
+ 属性：`hoodie.datasource.hive_sync.assume_date_partitioning`, 默认值：`false` <br/>
+ <span >假设分区格式是yyyy/mm/dd</span>
+
+### 读选项
+
+用于通过`read.format.option(...)`读取数据集的选项
+
+#### VIEW_TYPE_OPT_KEY
+属性：`hoodie.datasource.view.type`, 默认值：`read_optimized` <br/>
+<span >是否需要以某种模式读取数据，增量模式（自InstantTime以来的新数据）
+（或）读优化模式（基于列数据获取最新视图）
+（或）实时模式（基于行和列数据获取最新视图）</span>
+
+#### BEGIN_INSTANTTIME_OPT_KEY
+属性：`hoodie.datasource.read.begin.instanttime`, [在增量模式下必须] <br/>
+<span >开始增量提取数据的即时时间。这里的instanttime不必一定与时间轴上的即时相对应。
+取出以`instant_time > BEGIN_INSTANTTIME`写入的新数据。
+例如：'20170901080000'将获取2017年9月1日08:00 AM之后写入的所有新数据。</span>
+
+#### END_INSTANTTIME_OPT_KEY
+属性：`hoodie.datasource.read.end.instanttime`, 默认值：最新即时（即从开始即时获取所有新数据） <br/>
+<span >限制增量提取的数据的即时时间。取出以`instant_time <= END_INSTANTTIME`写入的新数据。</span>
+
+
+## WriteClient 配置
+
+直接使用RDD级别api进行编程的Jobs可以构建一个`HoodieWriteConfig`对象，并将其传递给`HoodieWriteClient`构造函数。
+HoodieWriteConfig可以使用以下构建器模式构建。
+
+```java
+HoodieWriteConfig cfg = HoodieWriteConfig.newBuilder()
+ .withPath(basePath)
+ .forTable(tableName)
+ .withSchema(schemaStr)
+ .withProps(props) // 从属性文件传递原始k、v对。
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder().withXXX(...).build())
+ .withIndexConfig(HoodieIndexConfig.newBuilder().withXXX(...).build())
+ ...
+ .build();
+```
+
+以下各节介绍了写配置的不同方面，并解释了最重要的配置及其属性名称和默认值。
+
+#### withPath(hoodie_base_path)
+属性：`hoodie.base.path` [必须] <br/>
+<span >创建所有数据分区所依据的基本DFS路径。
+始终在前缀中明确指明存储方式（例如hdfs://，s3://等）。
+Hudi将有关提交、保存点、清理审核日志等的所有主要元数据存储在基本目录下的.hoodie目录中。</span>
+
+#### withSchema(schema_str)
+属性：`hoodie.avro.schema` [必须]<br/>
+<span >这是数据集的当前读取器的avro模式（schema）。
+这是整个模式的字符串。HoodieWriteClient使用此模式传递到HoodieRecordPayload的实现，以从源格式转换为avro记录。
+在更新过程中重写记录时也使用此模式。</span>
+
+#### forTable(table_name)
+属性：`hoodie.table.name` [必须] <br/>
+ <span >数据集的表名，将用于在Hive中注册。每次运行需要相同。</span>
+
+#### withBulkInsertParallelism(bulk_insert_parallelism = 1500)
+属性：`hoodie.bulkinsert.shuffle.parallelism`<br/>
+<span >批量插入旨在用于较大的初始导入，而此处的并行度决定了数据集中文件的初始数量。
+调整此值以达到在初始导入期间所需的最佳尺寸。</span>
+
+#### withParallelism(insert_shuffle_parallelism = 1500, upsert_shuffle_parallelism = 1500)
+属性：`hoodie.insert.shuffle.parallelism`, `hoodie.upsert.shuffle.parallelism`<br/>
+<span >最初导入数据后，此并行度将控制用于读取输入记录的初始并行度。
+确保此值足够高，例如：1个分区用于1 GB的输入数据</span>
+
+#### combineInput(on_insert = false, on_update=true)
+属性：`hoodie.combine.before.insert`, `hoodie.combine.before.upsert`<br/>
+<span >在DFS中插入或更新之前先组合输入RDD并将多个部分记录合并为单个记录的标志</span>
+
+#### withWriteStatusStorageLevel(level = MEMORY_AND_DISK_SER)
+属性：`hoodie.write.status.storage.level`<br/>
+<span >HoodieWriteClient.insert和HoodieWriteClient.upsert返回一个持久的RDD[WriteStatus]，
+这是因为客户端可以选择检查WriteStatus并根据失败选择是否提交。这是此RDD的存储级别的配置</span>
+
+#### withAutoCommit(autoCommit = true)
+属性：`hoodie.auto.commit`<br/>
+<span >插入和插入更新后，HoodieWriteClient是否应该自动提交。
+客户端可以选择关闭自动提交，并在"定义的成功条件"下提交</span>
+
+#### withAssumeDatePartitioning(assumeDatePartitioning = false)
+属性：`hoodie.assume.date.partitioning`<br/>
+<span >HoodieWriteClient是否应该假设数据按日期划分，即从基本路径划分为三个级别。
+这是支持<0.3.1版本创建的表的一个补丁。最终将被删除</span>
+
+#### withConsistencyCheckEnabled(enabled = false)
+属性：`hoodie.consistency.check.enabled`<br/>
+<span >HoodieWriteClient是否应该执行其他检查，以确保写入的文件在基础文件系统/存储上可列出。
+将其设置为true可以解决S3的最终一致性模型，并确保作为提交的一部分写入的所有数据均能准确地用于查询。</span>
+
+### 索引配置
+以下配置控制索引行为，该行为将传入记录标记为对较旧记录的插入或更新。
+
+[withIndexConfig](#withIndexConfig) (HoodieIndexConfig) <br/>
+<span >可插入以具有外部索引（HBase）或使用存储在Parquet文件中的默认布隆过滤器（bloom filter）</span>
+
+#### withIndexClass(indexClass = "x.y.z.UserDefinedIndex")
+属性：`hoodie.index.class` <br/>
+<span >用户自定义索引的全路径名，索引类必须为HoodieIndex的子类，当指定该配置时，其会优先于`hoodie.index.type`配置</span>
+
+#### withIndexType(indexType = BLOOM)
+属性：`hoodie.index.type` <br/>
+<span >要使用的索引类型。默认为布隆过滤器。可能的选项是[BLOOM | HBASE | INMEMORY]。
+布隆过滤器消除了对外部系统的依赖，并存储在Parquet数据文件的页脚中</span>
+
+#### bloomFilterNumEntries(numEntries = 60000)
+属性：`hoodie.index.bloom.num_entries` <br/>
+<span >仅在索引类型为BLOOM时适用。<br/>这是要存储在布隆过滤器中的条目数。
+我们假设maxParquetFileSize为128MB，averageRecordSize为1024B，因此，一个文件中的记录总数约为130K。
+默认值（60000）大约是此近似值的一半。[HUDI-56](https://issues.apache.org/jira/browse/HUDI-56)
+描述了如何动态地对此进行计算。
+警告：将此值设置得太低，将产生很多误报，并且索引查找将必须扫描比其所需的更多的文件；如果将其设置得非常高，将线性增加每个数据文件的大小（每50000个条目大约4KB）。</span>
+
+#### bloomFilterFPP(fpp = 0.000000001)
+属性：`hoodie.index.bloom.fpp` <br/>
+<span >仅在索引类型为BLOOM时适用。<br/>根据条目数允许的错误率。
+这用于计算应为布隆过滤器分配多少位以及哈希函数的数量。通常将此值设置得很低（默认值：0.000000001），我们希望在磁盘空间上进行权衡以降低误报率</span>
+
+#### bloomIndexPruneByRanges(pruneRanges = true)
+属性：`hoodie.bloom.index.prune.by.ranges` <br/>
+<span >仅在索引类型为BLOOM时适用。<br/>为true时，从文件框定信息，可以加快索引查找的速度。 如果键具有单调递增的前缀，例如时间戳，则特别有用。</span>
+
+#### bloomIndexUseCaching(useCaching = true)
+属性：`hoodie.bloom.index.use.caching` <br/>
+<span >仅在索引类型为BLOOM时适用。<br/>为true时，将通过减少用于计算并行度或受影响分区的IO来缓存输入的RDD以加快索引查找</span>
+
+#### bloomIndexTreebasedFilter(useTreeFilter = true)
+属性：`hoodie.bloom.index.use.treebased.filter` <br/>
+<span >仅在索引类型为BLOOM时适用。<br/>为true时，启用基于间隔树的文件过滤优化。与暴力模式相比，此模式可根据键范围加快文件过滤速度</span>
+
+#### bloomIndexBucketizedChecking(bucketizedChecking = true)
+属性：`hoodie.bloom.index.bucketized.checking` <br/>
+<span >仅在索引类型为BLOOM时适用。<br/>为true时，启用了桶式布隆过滤。这减少了在基于排序的布隆索引查找中看到的偏差</span>
+
+#### bloomIndexKeysPerBucket(keysPerBucket = 10000000)
+属性：`hoodie.bloom.index.keys.per.bucket` <br/>
+<span >仅在启用bloomIndexBucketizedChecking并且索引类型为bloom的情况下适用。<br/>
+此配置控制“存储桶”的大小，该大小可跟踪对单个文件进行的记录键检查的次数，并且是分配给执行布隆过滤器查找的每个分区的工作单位。
+较高的值将分摊将布隆过滤器读取到内存的固定成本。</span>
+
+#### bloomIndexParallelism(0)
+属性：`hoodie.bloom.index.parallelism` <br/>
+<span >仅在索引类型为BLOOM时适用。<br/>这是索引查找的并行度，其中涉及Spark Shuffle。 默认情况下，这是根据输入的工作负载特征自动计算的</span>
+
+#### hbaseZkQuorum(zkString) [必须]
+属性：`hoodie.index.hbase.zkquorum` <br/>
+<span >仅在索引类型为HBASE时适用。要连接的HBase ZK Quorum URL。</span>
+
+#### hbaseZkPort(port) [必须]
+属性：`hoodie.index.hbase.zkport` <br/>
+<span >仅在索引类型为HBASE时适用。要连接的HBase ZK Quorum端口。</span>
+
+#### hbaseZkZnodeParent(zkZnodeParent) [必须]
+属性：`hoodie.index.hbase.zknode.path` <br/>
+<span >仅在索引类型为HBASE时适用。这是根znode，它将包含HBase创建及使用的所有znode。</span>
+
+#### hbaseTableName(tableName) [必须]
+属性：`hoodie.index.hbase.table` <br/>
+<span >仅在索引类型为HBASE时适用。HBase表名称，用作索引。Hudi将row_key和[partition_path, fileID, commitTime]映射存储在表中。</span>
+
+##### bloomIndexUpdatePartitionPath(updatePartitionPath = false)
+属性：`hoodie.bloom.index.update.partition.path` <br/>
+<span >仅在索引类型为GLOBAL_BLOOM时适用。<br/>为true时，当对一个已有记录执行包含分区路径的更新操作时，将会导致把新记录插入到新分区，而把原有记录从旧分区里删除。为false时，只对旧分区的原有记录进行更新。</span>
+
+
+### 存储选项
+控制有关调整parquet和日志文件大小的方面。
+
+[withStorageConfig](#withStorageConfig) (HoodieStorageConfig) <br/>
+
+#### limitFileSize (size = 120MB)
+属性：`hoodie.parquet.max.file.size` <br/>
+<span >Hudi写阶段生成的parquet文件的目标大小。对于DFS，这需要与基础文件系统块大小保持一致，以实现最佳性能。</span>
+
+#### parquetBlockSize(rowgroupsize = 120MB)
+属性：`hoodie.parquet.block.size` <br/>
+<span >Parquet行组大小。最好与文件大小相同，以便将文件中的单个列连续存储在磁盘上</span>
+
+#### parquetPageSize(pagesize = 1MB)
+属性：`hoodie.parquet.page.size` <br/>
+<span >Parquet页面大小。页面是parquet文件中的读取单位。 在一个块内，页面被分别压缩。</span>
+
+#### parquetCompressionRatio(parquetCompressionRatio = 0.1)
+属性：`hoodie.parquet.compression.ratio` <br/>
+<span >当Hudi尝试调整新parquet文件的大小时，预期对parquet数据进行压缩的比例。
+如果bulk_insert生成的文件小于预期大小，请增加此值</span>
+
+#### parquetCompressionCodec(parquetCompressionCodec = gzip)
+属性：`hoodie.parquet.compression.codec` <br/>
+<span >Parquet压缩编解码方式名称。默认值为gzip。可能的选项是[gzip | snappy | uncompressed | lzo]</span>
+
+#### logFileMaxSize(logFileSize = 1GB)
+属性：`hoodie.logfile.max.size` <br/>
+<span >LogFile的最大大小。这是在将日志文件移到下一个版本之前允许的最大大小。</span>
+
+#### logFileDataBlockMaxSize(dataBlockSize = 256MB)
+属性：`hoodie.logfile.data.block.max.size` <br/>
+<span >LogFile数据块的最大大小。这是允许将单个数据块附加到日志文件的最大大小。
+这有助于确保附加到日志文件的数据被分解为可调整大小的块，以防止发生OOM错误。此大小应大于JVM内存。</span>
+
+#### logFileToParquetCompressionRatio(logFileToParquetCompressionRatio = 0.35)
+属性：`hoodie.logfile.to.parquet.compression.ratio` <br/>
+<span >随着记录从日志文件移动到parquet，预期会进行额外压缩的比例。
+用于merge_on_read存储，以将插入内容发送到日志文件中并控制压缩parquet文件的大小。</span>
+
+#### parquetCompressionCodec(parquetCompressionCodec = gzip)
+属性：`hoodie.parquet.compression.codec` <br/>
+<span >Parquet文件的压缩编解码方式</span>
+
+### 压缩配置
+压缩配置用于控制压缩（将日志文件合并到新的parquet基本文件中）、清理（回收较旧及未使用的文件组）。
+[withCompactionConfig](#withCompactionConfig) (HoodieCompactionConfig) <br/>
+
+#### withCleanerPolicy(policy = KEEP_LATEST_COMMITS)
+属性：`hoodie.cleaner.policy` <br/>
+<span >要使用的清理政策。Hudi将删除旧版本的parquet文件以回收空间。
+任何引用此版本文件的查询和计算都将失败。最好确保数据保留的时间超过最大查询执行时间。</span>
+
+#### retainCommits(no_of_commits_to_retain = 24)
+属性：`hoodie.cleaner.commits.retained` <br/>
+<span >保留的提交数。因此，数据将保留为num_of_commits * time_between_commits（计划的）。
+这也直接转化为您可以逐步提取此数据集的数量</span>
+
+#### archiveCommitsWith(minCommits = 96, maxCommits = 128)
+属性：`hoodie.keep.min.commits`, `hoodie.keep.max.commits` <br/>
+<span >每个提交都是`.hoodie`目录中的一个小文件。由于DFS通常不支持大量小文件，因此Hudi将较早的提交归档到顺序日志中。
+提交通过重命名提交文件以原子方式发布。</span>
+
+#### withCommitsArchivalBatchSize(batch = 10)
+属性：`hoodie.commits.archival.batch` <br/>
+<span >这控制着批量读取并一起归档的提交即时的数量。</span>
+
+#### compactionSmallFileSize(size = 0)
+属性：`hoodie.parquet.small.file.limit` <br/>
+<span >该值应小于maxFileSize，如果将其设置为0，会关闭此功能。
+由于批处理中分区中插入记录的数量众多，总会出现小文件。
+Hudi提供了一个选项，可以通过将对该分区中的插入作为对现有小文件的更新来解决小文件的问题。
+此处的大小是被视为“小文件大小”的最小文件大小。</span>
+
+#### insertSplitSize(size = 500000)
+属性：`hoodie.copyonwrite.insert.split.size` <br/>
+<span >插入写入并行度。为单个分区的总共插入次数。
+写出100MB的文件，至少1kb大小的记录，意味着每个文件有100K记录。默认值是超额配置为500K。
+为了改善插入延迟，请对其进行调整以匹配单个文件中的记录数。
+将此值设置为较小的值将导致文件变小（尤其是当compactionSmallFileSize为0时）</span>
+
+#### autoTuneInsertSplits(true)
+属性：`hoodie.copyonwrite.insert.auto.split` <br/>
+<span >Hudi是否应该基于最后24个提交的元数据动态计算insertSplitSize。默认关闭。</span>
+
+#### approxRecordSize(size = 1024)
+属性：`hoodie.copyonwrite.record.size.estimate` <br/>
+<span >平均记录大小。如果指定，hudi将使用它，并且不会基于最后24个提交的元数据动态地计算。
+没有默认值设置。这对于计算插入并行度以及将插入打包到小文件中至关重要。如上所述。</span>
+
+#### withInlineCompaction(inlineCompaction = false)
+属性：`hoodie.compact.inline` <br/>
+<span >当设置为true时，紧接在插入或插入更新或批量插入的提交或增量提交操作之后由摄取本身触发压缩</span>
+
+#### withMaxNumDeltaCommitsBeforeCompaction(maxNumDeltaCommitsBeforeCompaction = 10)
+属性：`hoodie.compact.inline.max.delta.commits` <br/>
+<span >触发内联压缩之前要保留的最大增量提交数</span>
+
+#### withCompactionLazyBlockReadEnabled(true)
+属性：`hoodie.compaction.lazy.block.read` <br/>
+<span >当CompactedLogScanner合并所有日志文件时，此配置有助于选择是否应延迟读取日志块。
+选择true以使用I/O密集型延迟块读取（低内存使用），或者为false来使用内存密集型立即块读取（高内存使用）</span>
+
+#### withCompactionReverseLogReadEnabled(false)
+属性：`hoodie.compaction.reverse.log.read` <br/>
+<span >HoodieLogFormatReader会从pos=0到pos=file_length向前读取日志文件。
+如果此配置设置为true，则Reader会从pos=file_length到pos=0反向读取日志文件</span>
+
+#### withCleanerParallelism(cleanerParallelism = 200)
+属性：`hoodie.cleaner.parallelism` <br/>
+<span >如果清理变慢，请增加此值。</span>
+
+#### withCompactionStrategy(compactionStrategy = org.apache.hudi.io.compact.strategy.LogFileSizeBasedCompactionStrategy)
+属性：`hoodie.compaction.strategy` <br/>
+<span >用来决定在每次压缩运行期间选择要压缩的文件组的压缩策略。
+默认情况下，Hudi选择具有累积最多未合并数据的日志文件</span>
+
+#### withTargetIOPerCompactionInMB(targetIOPerCompactionInMB = 500000)
+属性：`hoodie.compaction.target.io` <br/>
+<span >LogFileSizeBasedCompactionStrategy的压缩运行期间要花费的MB量。当压缩以内联模式运行时，此值有助于限制摄取延迟。</span>
+
+#### withTargetPartitionsPerDayBasedCompaction(targetPartitionsPerCompaction = 10)
+属性：`hoodie.compaction.daybased.target` <br/>
+<span >由org.apache.hudi.io.compact.strategy.DayBasedCompactionStrategy使用，表示在压缩运行期间要压缩的最新分区数。</span>
+
+#### withPayloadClass(payloadClassName = org.apache.hudi.common.model.HoodieAvroPayload)
+属性：`hoodie.compaction.payload.class` <br/>
+<span >这需要与插入/插入更新过程中使用的类相同。
+就像写入一样，压缩也使用记录有效负载类将日志中的记录彼此合并，再次与基本文件合并，并生成压缩后要写入的最终记录。</span>
+
+
+
+### 指标配置
+配置Hudi指标报告。
+[withMetricsConfig](#withMetricsConfig) (HoodieMetricsConfig) <br/>
+<span >Hudi会发布有关每次提交、清理、回滚等的指标。</span>
+
+#### GRAPHITE
+
+##### on(metricsOn = false)
+属性：`hoodie.metrics.on` <br/>
+<span >打开或关闭发送指标。默认情况下处于关闭状态。</span>
+
+##### withReporterType(reporterType = GRAPHITE)
+属性：`hoodie.metrics.reporter.type` <br/>
+<span >指标报告者的类型。默认使用graphite。</span>
+
+##### toGraphiteHost(host = localhost)
+属性：`hoodie.metrics.graphite.host` <br/>
+<span >要连接的graphite主机</span>
+
+##### onGraphitePort(port = 4756)
+属性：`hoodie.metrics.graphite.port` <br/>
+<span >要连接的graphite端口</span>
+
+##### usePrefix(prefix = "")
+属性：`hoodie.metrics.graphite.metric.prefix` <br/>
+<span >适用于所有指标的标准前缀。这有助于添加如数据中心、环境等信息</span>
+
+#### JMX
+
+##### on(metricsOn = false)
+属性：`hoodie.metrics.on` <br/>
+<span >打开或关闭发送指标。默认情况下处于关闭状态。</span>
+
+##### withReporterType(reporterType = JMX)
+属性：`hoodie.metrics.reporter.type` <br/>
+<span >指标报告者的类型。</span>
+
+##### toJmxHost(host = localhost)
+属性：`hoodie.metrics.jmx.host` <br/>
+<span >要连接的Jmx主机</span>
+
+##### onJmxPort(port = 1000-5000)
+属性：`hoodie.metrics.graphite.port` <br/>
+<span >要连接的Jmx端口</span>
+
+#### DATADOG
+
+##### on(metricsOn = false)
+属性：`hoodie.metrics.on` <br/>
+<span >打开或关闭发送指标。默认情况下处于关闭状态。</span>
+
+##### withReporterType(reporterType = DATADOG)
+属性： `hoodie.metrics.reporter.type` <br/>
+<span >指标报告者的类型。</span>
+
+##### withDatadogReportPeriodSeconds(period = 30)
+属性： `hoodie.metrics.datadog.report.period.seconds` <br/>
+<span >Datadog报告周期，单位为秒，默认30秒。</span>
+
+##### withDatadogApiSite(apiSite)
+属性： `hoodie.metrics.datadog.api.site` <br/>
+<span >Datadog API站点：EU 或者 US</span>
+
+##### withDatadogApiKey(apiKey)
+属性： `hoodie.metrics.datadog.api.key` <br/>
+<span >Datadog API密匙</span>
+
+##### withDatadogApiKeySkipValidation(skip = false)
+属性： `hoodie.metrics.datadog.api.key.skip.validation` <br/>
+<span >在通过Datadog API发送指标前，选择是否跳过验证API密匙。默认不跳过。</span>
+
+##### withDatadogApiKeySupplier(apiKeySupplier)
+属性： `hoodie.metrics.datadog.api.key.supplier` <br/>
+<span >Datadog API 密匙提供者，用来在运行时提供密匙。只有当`hoodie.metrics.datadog.api.key`未设定的情况下才有效。</span>
+
+##### withDatadogApiTimeoutSeconds(timeout = 3)
+属性： `hoodie.metrics.datadog.metric.prefix` <br/>
+<span >Datadog API超时时长，单位为秒，默认3秒。</span>
+
+##### withDatadogPrefix(prefix)
+属性： `hoodie.metrics.datadog.metric.prefix` <br/>
+<span >Datadog指标前缀。将被加在所有指标名称前，以点间隔。例如：如果设成`foo`，`foo.`将被用作实际前缀。</span>
+
+##### withDatadogHost(host)
+属性： `hoodie.metrics.datadog.metric.host` <br/>
+<span >Datadog指标主机，将和指标数据一并发送。</span>
+
+##### withDatadogTags(tags)
+属性： `hoodie.metrics.datadog.metric.tags` <br/>
+<span >Datadog指标标签（逗号分隔），将和指标数据一并发送。</span>
+
+#### 用户自定义发送器
+
+##### on(metricsOn = false)
+属性： `hoodie.metrics.on` <br/>
+<span >打开或关闭发送指标。默认情况下处于关闭状态。</span>
+
+##### withReporterClass(className = "")
+属性： `hoodie.metrics.reporter.class` <br/>
+<span >用于处理发送指标的用户自定义类，必须是AbstractUserDefinedMetricsReporter类的子类.</span>
+
+### 内存配置
+控制由Hudi内部执行的压缩和合并的内存使用情况
+[withMemoryConfig](#withMemoryConfig) (HoodieMemoryConfig) <br/>
+<span >内存相关配置</span>
+
+#### withMaxMemoryFractionPerPartitionMerge(maxMemoryFractionPerPartitionMerge = 0.6)
+属性：`hoodie.memory.merge.fraction` <br/>
+<span >该比例乘以用户内存比例（1-spark.memory.fraction）以获得合并期间要使用的堆空间的最终比例</span>
+
+#### withMaxMemorySizePerCompactionInBytes(maxMemorySizePerCompactionInBytes = 1GB)
+属性：`hoodie.memory.compaction.fraction` <br/>
+<span >HoodieCompactedLogScanner读取日志块，将记录转换为HoodieRecords，然后合并这些日志块和记录。
+在任何时候，日志块中的条目数可以小于或等于相应的parquet文件中的条目数。这可能导致Scanner出现OOM。
+因此，可溢出的映射有助于减轻内存压力。使用此配置来设置可溢出映射的最大允许inMemory占用空间。</span>
+
+#### withWriteStatusFailureFraction(failureFraction = 0.1)
+属性：`hoodie.memory.writestatus.failure.fraction` <br/>
+<span >此属性控制报告给驱动程序的失败记录和异常的比例</span>
+
+### 写提交回调配置
+控制写提交的回调。 如果用户启用了回调并且回调过程发生了错误，则会抛出异常。 当前支持HTTP, Kafka 两种回调方式。
+[withCallbackConfig](#withCallbackConfig) (HoodieWriteCommitCallbackConfig) <br/>
+<span >写提交回调相关配置</span>
+
+##### writeCommitCallbackOn(callbackOn = false)
+Property: `hoodie.write.commit.callback.on` <br/>
+<span >打开或关闭回调功能. 默认关闭.</span>
+
+##### withCallbackClass(callbackClass)
+Property: `hoodie.write.commit.callback.class` <br/>
+<span >回调类的完全限定名，必须实现HoodieWriteCommitCallback接口。默认 org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback</span>
+
+#### HTTP CALLBACK
+通过 HTTP 发送写提交回调信息. 这是默认的实现方式，用户不需要显式指定。
+
+##### withCallbackHttpUrl(url)
+Property: `hoodie.write.commit.callback.http.url` <br/>
+<span >Http回调主机，回调信息将会发送到该主机</span>
+
+##### withCallbackHttpTimeoutSeconds(timeoutSeconds = 3)
+Property: `hoodie.write.commit.callback.http.timeout.seconds` <br/>
+<span >Http回调超时时间（单位秒），默认3秒</span>
+
+##### withCallbackHttpApiKey(apiKey)
+Property: `hoodie.write.commit.callback.http.api.key` <br/>
+<span >Http 回调秘钥. 默认 hudi_write_commit_http_callback</span>
+
+#### KAFKA CALLBACK
+使用Kafka发送写提交回调信息, 用户需要配置 `hoodie.write.commit.callback.class` = `org.apache.hudi.utilities.callback.kafka.HoodieWriteCommitKafkaCallback`
+
+##### CALLBACK_KAFKA_BOOTSTRAP_SERVERS
+Property: `hoodie.write.commit.callback.kafka.bootstrap.servers` <br/>
+<span >Kafka 集群地址</span>
+
+##### CALLBACK_KAFKA_TOPIC
+Property: `hoodie.write.commit.callback.kafka.topic` <br/>
+<span >发送回调信息的topic</span>
+
+##### CALLBACK_KAFKA_PARTITION
+Property: `hoodie.write.commit.callback.kafka.partition` <br/>
+<span >指定发送的分区, 默认 0 </span>
+
+##### CALLBACK_KAFKA_ACKS
+Property: `hoodie.write.commit.callback.kafka.acks` <br/>
+<span >Acks 级别, 默认 `all`</span>
+
+##### CALLBACK_KAFKA_RETRIES
+Property: `hoodie.write.commit.callback.kafka.retries` <br/>
+<span >Kafka 发送数据失败重试次数. 默认 3 次</span>
+
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/cos_hoodie.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/cos_hoodie.md
new file mode 100644
index 00000000000..043e3768619
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/cos_hoodie.md
@@ -0,0 +1,73 @@
+---
+title: COS Filesystem
+keywords: [ hudi, hive, tencent, cos, spark, presto]
+summary: In this page, we go over how to configure Hudi with COS filesystem.
+last_modified_at: 2020-04-21T12:50:50-10:00
+language: cn
+---
+这个页面描述了如何让你的Hudi spark任务使用Tencent Cloud COS存储。
+
+## Tencent Cloud COS 部署
+
+为了让Hudi使用COS，需要增加两部分的配置:
+
+- 为Hidi增加Tencent Cloud COS的相关配置
+- 增加Jar包的MVN依赖
+
+### Tencent Cloud COS 相关的配置
+
+新增下面的配置到你的Hudi能访问的core-site.xml文件。使用你的COS bucket name替换掉`fs.defaultFS`，使用COS的key和secret分别替换`fs.cosn.userinfo.secretKey`和`fs.cosn.userinfo.secretId`。主要Hudi就能读写相应的bucket。
+
+
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>cosn://bucketname</value>
+ <description>COS bucket name</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.userinfo.secretId</name>
+ <value>cos-secretId</value>
+ <description>Tencent Cloud Secret Id</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.userinfo.secretKey</name>
+ <value>cos-secretkey</value>
+ <description>Tencent Cloud Secret Key</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.bucket.region</name>
+ <value>ap-region</value>
+ <description>The region where the bucket is located.</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.bucket.endpoint_suffix</name>
+ <value>cos.endpoint.suffix</value>
+ <description>
+ COS endpoint to connect to.
+ For public cloud users, it is recommended not to set this option, and only the correct area field is required.
+ </description>
+ </property>
+
+ <property>
+ <name>fs.cosn.impl</name>
+ <value>org.apache.hadoop.fs.CosFileSystem</value>
+ <description>The implementation class of the CosN Filesystem.</description>
+ </property>
+
+ <property>
+ <name>fs.AbstractFileSystem.cosn.impl</name>
+ <value>org.apache.hadoop.fs.CosN</value>
+ <description>The implementation class of the CosN AbstractFileSystem.</description>
+ </property>
+
+```
+
+### Tencent Cloud COS Libs
+添加COS依赖jar包到classpath
+
+- org.apache.hadoop:hadoop-cos:2.8.5
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/deployment.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/deployment.md
new file mode 100644
index 00000000000..76c79cf4c8e
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/deployment.md
@@ -0,0 +1,433 @@
+---
+title: 管理 Hudi Pipelines
+keywords: [ hudi, administration, operation, devops]
+summary: This section offers an overview of tools available to operate an ecosystem of Hudi datasets
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+管理员/运维人员可以通过以下方式了解Hudi数据集/管道
+
+ - [通过Admin CLI进行管理](#admin-cli)
+ - [Graphite指标](#metrics)
+ - [Hudi应用程序的Spark UI](#spark-ui)
+
+本节简要介绍了每一种方法，并提供了有关[故障排除](#troubleshooting)的一些常规指南
+
+## Admin CLI {#admin-cli}
+
+一旦构建了hudi，就可以通过`cd hudi-cli && ./hudi-cli.sh`启动shell。
+一个hudi数据集位于DFS上的**basePath**位置，我们需要该位置才能连接到Hudi数据集。
+Hudi库使用.hoodie子文件夹跟踪所有元数据，从而有效地在内部管理该数据集。
+
+初始化hudi表，可使用如下命令。
+
+```java
+18/09/06 15:56:52 INFO annotation.AutowiredAnnotationBeanPostProcessor: JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
+============================================
+* *
+* _ _ _ _ *
+* | | | | | | (_) *
+* | |__| | __| | - *
+* | __ || | / _` | || *
+* | | | || || (_| | || *
+* |_| |_|\___/ \____/ || *
+* *
+============================================
+
+Welcome to Hoodie CLI. Please type help if you are looking for help.
+hudi->create --path /user/hive/warehouse/table1 --tableName hoodie_table_1 --tableType COPY_ON_WRITE
+.....
+18/09/06 15:57:15 INFO table.HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE from ...
+```
+
+使用desc命令可以查看hudi表的描述信息:
+
+```java
+hoodie:hoodie_table_1->desc
+18/09/06 15:57:19 INFO timeline.HoodieActiveTimeline: Loaded instants []
+ _________________________________________________________
+ | Property | Value |
+ |========================================================|
+ | basePath | ... |
+ | metaPath | ... |
+ | fileSystem | hdfs |
+ | hoodie.table.name | hoodie_table_1 |
+ | hoodie.table.type | COPY_ON_WRITE |
+ | hoodie.archivelog.folder| |
+```
+
+以下是连接到包含uber trips的Hudi数据集的示例命令。
+
+```java
+hoodie:trips->connect --path /app/uber/trips
+
+16/10/05 23:20:37 INFO model.HoodieTableMetadata: Attempting to load the commits under /app/uber/trips/.hoodie with suffix .commit
+16/10/05 23:20:37 INFO model.HoodieTableMetadata: Attempting to load the commits under /app/uber/trips/.hoodie with suffix .inflight
+16/10/05 23:20:37 INFO model.HoodieTableMetadata: All commits :HoodieCommits{commitList=[20161002045850, 20161002052915, 20161002055918, 20161002065317, 20161002075932, 20161002082904, 20161002085949, 20161002092936, 20161002105903, 20161002112938, 20161002123005, 20161002133002, 20161002155940, 20161002165924, 20161002172907, 20161002175905, 20161002190016, 20161002192954, 20161002195925, 20161002205935, 20161002215928, 20161002222938, 20161002225915, 20161002232906, 20161003003028, 201 [...]
+Metadata for table trips loaded
+hoodie:trips->
+```
+
+连接到数据集后，便可使用许多其他命令。该shell程序具有上下文自动完成帮助(按TAB键)，下面是所有命令的列表，本节中对其中的一些命令进行了详细示例。
+
+
+```java
+hoodie:trips->help
+* ! - Allows execution of operating system (OS) commands
+* // - Inline comment markers (start of line only)
+* ; - Inline comment markers (start of line only)
+* addpartitionmeta - Add partition metadata to a dataset, if not present
+* clear - Clears the console
+* cls - Clears the console
+* commit rollback - Rollback a commit
+* commits compare - Compare commits with another Hoodie dataset
+* commit showfiles - Show file level details of a commit
+* commit showpartitions - Show partition level details of a commit
+* commits refresh - Refresh the commits
+* commits show - Show the commits
+* commits sync - Compare commits with another Hoodie dataset
+* connect - Connect to a hoodie dataset
+* date - Displays the local date and time
+* exit - Exits the shell
+* help - List all commands usage
+* quit - Exits the shell
+* records deduplicate - De-duplicate a partition path contains duplicates & produce repaired files to replace with
+* script - Parses the specified resource file and executes its commands
+* stats filesizes - File Sizes. Display summary stats on sizes of files
+* stats wa - Write Amplification. Ratio of how many records were upserted to how many records were actually written
+* sync validate - Validate the sync by counting the number of records
+* system properties - Shows the shell's properties
+* utils loadClass - Load a class
+* version - Displays shell version
+
+hoodie:trips->
+```
+
+
+### 检查提交
+
+在Hudi中，更新或插入一批记录的任务被称为**提交**。提交可提供基本的原子性保证，即只有提交的数据可用于查询。
+每个提交都有一个单调递增的字符串/数字，称为**提交编号**。通常，这是我们开始提交的时间。
+
+查看有关最近10次提交的一些基本信息，
+
+
+```java
+hoodie:trips->commits show --sortBy "Total Bytes Written" --desc true --limit 10
+ ________________________________________________________________________________________________________________________________________________________________________
+ | CommitTime | Total Bytes Written| Total Files Added| Total Files Updated| Total Partitions Written| Total Records Written| Total Update Records Written| Total Errors|
+ |=======================================================================================================================================================================|
+ ....
+ ....
+ ....
+hoodie:trips->
+```
+
+在每次写入开始时，Hudi还将.inflight提交写入.hoodie文件夹。您可以使用那里的时间戳来估计正在进行的提交已经花费的时间
+
+```java
+$ hdfs dfs -ls /app/uber/trips/.hoodie/*.inflight
+-rw-r--r-- 3 vinoth supergroup 321984 2016-10-05 23:18 /app/uber/trips/.hoodie/20161005225920.inflight
+```
+
+
+### 深入到特定的提交
+
+了解写入如何分散到特定分区，
+
+
+```java
+hoodie:trips->commit showpartitions --commit 20161005165855 --sortBy "Total Bytes Written" --desc true --limit 10
+ __________________________________________________________________________________________________________________________________________
+ | Partition Path| Total Files Added| Total Files Updated| Total Records Inserted| Total Records Updated| Total Bytes Written| Total Errors|
+ |=========================================================================================================================================|
+ ....
+ ....
+```
+
+如果您需要文件级粒度，我们可以执行以下操作
+
+```java
+hoodie:trips->commit showfiles --commit 20161005165855 --sortBy "Partition Path"
+ ________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File ID | Previous Commit| Total Records Updated| Total Records Written| Total Bytes Written| Total Errors|
+ |=======================================================================================================================================================|
+ ....
+ ....
+```
+
+
+### 文件系统视图
+
+Hudi将每个分区视为文件组的集合，每个文件组包含按提交顺序排列的文件切片列表(请参阅概念)。以下命令允许用户查看数据集的文件切片。
+
+```java
+ hoodie:stock_ticks_mor->show fsview all
+ ....
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta File Size| Delta Files |
+ |==============================================================================================================================================================================================================================================================================================================================================================================================================|
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]|
+
+
+
+ hoodie:stock_ticks_mor->show fsview latest --partitionPath "2018/08/31"
+ ......
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ [...]
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta Size| Delta Size - compaction scheduled| Delta Size - compaction unscheduled| Delta To Base Ratio - compaction scheduled| Delta To Base Ratio - compaction unscheduled| Delta Files - compaction scheduled | Delta Files - compaction unscheduled|
+ |========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== [...]
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | 20.8 KB | 0.0 B | 0.0 B | 0.0 B | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]| [] |
+
+ hoodie:stock_ticks_mor->
+```
+
+
+### 统计信息
+
+由于Hudi直接管理DFS数据集的文件大小，这些信息会帮助你全面了解Hudi的运行状况
+
+
+```java
+hoodie:trips->stats filesizes --partitionPath 2016/09/01 --sortBy "95th" --desc true --limit 10
+ ________________________________________________________________________________________________
+ | CommitTime | Min | 10th | 50th | avg | 95th | Max | NumFiles| StdDev |
+ |===============================================================================================|
+ | <COMMIT_ID> | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 2 | 2.3 KB |
+ ....
+ ....
+```
+
+如果Hudi写入花费的时间更长，那么可以通过观察写放大指标来发现任何异常
+
+```java
+hoodie:trips->stats wa
+ __________________________________________________________________________
+ | CommitTime | Total Upserted| Total Written| Write Amplifiation Factor|
+ |=========================================================================|
+ ....
+ ....
+```
+
+
+### 归档的提交
+
+为了限制DFS上.commit文件的增长量，Hudi将较旧的.commit文件(适当考虑清理策略)归档到commits.archived文件中。
+这是一个序列文件，其包含commitNumber => json的映射，及有关提交的原始信息(上面已很好地汇总了相同的信息)。
+
+### 压缩
+
+要了解压缩和写程序之间的时滞，请使用以下命令列出所有待处理的压缩。
+
+```java
+hoodie:trips->compactions show all
+ ___________________________________________________________________
+ | Compaction Instant Time| State | Total FileIds to be Compacted|
+ |==================================================================|
+ | <INSTANT_1> | REQUESTED| 35 |
+ | <INSTANT_2> | INFLIGHT | 27 |
+```
+
+要检查特定的压缩计划，请使用
+
+```java
+hoodie:trips->compaction show --instant <INSTANT_1>
+ _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File Id | Base Instant | Data File Path | Total Delta Files| getMetrics |
+ |================================================================================================================================================================================================================================================
+ | 2018/07/17 | <UUID> | <INSTANT_1> | viewfs://ns-default/.../../UUID_<INSTANT>.parquet | 1 | {TOTAL_LOG_FILES=1.0, TOTAL_IO_READ_MB=1230.0, TOTAL_LOG_FILES_SIZE=2.51255751E8, TOTAL_IO_WRITE_MB=991.0, TOTAL_IO_MB=2221.0}|
+
+```
+
+要手动调度或运行压缩，请使用以下命令。该命令使用spark启动器执行压缩操作。
+注意：确保没有其他应用程序正在同时调度此数据集的压缩
+
+```java
+hoodie:trips->help compaction schedule
+Keyword: compaction schedule
+Description: Schedule Compaction
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: false
+ Default if specified: '__NULL__'
+ Default if unspecified: '1G'
+
+* compaction schedule - Schedule Compaction
+```
+
+```java
+hoodie:trips->help compaction run
+Keyword: compaction run
+Description: Run Compaction for given instant time
+ Keyword: tableName
+ Help: Table name
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: parallelism
+ Help: Parallelism for hoodie compaction
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: schemaFilePath
+ Help: Path for Avro schema file
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: retry
+ Help: Number of retries
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: compactionInstant
+ Help: Base path for the target hoodie dataset
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+* compaction run - Run Compaction for given instant time
+```
+
+### 验证压缩
+
+验证压缩计划：检查压缩所需的所有文件是否都存在且有效
+
+```java
+hoodie:stock_ticks_mor->compaction validate --instant 20181005222611
+...
+
+ COMPACTION PLAN VALID
+
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error|
+ |==========================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | true | |
+
+
+
+hoodie:stock_ticks_mor->compaction validate --instant 20181005222601
+
+ COMPACTION PLAN INVALID
+
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error |
+ |=====================================================================================================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | false| All log files specified in compaction operation is not present. Missing .... |
+```
+
+### 注意
+
+必须在其他写入/摄取程序没有运行的情况下执行以下命令。
+
+有时，有必要从压缩计划中删除fileId以便加快或取消压缩操作。
+压缩计划之后在此文件上发生的所有新日志文件都将被安全地重命名以便进行保留。Hudi提供以下CLI来支持
+
+
+### 取消调度压缩
+
+```java
+hoodie:trips->compaction unscheduleFileId --fileId <FileUUID>
+....
+No File renames needed to unschedule file from pending compaction. Operation successful.
+```
+
+在其他情况下，需要撤销整个压缩计划。以下CLI支持此功能
+
+```java
+hoodie:trips->compaction unschedule --instant <compactionInstant>
+.....
+No File renames needed to unschedule pending compaction. Operation successful.
+```
+
+### 修复压缩
+
+上面的压缩取消调度操作有时可能会部分失败(例如：DFS暂时不可用)。
+如果发生部分故障，则压缩操作可能与文件切片的状态不一致。
+当您运行`压缩验证`时，您会注意到无效的压缩操作(如果有的话)。
+在这种情况下，修复命令将立即执行，它将重新排列文件切片，以使文件不丢失，并且文件切片与压缩计划一致
+
+```java
+hoodie:stock_ticks_mor->compaction repair --instant 20181005222611
+......
+Compaction successfully repaired
+.....
+```
+
+
+## [指标](/cn/docs/configurations#指标配置) {#metrics}
+
+为Hudi Client配置正确的数据集名称和指标环境后，它将生成以下graphite指标，以帮助调试hudi数据集
+
+ - **提交持续时间** - 这是成功提交一批记录所花费的时间
+ - **回滚持续时间** - 同样，撤消失败的提交所剩余的部分数据所花费的时间(每次写入失败后都会自动发生)
+ - **文件级别指标** - 显示每次提交中新增、版本、删除(清除)的文件数量
+ - **记录级别指标** - 每次提交插入/更新的记录总数
+ - **分区级别指标** - 更新的分区数量(对于了解提交持续时间的突然峰值非常有用)
+
+然后可以将这些指标绘制在grafana等标准工具上。以下是提交持续时间图表示例。
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_commit_duration.png").default} alt="hudi_commit_duration.png" />
+</figure>
+
+
+## 故障排除 {#troubleshooting}
+
+以下部分通常有助于调试Hudi故障。以下元数据已被添加到每条记录中，可以通过标准Hadoop SQL引擎(Hive/Presto/Spark)检索，来更容易地诊断问题的严重性。
+
+ - **_hoodie_record_key** - 作为每个DFS分区内的主键，是所有更新/插入的基础
+ - **_hoodie_commit_time** - 该记录上次的提交
+ - **_hoodie_file_name** - 包含记录的实际文件名(对检查重复非常有用)
+ - **_hoodie_partition_path** - basePath的路径，该路径标识包含此记录的分区
+
+请注意，到目前为止，Hudi假定应用程序为给定的recordKey传递相同的确定性分区路径。即仅在每个分区内保证recordKey(主键)的唯一性。
+
+### 缺失记录
+
+请在可能写入记录的窗口中，使用上面的admin命令检查是否存在任何写入错误。
+如果确实发现错误，那么记录实际上不是由Hudi写入的，而是交还给应用程序来决定如何处理。
+
+### 重复
+
+首先，请确保访问Hudi数据集的查询是[没有问题的](querying_data)，并之后确认的确有重复。
+
+ - 如果确认，请使用上面的元数据字段来标识包含记录的物理文件和分区文件。
+ - 如果重复的记录存在于不同分区路径下的文件，则意味着您的应用程序正在为同一recordKey生成不同的分区路径，请修复您的应用程序.
+ - 如果重复的记录存在于同一分区路径下的多个文件，请使用邮件列表汇报这个问题。这不应该发生。您可以使用`records deduplicate`命令修复数据。
+
+### Spark故障 {#spark-ui}
+
+典型的upsert() DAG如下所示。请注意，Hudi客户端会缓存中间的RDD，以智能地并调整文件大小和Spark并行度。
+另外，由于还显示了探针作业，Spark UI显示了两次sortByKey，但它只是一个排序。
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_upsert_dag.png").default} alt="hudi_upsert_dag.png" />
+</figure>
+
+
+概括地说，有两个步骤
+
+**索引查找以标识要更改的文件**
+
+ - Job 1 : 触发输入数据读取，转换为HoodieRecord对象，然后根据输入记录拿到目标分区路径。
+ - Job 2 : 加载我们需要检查的文件名集。
+ - Job 3 & 4 : 通过联合上面1和2中的RDD，智能调整spark join并行度，然后进行实际查找。
+ - Job 5 : 生成带有位置的recordKeys作为标记的RDD。
+
+**执行数据的实际写入**
+
+ - Job 6 : 将记录与recordKey(位置)进行懒惰连接，以提供最终的HoodieRecord集，现在它包含每条记录的文件/分区路径信息(如果插入，则为null)。然后还要再次分析工作负载以确定文件的大小。
+ - Job 7 : 实际写入数据(更新 + 插入 + 插入转为更新以保持文件大小)
+
+根据异常源(Hudi/Spark)，上述关于DAG的信息可用于查明实际问题。最常遇到的故障是由YARN/DFS临时故障引起的。
+将来，将在项目中添加更复杂的调试/管理UI，以帮助自动进行某些调试。
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/docker_demo.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/docker_demo.md
new file mode 100644
index 00000000000..eea0e884d08
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/docker_demo.md
@@ -0,0 +1,1122 @@
+---
+title: Docker Demo
+keywords: [ hudi, docker, demo]
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+## 一个使用 Docker 容器的 Demo
+
+我们来使用一个真实世界的案例，来看看 Hudi 是如何闭环运转的。 为了这个目的，在你的计算机中的本地 Docker 集群中组建了一个自包含的数据基础设施。
+
+以下步骤已经在一台 Mac 笔记本电脑上测试过了。
+
+### 前提条件
+
+ * Docker 安装 : 对于 Mac ，请依照 [https://docs.docker.com/v17.12/docker-for-mac/install/] 当中定义的步骤。 为了运行 Spark-SQL 查询，请确保至少分配给 Docker 6 GB 和 4 个 CPU 。（参见 Docker -> Preferences -> Advanced）。否则，Spark-SQL 查询可能被因为内存问题而被杀停。
+ * kafkacat : 一个用于发布/消费 Kafka Topic 的命令行工具集。使用 `brew install kafkacat` 来安装 kafkacat 。
+ * /etc/hosts : Demo 通过主机名引用了多个运行在容器中的服务。将下列设置添加到 /etc/hosts ：
+
+
+```java
+ 127.0.0.1 adhoc-1
+ 127.0.0.1 adhoc-2
+ 127.0.0.1 namenode
+ 127.0.0.1 datanode1
+ 127.0.0.1 hiveserver
+ 127.0.0.1 hivemetastore
+ 127.0.0.1 kafkabroker
+ 127.0.0.1 sparkmaster
+ 127.0.0.1 zookeeper
+```
+
+此外，这未在其它一些环境中进行测试，例如 Windows 上的 Docker 。
+
+
+## 设置 Docker 集群
+
+
+### 构建 Hudi
+
+构建 Hudi 的第一步：
+```java
+cd <HUDI_WORKSPACE>
+mvn package -DskipTests
+```
+
+### 组建 Demo 集群
+
+下一步是运行 Docker 安装脚本并设置配置项以便组建集群。
+这需要从 Docker 镜像库拉取 Docker 镜像，并设置 Docker 集群。
+
+```java
+cd docker
+./setup_demo.sh
+....
+....
+....
+Stopping spark-worker-1 ... done
+Stopping hiveserver ... done
+Stopping hivemetastore ... done
+Stopping historyserver ... done
+.......
+......
+Creating network "hudi_demo" with the default driver
+Creating hive-metastore-postgresql ... done
+Creating namenode ... done
+Creating zookeeper ... done
+Creating kafkabroker ... done
+Creating hivemetastore ... done
+Creating historyserver ... done
+Creating hiveserver ... done
+Creating datanode1 ... done
+Creating presto-coordinator-1 ... done
+Creating sparkmaster ... done
+Creating presto-worker-1 ... done
+Creating adhoc-1 ... done
+Creating adhoc-2 ... done
+Creating spark-worker-1 ... done
+Copying spark default config and setting up configs
+Copying spark default config and setting up configs
+Copying spark default config and setting up configs
+$ docker ps
+```
+
+至此， Docker 集群将会启动并运行。 Demo 集群提供了下列服务：
+
+ * HDFS 服务（ NameNode, DataNode ）
+ * Spark Master 和 Worker
+ * Hive 服务（ Metastore, HiveServer2 以及 PostgresDB ）
+ * Kafka Broker 和一个 Zookeeper Node （ Kafka 将被用来当做 Demo 的上游数据源 ）
+ * 用来运行 Hudi/Hive CLI 命令的 Adhoc 容器
+
+## Demo
+
+Stock Tracker 数据将用来展示不同的 Hudi 视图以及压缩带来的影响。
+
+看一下 `docker/demo/data` 目录。那里有 2 批股票数据——都是 1 分钟粒度的。
+第 1 批数据包含一些股票代码在交易窗口（9:30 a.m 至 10:30 a.m）的第一个小时里的行情数据数据。第 2 批包含接下来 30 分钟（10:30 - 11 a.m）的交易数据。 Hudi 将被用来将两个批次的数据采集到一个数据集中，这个数据集将会包含最新的小时级股票行情数据。
+两个批次被有意地按窗口切分，这样在第 2 批数据中包含了一些针对第 1 批数据条目的更新数据。
+
+### Step 1 : 将第 1 批数据发布到 Kafka
+
+将第 1 批数据上传到 Kafka 的 Topic “stock ticks” 中 `cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks -P`
+
+为了检查新的 Topic 是否出现，使用
+```java
+kafkacat -b kafkabroker -L -J | jq .
+{
+ "originating_broker": {
+ "id": 1001,
+ "name": "kafkabroker:9092/1001"
+ },
+ "query": {
+ "topic": "*"
+ },
+ "brokers": [
+ {
+ "id": 1001,
+ "name": "kafkabroker:9092"
+ }
+ ],
+ "topics": [
+ {
+ "topic": "stock_ticks",
+ "partitions": [
+ {
+ "partition": 0,
+ "leader": 1001,
+ "replicas": [
+ {
+ "id": 1001
+ }
+ ],
+ "isrs": [
+ {
+ "id": 1001
+ }
+ ]
+ }
+ ]
+ }
+ ]
+}
+
+```
+
+### Step 2: 从 Kafka Topic 中增量采集数据
+
+Hudi 自带一个名为 DeltaStreamer 的工具。 这个工具能连接多种数据源（包括 Kafka），以便拉取变更，并通过 upsert/insert 操作应用到 Hudi 数据集。此处，我们将使用这个工具从 Kafka Topic 下载 JSON 数据，并采集到前面步骤中初始化的 COW 和 MOR 表中。如果数据集不存在，这个工具将自动初始化数据集到文件系统中。
+
+```java
+docker exec -it adhoc-2 /bin/bash
+
+# Run the following spark-submit command to execute the delta-streamer and ingest to stock_ticks_cow dataset in HDFS
+spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+
+
+# Run the following spark-submit command to execute the delta-streamer and ingest to stock_ticks_mor dataset in HDFS
+spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE --storage-type MERGE_ON_READ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts --target-base-path /user/hive/warehouse/stock_ticks_mor --target-table stock_ticks_mor --props /var/demo/config/kafka-source.properties --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider --disable-compaction
+
+
+# As part of the setup (Look at setup_demo.sh), the configs needed for DeltaStreamer is uploaded to HDFS. The configs
+# contain mostly Kafa connectivity settings, the avro-schema to be used for ingesting along with key and partitioning fields.
+
+exit
+```
+
+你可以使用 HDFS 的 Web 浏览器来查看数据集
+`http://namenode:50070/explorer#/user/hive/warehouse/stock_ticks_cow`.
+
+你可以浏览在数据集中新创建的分区文件夹，同时还有一个在 .hoodie 目录下的 deltacommit 文件。
+
+在 MOR 数据集中也有类似的设置
+`http://namenode:50070/explorer#/user/hive/warehouse/stock_ticks_mor`
+
+
+### Step 3: 与 Hive 同步
+
+到了这一步，数据集在 HDFS 中可用。我们需要与 Hive 同步来创建新 Hive 表并添加分区，以便在那些数据集上执行 Hive 查询。
+
+```java
+docker exec -it adhoc-2 /bin/bash
+
+# THis command takes in HIveServer URL and COW Hudi Dataset location in HDFS and sync the HDFS state to Hive
+/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh --jdbc-url jdbc:hive2://hiveserver:10000 --user hive --pass hive --partitioned-by dt --base-path /user/hive/warehouse/stock_ticks_cow --database default --table stock_ticks_cow
+.....
+2018-09-24 22:22:45,568 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(112)) - Sync complete for stock_ticks_cow
+.....
+
+# Now run hive-sync for the second data-set in HDFS using Merge-On-Read (MOR storage)
+/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh --jdbc-url jdbc:hive2://hiveserver:10000 --user hive --pass hive --partitioned-by dt --base-path /user/hive/warehouse/stock_ticks_mor --database default --table stock_ticks_mor
+...
+2018-09-24 22:23:09,171 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(112)) - Sync complete for stock_ticks_mor
+...
+2018-09-24 22:23:09,559 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(112)) - Sync complete for stock_ticks_mor_rt
+....
+exit
+```
+执行了以上命令后，你会发现：
+
+1. 一个名为 `stock_ticks_cow` 的 Hive 表被创建，它为写时复制数据集提供了读优化视图。
+2. 两个新表 `stock_ticks_mor` 和 `stock_ticks_mor_rt` 被创建用于读时合并数据集。 前者为 Hudi 数据集提供了读优化视图，而后者为数据集提供了实时视图。
+
+
+### Step 4 (a): 运行 Hive 查询
+
+执行一个 Hive 查询来为股票 GOOG 找到采集到的最新时间戳。你会注意到读优化视图（ COW 和 MOR 数据集都是如此）和实时视图（仅对 MOR 数据集）给出了相同的值 “10:29 a.m”，这是因为 Hudi 为每个批次的数据创建了一个 Parquet 文件。
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat --hiveconf hive.stats.autogather=false
+# List Tables
+0: jdbc:hive2://hiveserver:10000> show tables;
++---------------------+--+
+| tab_name |
++---------------------+--+
+| stock_ticks_cow |
+| stock_ticks_mor |
+| stock_ticks_mor_rt |
++---------------------+--+
+2 rows selected (0.801 seconds)
+0: jdbc:hive2://hiveserver:10000>
+
+
+# Look at partitions that were added
+0: jdbc:hive2://hiveserver:10000> show partitions stock_ticks_mor_rt;
++----------------+--+
+| partition |
++----------------+--+
+| dt=2018-08-31 |
++----------------+--+
+1 row selected (0.24 seconds)
+
+
+# COPY-ON-WRITE Queries:
+=========================
+
+
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+
+Now, run a projection query:
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924221953 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924221953 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against M-O-R dataset. Lets look at both
+ReadOptimized and Realtime views supported by M-O-R dataset
+
+# Run against ReadOptimized View. Notice that the latest timestamp is 10:29
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (6.326 seconds)
+
+
+# Run against Realtime View. Notice that the latest timestamp is again 10:29
+
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (1.606 seconds)
+
+
+# Run projection query against Read Optimized and Realtime tables
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+exit
+exit
+```
+
+### Step 4 (b): 执行 Spark-SQL 查询
+Hudi 支持以 Spark 作为类似 Hive 的查询引擎。这是在 Spartk-SQL 中执行与 Hive 相同的查询
+
+```java
+docker exec -it adhoc-1 /bin/bash
+$SPARK_INSTALL/bin/spark-shell --jars $HUDI_SPARK_BUNDLE --master local[2] --driver-class-path $HADOOP_CONF_DIR --conf spark.sql.hive.convertMetastoreParquet=false --deploy-mode client --driver-memory 1G --executor-memory 3G --num-executors 1 --packages com.databricks:spark-avro_2.11:4.0.0
+...
+
+Welcome to
+ ____ __
+ / __/__ ___ _____/ /__
+ _\ \/ _ \/ _ `/ __/ '_/
+ /___/ .__/\_,_/_/ /_/\_\ version 2.3.1
+ /_/
+
+Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181)
+Type in expressions to have them evaluated.
+Type :help for more information.
+
+scala>
+scala> spark.sql("show tables").show(100, false)
++--------+------------------+-----------+
+|database|tableName |isTemporary|
++--------+------------------+-----------+
+|default |stock_ticks_cow |false |
+|default |stock_ticks_mor |false |
+|default |stock_ticks_mor_rt|false |
++--------+------------------+-----------+
+
+# Copy-On-Write Table
+
+## Run max timestamp query against COW table
+
+scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
+[Stage 0:> (0 + 1) / 1]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
+SLF4J: Defaulting to no-operation (NOP) logger implementation
+SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:29:00|
++------+-------------------+
+
+## Projection Query
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG'").show(100, false)
++-------------------+------+-------------------+------+---------+--------+
+|_hoodie_commit_time|symbol|ts |volume|open |close |
++-------------------+------+-------------------+------+---------+--------+
+|20180924221953 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
+|20180924221953 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
++-------------------+------+-------------------+------+---------+--------+
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against M-O-R dataset. Lets look at both
+ReadOptimized and Realtime views supported by M-O-R dataset
+
+# Run against ReadOptimized View. Notice that the latest timestamp is 10:29
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG'").show(100, false)
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:29:00|
++------+-------------------+
+
+
+# Run against Realtime View. Notice that the latest timestamp is again 10:29
+
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:29:00|
++------+-------------------+
+
+# Run projection query against Read Optimized and Realtime tables
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG'").show(100, false)
++-------------------+------+-------------------+------+---------+--------+
+|_hoodie_commit_time|symbol|ts |volume|open |close |
++-------------------+------+-------------------+------+---------+--------+
+|20180924222155 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
+|20180924222155 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
++-------------------+------+-------------------+------+---------+--------+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
++-------------------+------+-------------------+------+---------+--------+
+|_hoodie_commit_time|symbol|ts |volume|open |close |
++-------------------+------+-------------------+------+---------+--------+
+|20180924222155 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
+|20180924222155 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
++-------------------+------+-------------------+------+---------+--------+
+
+```
+
+### Step 4 (c): 执行 Presto 查询
+
+这是 Presto 查询，它们与 Hive 和 Spark 的查询类似。目前 Hudi 的实时视图不支持 Presto 。
+
+```java
+docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
+presto> show catalogs;
+ Catalog
+-----------
+ hive
+ jmx
+ localfile
+ system
+(4 rows)
+
+Query 20190817_134851_00000_j8rcz, FINISHED, 1 node
+Splits: 19 total, 19 done (100.00%)
+0:04 [0 rows, 0B] [0 rows/s, 0B/s]
+
+presto> use hive.default;
+USE
+presto:default> show tables;
+ Table
+--------------------
+ stock_ticks_cow
+ stock_ticks_mor
+ stock_ticks_mor_rt
+(3 rows)
+
+Query 20190822_181000_00001_segyw, FINISHED, 2 nodes
+Splits: 19 total, 19 done (100.00%)
+0:05 [3 rows, 99B] [0 rows/s, 18B/s]
+
+
+# COPY-ON-WRITE Queries:
+=========================
+
+
+presto:default> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20190822_181011_00002_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:12 [197 rows, 613B] [16 rows/s, 50B/s]
+
+presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180221 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822180221 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20190822_181141_00003_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [109 rows/s, 341B/s]
+
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against M-O-R dataset.
+
+# Run against ReadOptimized View. Notice that the latest timestamp is 10:29
+presto:default> select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20190822_181158_00004_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:02 [197 rows, 613B] [110 rows/s, 343B/s]
+
+
+presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180250 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822180250 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20190822_181256_00006_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [92 rows/s, 286B/s]
+
+presto:default> exit
+```
+
+### Step 5: 将第 2 批次上传到 Kafka 并运行 DeltaStreamer 进行采集
+
+上传第 2 批次数据，并使用 DeltaStreamer 采集。由于这个批次不会引入任何新分区，因此不需要执行 Hive 同步。
+
+```java
+cat docker/demo/data/batch_2.json | kafkacat -b kafkabroker -t stock_ticks -P
+
+# Within Docker container, run the ingestion command
+docker exec -it adhoc-2 /bin/bash
+
+# Run the following spark-submit command to execute the delta-streamer and ingest to stock_ticks_cow dataset in HDFS
+spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+
+
+# Run the following spark-submit command to execute the delta-streamer and ingest to stock_ticks_mor dataset in HDFS
+spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE --storage-type MERGE_ON_READ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts --target-base-path /user/hive/warehouse/stock_ticks_mor --target-table stock_ticks_mor --props /var/demo/config/kafka-source.properties --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider --disable-compaction
+
+exit
+```
+
+使用写时复制表， DeltaStreamer 的第 2 批数据采集将导致 Parquet 文件创建一个新版本。
+参考： `http://namenode:50070/explorer#/user/hive/warehouse/stock_ticks_cow/2018/08/31`
+
+使用读时合并表, 第 2 批数据采集仅仅将数据追加到没有合并的 delta （日志） 文件中。看一下 HDFS 文件系统来了解这一点： `http://namenode:50070/explorer#/user/hive/warehouse/stock_ticks_mor/2018/08/31`
+
+### Step 6 (a): 执行 Hive 查询
+
+使用写时复制表，在每一个批次被提交采集并创建新版本的 Parquet 文件时，读优化视图会立即发现变更，这些变更被当第 2 批次的一部分。
+
+使用读时合并表，第 2 批数据采集仅仅将数据追加到没有合并的 delta （日志） 文件中。
+此时，读优化视图和实时视图将提供不同的结果。读优化视图仍会返回“10:29 am”，因为它会只会从 Parquet 文件中读取。实时视图会做即时合并并返回最新提交的数据，即“10:59 a.m”。
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat --hiveconf hive.stats.autogather=false
+
+# Copy On Write Table:
+
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+1 row selected (1.932 seconds)
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924221953 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224524 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+
+# Merge On Read Table:
+
+# Read Optimized View
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (1.6 seconds)
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+# Realtime View
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224537 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+exit
+exit
+```
+
+### Step 6 (b): 执行 Spark SQL 查询
+
+以 Spark SQL 执行类似的查询：
+
+```java
+docker exec -it adhoc-1 /bin/bash
+bash-4.4# $SPARK_INSTALL/bin/spark-shell --jars $HUDI_SPARK_BUNDLE --driver-class-path $HADOOP_CONF_DIR --conf spark.sql.hive.convertMetastoreParquet=false --deploy-mode client --driver-memory 1G --master local[2] --executor-memory 3G --num-executors 1 --packages com.databricks:spark-avro_2.11:4.0.0
+
+# Copy On Write Table:
+
+scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:59:00|
++------+-------------------+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG'").show(100, false)
+
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924221953 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224524 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+
+# Merge On Read Table:
+
+# Read Optimized View
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (1.6 seconds)
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+# Realtime View
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224537 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+exit
+exit
+```
+
+### Step 6 (c): 执行 Presto 查询
+
+在 Presto 中为读优化视图执行类似的查询：
+
+
+```java
+docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
+presto> use hive.default;
+USE
+
+# Copy On Write Table:
+
+presto:default>select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:59:00
+(1 row)
+
+Query 20190822_181530_00007_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:02 [197 rows, 613B] [125 rows/s, 389B/s]
+
+presto:default>select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180221 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822181433 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215
+(2 rows)
+
+Query 20190822_181545_00008_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [106 rows/s, 332B/s]
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+
+# Merge On Read Table:
+
+# Read Optimized View
+presto:default> select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20190822_181602_00009_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:01 [197 rows, 613B] [139 rows/s, 435B/s]
+
+presto:default>select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180250 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822180250 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20190822_181615_00010_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:01 [197 rows, 613B] [154 rows/s, 480B/s]
+
+presto:default> exit
+```
+
+
+### Step 7 : 写时复制表的增量查询
+
+使用采集的两个批次的数据，我们展示 Hudi 写时复制数据集中支持的增量查询。
+
+我们使用类似的工程查询样例：
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat --hiveconf hive.stats.autogather=false
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924064621 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924065039 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+```
+
+正如你在上面的查询中看到的，有两个提交——按时间线排列是 20180924064621 和 20180924065039 。
+当你按照这些步骤执行后，你的提交会得到不同的时间戳。将它们替换到上面时间戳的位置。
+
+为了展示增量查询的影响，我们假设有一位读者已经在第 1 批数据中一部分看到了变化。那么，为了让读者看到第 2 批数据的影响，他/她需要保留第 1 批次提交时间中的开始时间（ 20180924064621 ）并执行增量查询：
+
+Hudi 的增量模式为增量查询提供了高效的扫描，通过 Hudi 管理的元数据，过滤掉了那些不包含候选记录的文件。
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat --hiveconf hive.stats.autogather=false
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.mode=INCREMENTAL;
+No rows affected (0.009 seconds)
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.max.commits=3;
+No rows affected (0.009 seconds)
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.start.timestamp=20180924064621;
+```
+
+使用上面的设置，那些在提交 20180924065039 之后没有任何更新的文件ID将被过滤掉，不进行扫描。
+以下是增量查询：
+
+```java
+0: jdbc:hive2://hiveserver:10000>
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG' and `_hoodie_commit_time` > '20180924064621';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924065039 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+1 row selected (0.83 seconds)
+0: jdbc:hive2://hiveserver:10000>
+```
+
+### 使用 Spark SQL 做增量查询
+```java
+docker exec -it adhoc-1 /bin/bash
+bash-4.4# $SPARK_INSTALL/bin/spark-shell --jars $HUDI_SPARK_BUNDLE --driver-class-path $HADOOP_CONF_DIR --conf spark.sql.hive.convertMetastoreParquet=false --deploy-mode client --driver-memory 1G --master local[2] --executor-memory 3G --num-executors 1 --packages com.databricks:spark-avro_2.11:4.0.0
+Welcome to
+ ____ __
+ / __/__ ___ _____/ /__
+ _\ \/ _ \/ _ `/ __/ '_/
+ /___/ .__/\_,_/_/ /_/\_\ version 2.3.1
+ /_/
+
+Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181)
+Type in expressions to have them evaluated.
+Type :help for more information.
+
+scala> import org.apache.hudi.DataSourceReadOptions
+import org.apache.hudi.DataSourceReadOptions
+
+# In the below query, 20180925045257 is the first commit's timestamp
+scala> val hoodieIncViewDF = spark.read.format("org.apache.hudi").option(DataSourceReadOptions.VIEW_TYPE_OPT_KEY, DataSourceReadOptions.VIEW_TYPE_INCREMENTAL_OPT_VAL).option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY, "20180924064621").load("/user/hive/warehouse/stock_ticks_cow")
+SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
+SLF4J: Defaulting to no-operation (NOP) logger implementation
+SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
+hoodieIncViewDF: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 15 more fields]
+
+scala> hoodieIncViewDF.registerTempTable("stock_ticks_cow_incr_tmp1")
+warning: there was one deprecation warning; re-run with -deprecation for details
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow_incr_tmp1 where symbol = 'GOOG'").show(100, false);
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924065039 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+```
+
+
+### Step 8: 为读时合并数据集的调度并执行压缩
+
+我们来调度并运行一个压缩来创建一个新版本的列式文件，以便读优化读取器能看到新数据。
+再次强调，你可以使用 Hudi CLI 来人工调度并执行压缩。
+
+```java
+docker exec -it adhoc-1 /bin/bash
+root@adhoc-1:/opt# /var/hoodie/ws/hudi-cli/hudi-cli.sh
+============================================
+* *
+* _ _ _ _ *
+* | | | | | | (_) *
+* | |__| | __| | - *
+* | __ || | / _` | || *
+* | | | || || (_| | || *
+* |_| |_|\___/ \____/ || *
+* *
+============================================
+
+Welcome to Hoodie CLI. Please type help if you are looking for help.
+hudi->connect --path /user/hive/warehouse/stock_ticks_mor
+18/09/24 06:59:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+18/09/24 06:59:35 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
+18/09/24 06:59:35 INFO util.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://namenode:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1261652683_11, ugi=root (auth:SIMPLE)]]]
+18/09/24 06:59:35 INFO table.HoodieTableConfig: Loading dataset properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
+18/09/24 06:59:36 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ from /user/hive/warehouse/stock_ticks_mor
+Metadata for table stock_ticks_mor loaded
+
+# Ensure no compactions are present
+
+hoodie:stock_ticks_mor->compactions show all
+18/09/24 06:59:54 INFO timeline.HoodieActiveTimeline: Loaded instants [[20180924064636__clean__COMPLETED], [20180924064636__deltacommit__COMPLETED], [20180924065057__clean__COMPLETED], [20180924065057__deltacommit__COMPLETED]]
+ ___________________________________________________________________
+ | Compaction Instant Time| State | Total FileIds to be Compacted|
+ |==================================================================|
+
+
+
+
+# Schedule a compaction. This will use Spark Launcher to schedule compaction
+hoodie:stock_ticks_mor->compaction schedule
+....
+Compaction successfully completed for 20180924070031
+
+# Now refresh and check again. You will see that there is a new compaction requested
+
+hoodie:stock_ticks->connect --path /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:01:16 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:01:16 INFO util.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://namenode:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1261652683_11, ugi=root (auth:SIMPLE)]]]
+18/09/24 07:01:16 INFO table.HoodieTableConfig: Loading dataset properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
+18/09/24 07:01:16 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ from /user/hive/warehouse/stock_ticks_mor
+Metadata for table stock_ticks_mor loaded
+
+
+
+hoodie:stock_ticks_mor->compactions show all
+18/09/24 06:34:12 INFO timeline.HoodieActiveTimeline: Loaded instants [[20180924041125__clean__COMPLETED], [20180924041125__deltacommit__COMPLETED], [20180924042735__clean__COMPLETED], [20180924042735__deltacommit__COMPLETED], [==>20180924063245__compaction__REQUESTED]]
+ ___________________________________________________________________
+ | Compaction Instant Time| State | Total FileIds to be Compacted|
+ |==================================================================|
+ | 20180924070031 | REQUESTED| 1 |
+
+
+
+
+# Execute the compaction. The compaction instant value passed below must be the one displayed in the above "compactions show all" query
+hoodie:stock_ticks_mor->compaction run --compactionInstant 20180924070031 --parallelism 2 --sparkMemory 1G --schemaFilePath /var/demo/config/schema.avsc --retry 1
+....
+Compaction successfully completed for 20180924070031
+
+
+## Now check if compaction is completed
+
+hoodie:stock_ticks_mor->connect --path /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:03:00 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:03:00 INFO util.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://namenode:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1261652683_11, ugi=root (auth:SIMPLE)]]]
+18/09/24 07:03:00 INFO table.HoodieTableConfig: Loading dataset properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
+18/09/24 07:03:00 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ from /user/hive/warehouse/stock_ticks_mor
+Metadata for table stock_ticks_mor loaded
+
+
+
+hoodie:stock_ticks->compactions show all
+18/09/24 07:03:15 INFO timeline.HoodieActiveTimeline: Loaded instants [[20180924064636__clean__COMPLETED], [20180924064636__deltacommit__COMPLETED], [20180924065057__clean__COMPLETED], [20180924065057__deltacommit__COMPLETED], [20180924070031__commit__COMPLETED]]
+ ___________________________________________________________________
+ | Compaction Instant Time| State | Total FileIds to be Compacted|
+ |==================================================================|
+ | 20180924070031 | COMPLETED| 1 |
+
+```
+
+### Step 9: 执行包含增量查询的 Hive 查询
+
+你将看到读优化视图和实时视图都会展示最新提交的数据。
+让我们也对 MOR 表执行增量查询。
+通过查看下方的查询输出，能够明确 MOR 表的第一次提交时间是 20180924064636 而第二次提交时间是 20180924070031 。
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat --hiveconf hive.stats.autogather=false
+
+# Read Optimized View
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+1 row selected (1.6 seconds)
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924064636 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+# Realtime View
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924064636 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+# Incremental View:
+
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.mode=INCREMENTAL;
+No rows affected (0.008 seconds)
+# Max-Commits covers both second batch and compaction commit
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.max.commits=3;
+No rows affected (0.007 seconds)
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.start.timestamp=20180924064636;
+No rows affected (0.013 seconds)
+# Query:
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG' and `_hoodie_commit_time` > '20180924064636';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+exit
+exit
+```
+
+### Step 10: 压缩后在 MOR 的读优化视图与实时视图上使用 Spark-SQL
+
+```java
+docker exec -it adhoc-1 /bin/bash
+bash-4.4# $SPARK_INSTALL/bin/spark-shell --jars $HUDI_SPARK_BUNDLE --driver-class-path $HADOOP_CONF_DIR --conf spark.sql.hive.convertMetastoreParquet=false --deploy-mode client --driver-memory 1G --master local[2] --executor-memory 3G --num-executors 1 --packages com.databricks:spark-avro_2.11:4.0.0
+
+# Read Optimized View
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+1 row selected (1.6 seconds)
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924064636 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+# Realtime View
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924064636 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+```
+
+### Step 11: 压缩后在 MOR 数据集的读优化视图上进行 Presto 查询
+
+```java
+docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
+presto> use hive.default;
+USE
+
+# Read Optimized View
+resto:default> select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:59:00
+(1 row)
+
+Query 20190822_182319_00011_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:01 [197 rows, 613B] [133 rows/s, 414B/s]
+
+presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180250 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822181944 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215
+(2 rows)
+
+Query 20190822_182333_00012_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [98 rows/s, 307B/s]
+
+presto:default>
+
+```
+
+
+Demo 到此结束。
+
+## 在本地 Docker 环境中测试 Hudi
+
+你可以组建一个包含 Hadoop 、 Hive 和 Spark 服务的 Hadoop Docker 环境，并支持 Hudi 。
+```java
+$ mvn pre-integration-test -DskipTests
+```
+上面的命令为所有的服务构建了 Docker 镜像，它带有当前安装在 /var/hoodie/ws 的 Hudi 源，并使用一个部署文件引入了这些服务。我们当前在 Docker 镜像中使用 Hadoop （v2.8.4）、 Hive （v2.3.3）和 Spark （v2.3.1）。
+
+要销毁容器：
+```java
+$ cd hudi-integ-test
+$ mvn docker-compose:down
+```
+
+如果你想要组建 Docker 容器，使用：
+```java
+$ cd hudi-integ-test
+$ mvn docker-compose:up -DdetachedMode=true
+```
+
+Hudi 是一个在包含 Hadoop 、 Hive 和 Spark 的海量数据分析/采集环境中使用的库。与这些系统的互用性是我们的一个关键目标。 我们在积极地向 __hudi-integ-test/src/test/java__ 添加集成测试，这些测试利用了这个 Docker 环境（参考： __hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieSanity.java__ ）。
+
+
+### 构建本地 Docker 容器:
+
+Demo 和执行集成测试所需要的 Docker 镜像已经在 Docker 源中。 Docker 镜像和部署脚本经过了谨慎的实现以便服务与多种目的：
+
+1. Docker 镜像有内建的 Hudi jar 包，它包含一些指向其他 jar 包的环境变量（ HUDI_HADOOP_BUNDLE 等）
+2. 为了执行集成测试，我们需要使用本地生成的 jar 包在 Docker 中运行服务。 Docker 部署脚本（参考 `docker/compose/docker-compose_hadoop284_hive233_spark231.yml`）能确保本地 jar 包通过挂载 Docker 地址上挂载本地 Hudi 工作空间，从而覆盖了内建的 jar 包。
+3. 当这些 Docker 容器挂载到本地 Hudi 工作空间之后，任何发生在工作空间中的变更将会自动反映到容器中。这对于开发者来说是一种开发和验证 Hudi 的简便方法，这些开发者没有分布式的环境。要注意的是，这是集成测试的执行方式。
+
+这避免了维护分离的 Docker 镜像，也避免了本地构建 Docker 镜像的各个步骤的消耗。
+但是如果用户想要在有更低网络带宽的地方测试 Hudi ，他们仍可以构建本地镜像。
+在执行 `docker/setup_demo.sh` 之前执行脚本 `docker/build_local_docker_images.sh` 来构建本地 Docker 镜像。
+
+以下是执行的命令:
+
+```java
+cd docker
+./build_local_docker_images.sh
+.....
+
+[INFO] Reactor Summary:
+[INFO]
+[INFO] hoodie ............................................. SUCCESS [ 1.709 s]
+[INFO] hudi-common ...................................... SUCCESS [ 9.015 s]
+[INFO] hudi-hadoop-mr ................................... SUCCESS [ 1.108 s]
+[INFO] hudi-client ...................................... SUCCESS [ 4.409 s]
+[INFO] hudi-hive ........................................ SUCCESS [ 0.976 s]
+[INFO] hudi-spark ....................................... SUCCESS [ 26.522 s]
+[INFO] hudi-utilities ................................... SUCCESS [ 16.256 s]
+[INFO] hudi-cli ......................................... SUCCESS [ 11.341 s]
+[INFO] hudi-hadoop-mr-bundle ............................ SUCCESS [ 1.893 s]
+[INFO] hudi-hive-bundle ................................. SUCCESS [ 14.099 s]
+[INFO] hudi-spark-bundle ................................ SUCCESS [ 58.252 s]
+[INFO] hudi-hadoop-docker ............................... SUCCESS [ 0.612 s]
+[INFO] hudi-hadoop-base-docker .......................... SUCCESS [04:04 min]
+[INFO] hudi-hadoop-namenode-docker ...................... SUCCESS [ 6.142 s]
+[INFO] hudi-hadoop-datanode-docker ...................... SUCCESS [ 7.763 s]
+[INFO] hudi-hadoop-history-docker ....................... SUCCESS [ 5.922 s]
+[INFO] hudi-hadoop-hive-docker .......................... SUCCESS [ 56.152 s]
+[INFO] hudi-hadoop-sparkbase-docker ..................... SUCCESS [01:18 min]
+[INFO] hudi-hadoop-sparkmaster-docker ................... SUCCESS [ 2.964 s]
+[INFO] hudi-hadoop-sparkworker-docker ................... SUCCESS [ 3.032 s]
+[INFO] hudi-hadoop-sparkadhoc-docker .................... SUCCESS [ 2.764 s]
+[INFO] hudi-integ-test .................................. SUCCESS [ 1.785 s]
+[INFO] ------------------------------------------------------------------------
+[INFO] BUILD SUCCESS
+[INFO] ------------------------------------------------------------------------
+[INFO] Total time: 09:15 min
+[INFO] Finished at: 2018-09-10T17:47:37-07:00
+[INFO] Final Memory: 236M/1848M
+[INFO] ------------------------------------------------------------------------
+```
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/docs-versions.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/docs-versions.md
new file mode 100644
index 00000000000..942403ea2fb
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/docs-versions.md
@@ -0,0 +1,12 @@
+---
+title: 文档版本
+keywords: [ hudi, privacy]
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+<table className="docversions">
+ <tbody>
+ </tbody>
+</table>
+
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/flink-quick-start-guide.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/flink-quick-start-guide.md
new file mode 100644
index 00000000000..0a811ed8105
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/flink-quick-start-guide.md
@@ -0,0 +1,530 @@
+---
+title: "Flink 指南"
+toc: true
+last_modified_at: 2021-08-16T12:31:57-04:00
+language: cn
+---
+
+本指南提供了使用 Flink SQL 操作 Hudi 的文档。阅读本指南，您可以学习如何快速开始使用 Flink 读写 Hudi，同时对配置和任务优化有更深入的了解：
+
+- **快速开始** ：通过阅读 [快速开始](#快速开始)，你可以快速开始使用 Flink sql client 去读写 Hudi
+- **配置** ：对于 [Flink 配置](#flink-配置)，使用 `$FLINK_HOME/conf/flink-conf.yaml` 来配置。 对于任意一个作业的配置，通过[表参数](#表参数)来设置
+- **写功能** ：Flink 支持多种写功能用例，例如 [离线批量导入](#离线批量导入)，[全量接增量](#全量接增量)，[Changelog 模式](#changelog-模式)，[Insert 模式](#insert-模式) 和 [离线 Compaction](#离线-compaction)
+- **查询功能** ：Flink 支持多种查询功能用例，例如 [Hive 查询](#hive-查询)， [Presto 查询](#presto-查询)
+- **优化** ：针对 Flink 读写 Hudi 的操作，本指南提供了一些优化建议，例如 [内存优化](#内存优化) 和 [写入限流](#写入限流)
+
+## 快速开始
+
+### 安装
+
+我们推荐使用 [Flink Sql Client](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sqlclient/) 来读写 Hudi，因为 Flink
+sql client 对于 SQL 用户来说更容易上手。
+
+#### 步骤1 下载 Flink jar
+我们推荐使用 Flink-1.12.x 来读写 Hudi。 你可以按照 [Flink 安装文档](https://flink.apache.org/downloads) 的指导来安装 Flink。 `hudi-flink-bundle.jar`
+使用的是 scala 2.11，所以我们推荐 Flink-1.12.x 配合 scala 2.11 来使用。
+
+#### 步骤2 启动 Flink 集群
+在 Hadoop 环境下启动 standalone 的 Flink 集群。
+在你启动 Flink 集群前，我们推荐先配置如下参数：
+
+- 在 `$FLINK_HOME/conf/flink-conf.yaml` 中添加配置：`taskmanager.numberOfTaskSlots: 4`
+- 在 `$FLINK_HOME/conf/flink-conf.yaml` 中，根据数据量大小和集群大小来添加其他的 [Flink 配置](#flink-配置)
+- 在 `$FLINK_HOME/conf/workers` 中添加4核 `localhost` 来保证我们本地集群中有4个 workers
+
+启动集群：
+
+```bash
+# HADOOP_HOME 是 Hadoop 的根目录。
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+
+# 启动 Flink standalone 集群
+./bin/start-cluster.sh
+```
+#### 步骤3 启动 Flink SQL client
+
+Hudi 将 Flink 板块单独打包为 `hudi-flink-bundle.jar`，该 Jar 包需要在启动的时候加载。
+你可以在 `hudi-source-dir/packaging/hudi-flink-bundle` 下手动的打包这个 Jar 包，或者从 [Apache Official Repository](https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink-bundle_2.11/)
+中下载。
+
+启动 Flink SQL Client：
+
+```bash
+# HADOOP_HOME 是 Hadoop 的根目录。
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+
+./bin/sql-client.sh embedded -j .../hudi-flink-bundle_2.1?-*.*.*.jar shell
+```
+
+<div className="notice--info">
+ <h4>小提示：</h4>
+<ul>
+ <li>为了兼容大部分的对象存储，我们推荐使用 Hadoop 2.9 x+的 Hadoop 版本</li>
+ <li>flink-parquet 和 flink-avro 格式已经打包在 hudi-flink-bundle.jar 中了</li>
+</ul>
+</div>
+
+根据下面的不同功能来设置表名，存储路径和操作类型。
+Flink SQL Client 是逐行执行 SQL 的。
+
+### 插入数据
+
+先创建一个 Flink Hudi 表，然后在通过下面的 `VALUES` 语句往该表中插入数据。
+
+```sql
+-- 为了更直观的显示结果，推荐把 CLI 的输出结果模式设置为 tableau。
+set execution.result-mode=tableau;
+
+CREATE TABLE t1(
+ uuid VARCHAR(20),
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'schema://base-path',
+ 'table.type' = 'MERGE_ON_READ' -- 创建一个 MERGE_ON_READ类型的表，默认是 COPY_ON_ERITE
+);
+
+-- 使用 values 语句插入数据
+INSERT INTO t1 VALUES
+ ('id1','Danny',23,TIMESTAMP '1970-01-01 00:00:01','par1'),
+ ('id2','Stephen',33,TIMESTAMP '1970-01-01 00:00:02','par1'),
+ ('id3','Julian',53,TIMESTAMP '1970-01-01 00:00:03','par2'),
+ ('id4','Fabian',31,TIMESTAMP '1970-01-01 00:00:04','par2'),
+ ('id5','Sophia',18,TIMESTAMP '1970-01-01 00:00:05','par3'),
+ ('id6','Emma',20,TIMESTAMP '1970-01-01 00:00:06','par3'),
+ ('id7','Bob',44,TIMESTAMP '1970-01-01 00:00:07','par4'),
+ ('id8','Han',56,TIMESTAMP '1970-01-01 00:00:08','par4');
+```
+
+### 查询数据
+
+```sql
+-- 从 Hudi 表中查询
+select * from t1;
+```
+该查询语句提供的是 快照读（Snapshot Querying）。
+如果想了解更多关于表类型和查询类型的介绍，可以参考文档 [表类型和查询类型](/docs/concepts#table-types--queries)
+
+### 更新数据
+
+数据的更新和插入数据类似：
+
+```sql
+-- 这条语句会更新 key 为 'id1' 的记录
+insert into t1 values
+ ('id1','Danny',27,TIMESTAMP '1970-01-01 00:00:01','par1');
+```
+
+需要注意的是：现在使用的存储类型为 `Append`。通常我们都是使用 apennd 模式，除非你是第一次创建这个表。 再次 [查询数据](#查询数据) 就会显示更新后的结果。
+每一次的插入操作都会在时间轴上生成一个带时间戳的新的 [commit](/docs/concepts)，在元数据字段 `_hoodie_commit_time` 和同一 `_hoodie_record_key` 的 `age`字段中查看更新。
+
+### 流式查询
+
+Hudi Flink 也有能力来查询从指定时间戳开始的流式记录集合。 该功能可以使用Hudi的流式查询，只需要提供一个查询开始的时间戳就可以完成查询。如果我们需要的
+是指定时间戳后的所有数据，我们就不需要指定结束时间。
+
+```sql
+CREATE TABLE t1(
+ uuid VARCHAR(20),
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'oss://vvr-daily/hudi/t1',
+ 'table.type' = 'MERGE_ON_READ',
+ 'read.streaming.enabled' = 'true', -- 该参数开启流式查询
+ 'read.streaming.start-commit' = '20210316134557' -- 指定开始的时间戳
+ 'read.streaming.check-interval' = '4' -- 指定检查新的commit的周期，默认是60秒
+);
+
+-- 开启流式的查询
+select * from t1;
+```
+上述的查询会查询出 `read.streaming.start-commit` 时间戳后的所有数据。该功能的特殊在于可以同时在流和批的 pipeline 上执行。
+
+### 删除数据 {#deletes}
+
+在 [流式查询](#流式查询) 中使用数据时，Hudi Flink 源还可以接受来自底层数据源的更改日志，然后可以按行级别应用更新和删除。所以，你可以在 Hudi 上同步各种 RDBMS 的近实时快照。
+
+## Flink 配置
+
+在使用 Flink 前，你需要在 `$FLINK_HOME/conf/flink-conf.yaml` 中设置一些全局的 Flink 配置。
+
+### 并行度
+
+| 名称 | 默认值 | 类型 | 描述 |
+| ----------- | ------- | ------- | ------- |
+| `taskmanager.numberOfTaskSlots` | `1` | `Integer` | 单个 TaskManager 可以运行的并行 task 数。我们建议将该值设置为 > 4，实际值需根据数据量进行设置 |
+| `parallelism.default` | `1` | `Integer` | 当用户为指定算子并行度时，会使用这个并行度（默认值是1）。例如 [`write.bucket_assign.tasks`](#并行度-1) 没有设置，就会使用这个默认值 |
+
+### 内存
+
+| 名称 | 默认值 | 类型 | 描述 |
+| ----------- | ------- | ------- | ------- |
+| `jobmanager.memory.process.size` | `(none)` | `MemorySize` | JobManager 的总进程内存大小。代表 JobManager JVM 进程消耗的所有内存，包括总的 Flink 内存、JVM 元空间和 JVM 开销 |
+| `taskmanager.memory.task.heap.size` | `(none)` | `MemorySize` | TaskExecutor 的堆内存大小。这是为写缓存保留的 JVM 堆内存大小 |
+| `taskmanager.memory.managed.size` | `(none)` | `MemorySize` | 这是内存管理器管理的堆外内存的大小，用于排序和 RocksDB 状态后端。如果选择 RocksDB 作为状态后端，则需要设置此内存 |
+
+### Checkpoint
+
+| 名称 | 默认值 | 类型 | 描述 |
+| ----------- | ------- | ------- | ------- |
+| `execution.checkpointing.interval` | `(none)` | `Duration` | 设置该值的方式为 `execution.checkpointing.interval = 150000ms`，其中150000ms = 2.5min。 设置这个参数等同于开启了 Checkpoint |
+| `state.backend` | `(none)` | `String` | 保存状态信息的状态后端。 我们推荐设置状态后端为 `rocksdb` ：`state.backend: rocksdb` |
+| `state.backend.rocksdb.localdir` | `(none)` | `String` | RocksDB 存储状态信息的路径 |
+| `state.checkpoints.dir` | `(none)` | `String` | Checkpoint 的默认路径，用于在支持 Flink 的文件系统中存储检查点的数据文件和元数据。存储路径必须可从所有参与进程/节点（即所有 TaskManager 和 JobManager）访问，如 HDFS 和 OSS 路径 |
+| `state.backend.incremental` | `false` | `Boolean` | 选项状态后端是否创建增量检查点。对于增量检查点，只存储与前一个检查点的差异，而不是完整的检查点状态。如果存储状态设置为 `rocksdb`，建议打开这个选项 |
+
+## 表参数
+
+对于单个作业的配置，我们可以在 Flink SQL 语句的 [`WITH`](#表参数)中设置。
+所以，作业级别的配置如下：
+
+### 内存
+
+:::note
+我们在内存调优的时候需要先关注 TaskManager 的数量和 [内存](#内存) 配置，以及 write task 的并发（`write.tasks: 4` 的值），确认每个 write task 能够分配到足够的内存，
+再考虑以下相关的内存参数设置。
+:::
+
+| 名称 | 说明 | 默认值 | 备注 |
+| ----------- | ------- | ------- | ------- |
+| `write.task.max.size` | 一个 write task 的最大可用内存。 默认为 `1024MB` | `1024D` | 当前预留给 write buffer 的内存为 `write.task.max.size` - `compaction.max_memory`。 当 write task 的内存 buffer 达到该阈值后会将内存里最大的 buffer flush |
+| `write.batch.size` | Flink 的 write task 为了提高写数据效率，会按照写 bucket 提前缓存数据，每个 bucket 的数据在内存达到阈值之前会一直缓存在内存中，当阈值达到会把数据 buffer 传递给 Hudi 的 writer 执行写操作。默认为 `64MB` | `64D` | 推荐使用默认值 |
+| `write.log_block.size` | Hudi 的 log writer 在收到 write task 的数据后不会马上 flush 数据，writer 是以 LogBlock 为单位往磁盘刷数据的，在 LogBlock 攒够之前 records 会以序列化字节的形式缓存在 writer 内部。默认为 `128MB` | `128` | 推荐使用默认值 |
+| `write.merge.max_memory` | 当表的类型为 `COPY_ON_WRITE`，Hudi 会合并增量数据和 base file 中的数据。增量的数据会缓存在内存的 map 结构里，这个 map 是可溢写的，这个参数控制了 map 可以使用的堆内存大小。 默认为 `100MB` | `100` | 推荐使用默认值 |
+| `compaction.max_memory` | 同 `write.merge.max_memory` 类似，只是发生在压缩时。默认为 `100MB` | `100` | 如果是在线 compaction，资源充足时可以开大些，比如 `1024MB` |
+
+### 并行度
+
+| 名称 | 说明 | 默认值 | 备注 |
+| ----------- | ------- | ------- | ------- |
+| `write.tasks` | writer 的并发，每个 writer 顺序写 `1`~`N` 个 buckets。默认为 `4` | `4` | 增加这个并发，对小文件个数没影响 |
+| `write.bucket_assign.tasks` | bucket assigner 的并发，默认使用 Flink 默认并发 `parallelism.default` | [`parallelism.default`](#并行度) | 增加并发同时增加了并发写的 bucekt 数，也就变相增加了小文件(小 bucket)数 |
+| `write.index_boostrap.tasks` | Index bootstrap 算子的并发，增加并发可以加快 bootstrap 阶段的效率，bootstrap 阶段会阻塞 Checkpoint，因此需要设置多一些的 Checkpoint 失败容忍次数。 默认使用 Flink 默认并发 `parallelism.default` | [`parallelism.default`](#并行度) | 只在 `index.bootsrap.enabled` 为 `true` 时生效 |
+| `read.tasks` | 读算子的并发（batch 和 stream。默认为 `4` | `4` | |
+| `compaction.tasks` | 在线 compaction 的并发。默认为 `10` | `10` | 在线 compaction 比较耗费资源，建议使用 [`离线 compaction`](#离线-compaction) |
+
+### Compaction
+
+:::note
+这些参数都只服务于在线 compaction。
+:::
+
+:::note
+通过设置`compaction.async.enabled` = `false` 来关闭在线 compaction，但是 `compaction.schedule.enable` 仍然建议开启。之后通过[`离线 compaction`](#离线-compaction) 直接执行在线 compaction 产生的 compaction plan。
+:::
+
+| 名称 | 说明 | 默认值 | 备注 |
+| ----------- | ------- | ------- | ------- |
+| `compaction.schedule.enabled` | 是否阶段性生成 compaction plan | `true` | 建议开启，即使 `compaction.async.enabled` = `false` |
+| `compaction.async.enabled` | 是否开启异步压缩，MOR 表时默认开启 | `true` | 通过关闭此参数来关闭 `在线 compaction` |
+| `compaction.trigger.strategy` | 压缩策略 | `num_commits` | 可选择的策略有 `num_commits`：达到 `N` 个 delta commits 时触发 compaction; `time_elapsed`：距离上次 compaction 超过 `N` 秒触发 compaction ; `num_and_time`：`NUM_COMMITS` 和 `TIME_ELAPSED` 同时满足; `num_or_time`：`NUM_COMMITS` 或者 `TIME_ELAPSED` 中一个满足 |
+| `compaction.delta_commits` | 默认策略，`5` 个 delta commits 触发一次压缩 | `5` | -- |
+| `compaction.delta_seconds` | 默认 `1` 小时触发一次压缩 | `3600` | -- |
+| `compaction.max_memory` | compaction 的 hashMap 可用内存。 默认值为 `100MB` | `100` | 资源够用的话，建议调整到 `1024MB` |
+| `compaction.target_io` | 每个 compaction plan 的 IO 内存上限 (读和写)。 默认值为 `5GB`| `5120` | 离线 compaction 的默认值是 `500GB` |
+
+## 内存优化
+
+### MOR
+
+1. [把 Flink 的状态后端设置为 `rocksdb`](#checkpoint) (默认的 `in memory` 状态后端非常的消耗内存）
+2. 如果内存足够，`compaction.max_memory` 可以设置得更大些（默认为 `100MB`，可以调大到 `1024MB`）
+3. 关注 taskManager 分配给每个 write task 的内存，保证每个 write task 能够分配到 `write.task.max.size` 所配置的内存大小。 比如 taskManager 的内存是 `4GB`，
+运行了 `2` 个 `StreamWriteFunction`，那每个 write function 能分到 `2GB`，尽量预留一些缓存。因为网络缓存，taskManager 上其他类型的 task (比如 `BucketAssignFunction`)也会消耗一些内存
+4. 需要关注 compaction 的内存变化。 `compaction.max_memory` 控制了每个 compaction task 读 log 时可以利用的内存大小。`compaction.tasks` 控制了 compaction task 的并发
+
+### COW
+
+1. [把 Flink 的状态后端设置为 `rocksdb`](#checkpoint) (默认的 `in memory` 状态后端非常的消耗内存）
+2. 同时调大 `write.task.max.size` 和 `write.merge.max_memory` （默认值分别是 `1024MB` 和 `100MB`，可以调整为 `2014MB` 和 `1024MB`）
+3. 关注 taskManager 分配给每个 write task 的内存，保证每个 write task 能够分配到 `write.task.max.size` 所配置的内存大小。 比如 taskManager 的内存是 `4GB`，
+ 运行了 `2` 个 `StreamWriteFunction`，那每个 write function 能分到 `2GB`，尽量预留一些缓存。因为网络缓存，taskManager 上其他类型的 task （比如 `BucketAssignFunction`）也会消耗一些内存
+
+## 离线批量导入
+
+针对存量数据导入的需求，如果存量数据来源于其他数据源，可以使用离线批量导入功能（`bulk_insert`），快速将存量数据导入 Hudi。
+
+
+:::note
+`bulk_insert` 省去了 avro 的序列化以及数据的 merge 过程，后续也不会再有去重操作。所以，数据的唯一性需要自己来保证。
+:::
+
+:::note
+`bulk_insert` 在 `batch execution mode` 模式下执行更加高效。 `batch execution mode` 模式默认会按照 partition path 排序输入消息再写入 Hudi，
+避免 file handle 频繁切换导致性能下降。
+:::
+
+:::note
+`bulk_insert` 的 write tasks 的并发是通过参数 `write.tasks` 来指定，并发的数量会影响到小文件的数量，理论上，`bulk_insert` 的 write tasks 的并发数就是划分的 bucket 数，
+当然每个 bucket 在写到文件大小上限（parquet 120 MB）的时候会回滚到新的文件句柄，所以最后：写文件数量 >= [`write.bucket_assign.tasks`](#并行度)。
+:::
+
+### 参数
+
+| 名称 | Required | 默认值 | 备注 |
+| ----------- | ------- | ------- | ------- |
+| `write.operation` | `true` | `upsert` | 开启 `bulk_insert` 功能 |
+| `write.tasks` | `false` | `4` | `bulk_insert` write tasks 的并发，最后的文件数 >= [`write.bucket_assign.tasks`](#并行度) |
+| `write.bulk_insert.shuffle_by_partition` | `false` | `true` | 是否将数据按照 partition 字段 shuffle 后，再通过 write task 写入，开启该参数将减少小文件的数量，但是有数据倾斜的风险 |
+| `write.bulk_insert.sort_by_partition` | `false` | `true` | 是否将数据线按照 partition 字段排序后，再通过 write task 写入，当一个 write task 写多个 partition时，开启可以减少小文件数量 |
+| `write.sort.memory` | `false` | `128` | sort 算子的可用 managed memory（单位 MB）。默认为 `128` MB |
+
+## 全量接增量
+
+针对全量数据导入后，接增量的需求。如果已经有全量的离线 Hudi 表，需要接上实时写入，并且保证数据不重复，可以开启 全量接增量（`index bootstrap`）功能。
+
+:::note
+如果觉得流程冗长，可以在写入全量数据的时候资源调大直接走流模式写，全量走完接新数据再将资源调小（或者开启 [写入限流](#写入限流) ）。
+:::
+
+### 参数
+
+| 名称 | Required | 默认值 | 备注 |
+| ----------- | ------- | ------- | ------- |
+| `index.bootstrap.enabled` | `true` | `false` | 开启 index bootstrap 索引加载后，会将已存在的 Hudi 表的数据一次性加载到 state 中 |
+| `index.partition.regex` | `false` | `*` | 设置正则表达式进行分区筛选，默认为加载全部分区 |
+
+### 使用流程
+
+1. `CREATE TABLE` 创建和 Hudi 表对应的语句，注意 `table.type` 必须正确
+2. 设置 `index.bootstrap.enabled` = `true` 开启索引加载功能
+3. 在 `flink-conf.yaml` 中设置 Checkpoint 失败容忍 ：`execution.checkpointing.tolerable-failed-checkpoints = n`（取决于checkpoint 调度次数）
+4. 等待第一次 Checkpoint 完成，表示索引加载完成
+5. 索引加载完成后可以退出并保存 savepoint（也可以直接用 externalized checkpoint）
+6. 重启任务，将 `index.bootstrap.enable` 设置为 `false`，参数配置到合适的大小
+
+:::note
+1. 索引加载是阻塞式，所以在索引加载过程中 Checkpoint 无法完成
+2. 索引加载由数据流触发，需要确保每个 partition 都至少有1条数据，即上游 source 有数据进来
+3. 索引加载为并发加载，根据数据量大小加载时间不同，可以在log中搜索 `finish loading the index under partition` 和 `Load record form file` 日志内容来观察索引加载的进度
+4. 第一次 Checkpoint 成功就表示索引已经加载完成，后续从 Checkpoint 恢复时无需再次加载索引
+:::
+
+## Changelog 模式
+
+针对使用 Hudi 保留消息的所有变更（I / -U / U / D），之后接上 Flink 引擎的有状态计算实现全链路近实时数仓生产（增量计算）的需求，Hudi 的 MOR 表
+通过行存原生支持保留消息的所有变更（format 层面的集成），通过流读 MOR 表可以消费到所有的变更记录。
+
+### 参数
+
+| 名称 | Required | 默认值 | 备注 |
+| ----------- | ------- | ------- | ------- |
+| `changelog.enabled` | `false` | `false` | 默认是关闭状态，即 `UPSERT` 语义，所有的消息仅保证最后一条合并消息，中间的变更可能会被 merge 掉；改成 `true` 支持消费所有变更 |
+
+:::note
+批（快照）读仍然会合并所有的中间结果，不管 format 是否已存储中间状态。
+:::
+
+:::note
+设置 `changelog.enable` 为 `true` 后，中间的变更也只是 `best effort`：异步的压缩任务会将中间变更合并成 `1` 条，所以如果流读消费不够及时，被压缩后
+只能读到最后一条记录。当然，通过调整压缩的缓存时间可以预留一定的时间缓冲给 reader，比如调整压缩的两个参数：[`compaction.delta_commits`](#compaction) and [`compaction.delta_seconds`](#compaction)。
+:::
+
+
+## Insert 模式
+
+当前 Hudi 对于 `Insert 模式` 默认会采用小文件策略：MOR 会追加写 avro log 文件，COW 会不断合并之前的 parquet 文件（并且增量的数据会去重），这样会导致性能下降。
+
+
+如果想关闭文件合并，可以设置 `write.insert.deduplicate` 为 `false`。 关闭后，不会有任何的去重行为，每次 flush 都是直接写独立的 parquet（MOR 表也会直接写 parquet）。
+
+### 参数
+| 名称 | Required | 默认值 | 备注 |
+| ----------- | ------- | ------- | ------- |
+| `write.insert.deduplicate` | `false` | `true` | 默认 `Insert 模式` 会去重，关闭后每次 flush 都会直接写独立的 parquet |
+
+## Hive 查询
+
+### 打包
+
+第一步是打包 `hudi-flink-bundle_2.11-0.9.0.jar`。 `hudi-flink-bundle` module pom.xml 默认将 Hive 相关的依赖 scope 设置为 provided，
+如果想打入 Hive 的依赖，需要显示指定 Profile 为 `flink-bundle-shade-hive`。执行以下命令打入 Hive 依赖：
+
+```bash
+# Maven 打包命令
+mvn install -DskipTests -Drat.skip=true -Pflink-bundle-shade-hive2
+
+# 如果是 hive3 需要使用 profile -Pflink-bundle-shade-hive3
+# 如果是 hive1 需要使用 profile -Pflink-bundle-shade-hive1
+```
+
+:::note
+Hive1.x 现在只能实现同步 metadata 到 Hive，而无法使用 Hive 查询，如需查询可使用 Spark 查询 Hive 外表的方法查询。
+:::
+
+:::note
+使用 -Pflink-bundle-shade-hive x，需要修改 Profile 中 Hive 的版本为集群对应版本（只需修改 Profile 里的 Hive 版本）。修改位置为 `packaging/hudi-flink-bundle/pom.xml`
+最下面的对应 Profile 段，找到后修改 Profile 中的 Hive 版本为对应版本即可。
+:::
+
+### Hive 环境准备
+
+1. 第一步是将 `hudi-hadoop-mr-bundle.jar` 放到 Hive中。 在 Hive 的根目录下创建 `auxlib/` 文件夹，把 `hudi-hadoop-mr-bundle-0.x.x-SNAPSHOT.jar` 移入到 `auxlib`。
+`hudi-hadoop-mr-bundle-0.x.x-SNAPSHOT.jar` 可以在 `packaging/hudi-hadoop-mr-bundle/target` 目录下拷贝。
+
+2. 第二步是开启 Hive 相关的服务。Flink SQL Client 远程连接 Hive 的时候，要求 Hive 的 `hive metastore` and `hiveserver2` 两个服务都开启且需要记住端口号。
+服务开启的方法如下：
+
+```bash
+# 启动 Hive metastore 和 hiveserver2
+nohup ./bin/hive --service metastore &
+nohup ./bin/hive --service hiveserver2 &
+
+# 每次更新了 /auxlib 下的 jar 包都需要重启上述两个服务
+```
+
+### Hive 配置模版
+
+Flink hive sync 现在支持两种 hive sync mode，分别是 `hms` 和 `jdbc`。其中 `hms` 模式只需要配置 Metastore uris；而 `jdbc` 模式需要同时
+配置 JDBC 属性 和 Metastore uris，具体配置模版如下：
+
+```sql
+-- hms mode 配置模版
+CREATE TABLE t1(
+ uuid VARCHAR(20),
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'oss://vvr-daily/hudi/t1',
+ 'table.type' = 'COPY_ON_WRITE', --MERGE_ON_READ 方式在没生成 parquet 文件前，Hive 不会有输出
+ 'hive_sync.enable' = 'true', -- Required。开启 Hive 同步功能
+ 'hive_sync.mode' = 'hms' -- Required。将 hive sync mode 设置为 hms, 默认 jdbc
+ 'hive_sync.metastore.uris' = 'thrift://ip:9083' -- Required。metastore 的端口
+);
+
+
+-- jdbc mode 配置模版
+CREATE TABLE t1(
+ uuid VARCHAR(20),
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'oss://vvr-daily/hudi/t1',
+ 'table.type' = 'COPY_ON_WRITE', --MERGE_ON_READ 方式在没生成 parquet 文件前，Hive 不会有输出
+ 'hive_sync.enable' = 'true', -- Required。开启 Hive 同步功能
+ 'hive_sync.mode' = 'hms' -- Required。将 hive sync mode 设置为 hms, 默认 jdbc
+ 'hive_sync.metastore.uris' = 'thrift://ip:9083' -- Required。metastore 的端口
+ 'hive_sync.jdbc_url'='jdbc:hive2://ip:10000', -- required。hiveServer 的端口
+ 'hive_sync.table'='t1', -- required。hive 新建的表名
+ 'hive_sync.db'='testDB', -- required。hive 新建的数据库名
+ 'hive_sync.username'='root', -- required。HMS 用户名
+ 'hive_sync.password'='your password' -- required。HMS 密码
+);
+```
+
+### Hive 查询
+
+使用 beeline 查询时，需要手动设置：
+```bash
+set hive.input.format = org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat;
+```
+
+### 与其他包冲突
+
+当 Flink lib/ 下有 `flink-sql-connector-hive-xxx.jar` 时，会出现hive包冲突，解决方法是在 Install 时，另外再指定一个 Profile：`-Pinclude-flink-sql-connector-hive`，
+同时删除 Flink lib/ 下的 `flink-sql-connector-hive-xxx.jar`，新的 Install 的命令如下：
+
+```bash
+# Maven 打包命令
+mvn install -DskipTests -Drat.skip=true -Pflink-bundle-shade-hive2 -Pinclude-flink-sql-connector-hive
+```
+
+## Presto 查询
+
+### Hive 同步
+首先，参考 [Hive 查询](#hive-查询) 过程，完成 Hive 元数据的同步。
+
+### Presto 环境配置
+1. 根据 [Presto 配置文档](https://prestodb.io/docs/current/installation/deployment.html) 来配置 Presto。
+2. 根据下面的配置，在 ` /presto-server-0.2xxx/etc/catalog/hive.properties` 中配置 Hive catalog：
+
+```properties
+connector.name=hive-hadoop2
+hive.metastore.uri=thrift://xxx.xxx.xxx.xxx:9083
+hive.config.resources=.../hadoop-2.x/etc/hadoop/core-site.xml,.../hadoop-2.x/etc/hadoop/hdfs-site.xml
+```
+
+### Presto 查询
+
+通过 presto-cli 连接 Hive metastore 开启查询。 presto-cli 的设置参考 presto-cli 配置：
+
+```bash
+# 连接 presto server 的命令
+./presto --server xxx.xxx.xxx.xxx:9999 --catalog hive --schema default
+```
+
+:::note
+1. `Presto-server-0.2445` 版本较低，在查 MOR 表的 `rt` 表时，会出现包冲突，正在解决中
+2. 当 `Presto-server-xxx` 的版本 < 0.233 时，`hudi-presto-bundle.jar` 需要手动导入到 `{presto_install_dir}/plugin/hive-hadoop2/` 中。
+:::
+
+
+## 离线 Compaction
+
+MERGE_ON_READ 表的 compaction 默认是打开的，策略是 `5` 个 delta commits 执行一次压缩。因为压缩操作比较耗费内存，和写流程放在同一个 pipeline，在数据量比较大
+的时候（10w+/s），容易干扰写流程，此时采用离线定时任务的方式执行 compaction 任务更稳定。
+
+:::note
+一个 compaction 任务的执行包括两部分：1. 生成 compaction plan； 2. 执行对应的 compaction plan。其中，第一步生成 compaction plan 的过程推荐由写任务定时触发，
+ 写参数 `compaction.schedule.enable` 默认开启。
+:::
+
+离线 compaction 需要手动执行 Flink 任务，程序入口为： `hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar` ： `org.apache.hudi.sink.compact.HoodieFlinkCompactor`
+
+```bash
+# 命令行执行方式
+./bin/flink run -c org.apache.hudi.sink.compact.HoodieFlinkCompactor lib/hudi-flink-bundle_2.11-0.9.0.jar --path hdfs://xxx:9000/table
+```
+
+### 参数
+
+| 名称 | Required | 默认值 | 备注 |
+| ----------- | ------- | ------- | ------- |
+| `--path` | `frue` | `--` | 目标 Hudi 表的路径 |
+| `--compaction-max-memory` | `false` | `100` | 压缩时 log 数据的索引 HashMap 的内存大小，默认 `100MB`，内存足够时，可以调大 |
+| `--schedule` | `false` | `false` | 是否要执行 schedule compaction 的操作，当写流程还在持续写入表数据的时候，开启这个参数有丢失查询数据的风险，所以开启该参数一定要保证当前没有任务往表里写数据，写任务的 compaction plan 默认是一直 schedule 的，除非手动关闭（默认 `5` 个 delta commits 一次 compaction）|
+| `--seq` | `false` | `LIFO` | 执行压缩任务的顺序，默认是从最新的 compaction plan 开始执行，可选值：`LIFO` ：从最新的 plan 开始执行； `FIFO`：从最老的 plan 开始执行 |
+
+## 写入限流
+
+针对将全量数据（百亿数量级）和增量先同步到 Kafka，再通过 Flink 流式消费的方式将库表数据直接导成 Hudi 表的需求，因为直接消费全量数据：量大
+（吞吐高）、乱序严重（写入的 partition 随机），会导致写入性能退化，出现吞吐毛刺等情况，这时候可以开启限速参数，保证流量平稳写入。
+
+### 参数
+
+| 名称 | Required | 默认值 | 备注 |
+| ----------- | ------- | ------- | ------- |
+| `write.rate.limit` | `false` | `0` | 默认关闭限速 |
+
+## 从这开始下一步？
+
+您也可以通过[自己构建hudi](https://github.com/apache/hudi#building-apache-hudi-from-source)来快速开始，
+并在spark-shell命令中使用`--jars <path to hudi_code>/packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.1?-*.*.*-SNAPSHOT.jar`，
+而不是`--packages org.apache.hudi:hudi-spark3-bundle_2.12:0.8.0`
+
+
+这里我们使用Spark演示了Hudi的功能。但是，Hudi可以支持多种存储类型/视图，并且可以从Hive，Spark，Presto等查询引擎中查询Hudi数据集。
+我们制作了一个基于Docker设置、所有依赖系统都在本地运行的[演示视频](https://www.youtube.com/watch?v=VhNgUsxdrD0)，
+我们建议您复制相同的设置然后按照[这里](/cn/docs/docker_demo)的步骤自己运行这个演示。
+另外，如果您正在寻找将现有数据迁移到Hudi的方法，请参考[迁移指南](/cn/docs/migration_guide)。
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/gcs_hoodie.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/gcs_hoodie.md
new file mode 100644
index 00000000000..7906b74c150
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/gcs_hoodie.md
@@ -0,0 +1,61 @@
+---
+title: GCS 文件系统
+keywords: [ hudi, hive, google cloud, storage, spark, presto, 存储 ]
+summary: 在本页中，我们探讨如何在 Google Cloud Storage 中配置 Hudi。
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+对于存储在 GCS 上的 Hudi ， **区域** Bucket 提供了带有强一致性的 DFS API 。
+
+## GCS 配置
+
+Hudi 的 GCS 适配需要两项配置：
+
+- 为 Hudi 添加 GCS 凭证
+- 将需要的 jar 包添加到类路径
+
+### GCS 凭证
+
+在你的 core-site.xml 文件中添加必要的配置，Hudi 将从那里获取这些配置。 用你的 GCS 分区名称替换掉 `fs.defaultFS` ，以便 Hudi 能够在 Bucket 中读取/写入。
+
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>gs://hudi-bucket</value>
+ </property>
+
+ <property>
+ <name>fs.gs.impl</name>
+ <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem</value>
+ <description>The FileSystem for gs: (GCS) uris.</description>
+ </property>
+
+ <property>
+ <name>fs.AbstractFileSystem.gs.impl</name>
+ <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS</value>
+ <description>The AbstractFileSystem for gs: (GCS) uris.</description>
+ </property>
+
+ <property>
+ <name>fs.gs.project.id</name>
+ <value>GCS_PROJECT_ID</value>
+ </property>
+ <property>
+ <name>google.cloud.auth.service.account.enable</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>google.cloud.auth.service.account.email</name>
+ <value>GCS_SERVICE_ACCOUNT_EMAIL</value>
+ </property>
+ <property>
+ <name>google.cloud.auth.service.account.keyfile</name>
+ <value>GCS_SERVICE_ACCOUNT_KEYFILE</value>
+ </property>
+```
+
+### GCS 库
+
+将 GCS Hadoop 库添加到我们的类路径
+
+- com.google.cloud.bigdataoss:gcs-connector:1.6.0-hadoop2
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/ibm_cos_hoodie.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/ibm_cos_hoodie.md
new file mode 100644
index 00000000000..7cd03ab9142
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/ibm_cos_hoodie.md
@@ -0,0 +1,78 @@
+---
+title: IBM Cloud Object Storage 文件系统
+keywords: [ hudi, hive, ibm, cos, spark, presto]
+summary: 在本页中，我们讨论在 IBM Cloud Object Storage 文件系统中配置 Hudi 。
+last_modified_at: 2020-10-01T11:38:24-10:00
+language: cn
+---
+在本页中，我们解释如何将你的 Hudi Spark 作业存储到 IBM Cloud Object Storage 当中。
+
+## IBM COS 配置
+
+Hudi 适配 IBM Cloud Object Storage 需要两项配置：
+
+- 为 Hudi 添加 IBM COS 凭证
+- 添加需要的 jar 包到类路径
+
+### IBM Cloud Object Storage 凭证
+
+在 IBM Cloud Object Storage 上使用 Hudi 的最简单的办法，就是使用 [Stocator](https://github.com/CODAIT/stocator) 的 Spark 存储连接器为 `SparkSession` 或 `SparkContext` 配置 IBM Cloud Object Storage 凭证。 Hudi 将自动拾取配置并告知 IBM Cloud Object Storage 。
+
+或者，向你的 core-site.xml 文件中添加必要的配置，Hudi 可以从那里获取这些配置。用你的 IBM Cloud Object Storage 的 Bucket 名称替换 `fs.defaultFS` 以便 Hudi 能够在 Bucket 中读取/写入。
+
+例如，使用 HMAC 密钥以及服务名 `myCOS` ：
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>cos://myBucket.myCOS</value>
+ </property>
+
+ <property>
+ <name>fs.cos.flat.list</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>fs.stocator.scheme.list</name>
+ <value>cos</value>
+ </property>
+
+ <property>
+ <name>fs.cos.impl</name>
+ <value>com.ibm.stocator.fs.ObjectStoreFileSystem</value>
+ </property>
+
+ <property>
+ <name>fs.stocator.cos.impl</name>
+ <value>com.ibm.stocator.fs.cos.COSAPIClient</value>
+ </property>
+
+ <property>
+ <name>fs.stocator.cos.scheme</name>
+ <value>cos</value>
+ </property>
+
+ <property>
+ <name>fs.cos.myCos.access.key</name>
+ <value>ACCESS KEY</value>
+ </property>
+
+ <property>
+ <name>fs.cos.myCos.endpoint</name>
+ <value>http://s3-api.us-geo.objectstorage.softlayer.net</value>
+ </property>
+
+ <property>
+ <name>fs.cos.myCos.secret.key</name>
+ <value>SECRET KEY</value>
+ </property>
+
+```
+
+更多信息请参考 Stocator [文档](https://github.com/CODAIT/stocator/blob/master/README.md) 。
+
+### IBM Cloud Object Storage 库
+
+将 IBM Cloud Object Storage Hadoop 库添加到我们的类路径中：
+
+ - com.ibm.stocator:stocator:1.1.3
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/migration_guide.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/migration_guide.md
new file mode 100644
index 00000000000..c7b61cab4ff
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/migration_guide.md
@@ -0,0 +1,60 @@
+---
+title: 迁移指南
+keywords: [ hudi, migration, use case, 迁移, 用例]
+summary: 在本页中，我们将讨论有效的工具，他们能将你的现有数据集迁移到 Hudi 数据集。
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+Hudi 维护了元数据，包括提交的时间线和索引，来管理一个数据集。提交的时间线帮助理解一个数据集上发生的操作，以及数据集的当前状态。索引则被 Hudi 用来维护记录键到文件 ID 的映射，它能高效地定位一条记录。目前， Hudi 仅支持写 Parquet 列式格式 。
+
+为了在你的现有数据集上开始使用 Hudi ，你需要将你的现有数据集迁移到 Hudi 管理的数据集中。以下有多种方法实现这个目的。
+
+
+## 方法
+
+
+### 将 Hudi 仅用于新分区
+
+Hudi 可以被用来在不影响/改变数据集历史数据的情况下管理一个现有的数据集。 Hudi 已经实现兼容这样的数据集，需要注意的是，单个 Hive 分区要么完全由 Hudi 管理，要么不由 Hudi 管理。因此， Hudi 管理一个数据集的最低粒度是一个 Hive 分区。使用数据源 API 或 WriteClient 来写入数据集，并确保你开始写入的是一个新分区，或者将过去的 N 个分区而非整张表转换为 Hudi 。需要注意的是，由于历史分区不是由 Hudi 管理的， Hudi 提供的任何操作在那些分区上都不生效。更具体地说，无法在这些非 Hudi 管理的旧分区上进行插入更新或增量拉取。
+
+如果你的数据集是追加型的数据集，并且你不指望在已经存在的（或者非 Hudi 管理的）分区上进行更新操作，就使用这个方法。
+
+### 将现有的数据集转换为 Hudi
+
+将你的现有数据集导入到一个 Hudi 管理的数据集。由于全部数据都是 Hudi 管理的，方法 1 的任何限制在这里都不适用。跨分区的更新可以被应用到这个数据集，而 Hudi 会高效地让这些更新对查询可用。值得注意的是，你不仅可以在这个数据集上使用所有 Hudi 提供的操作，这样做还有额外的好处。 Hudi 会自动管理受管数据集的文件大小。你可以在转换数据集的时候设置期望的文件大小， Hudi 将确保它写出的文件符合这个配置。Hudi 还会确保小文件在后续被修正，这个过程是通过将新的插入引导到这些小文件而不是写入新的小文件来实现的，这样能维持你的集群的健康度。
+
+选择这个方法后，有几种选择。
+
+**选择 1**
+使用 HDFSParquetImporter 工具。正如名字表明的那样，这仅仅适用于你的现有数据集是 Parquet 文件格式的。
+这个工具本质上是启动一个 Spark 作业来读取现有的 Parquet 数据集，并通过重写全部记录的方式将它转换为 HUDI 管理的数据集。
+
+**选择 2**
+对于大数据集，这可以简单地：
+```java
+for partition in [list of partitions in source dataset] {
+ val inputDF = spark.read.format("any_input_format").load("partition_path")
+ inputDF.write.format("org.apache.hudi").option()....save("basePath")
+}
+```
+
+**选择 3**
+写下你自定义的逻辑来定义如何将现有数据集加载到一个 Hudi 管理的数据集中。请在 [这里](/cn/docs/quick-start-guide) 阅读 RDD API 的相关资料。使用 HDFSParquetImporter 工具。一旦 Hudi 通过 `mvn clean install -DskipTests` 被构建了， Shell 将被 `cd hudi-cli && ./hudi-cli.sh` 调启。
+
+```java
+hudi->hdfsparquetimport
+ --upsert false
+ --srcPath /user/parquet/dataset/basepath
+ --targetPath
+ /user/hoodie/dataset/basepath
+ --tableName hoodie_table
+ --tableType COPY_ON_WRITE
+ --rowKeyField _row_key
+ --partitionPathField partitionStr
+ --parallelism 1500
+ --schemaFilePath /user/table/schema
+ --format parquet
+ --sparkMemory 6g
+ --retry 2
+```
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/oss_hoodie.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/oss_hoodie.md
new file mode 100644
index 00000000000..2863d3351c1
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/oss_hoodie.md
@@ -0,0 +1,71 @@
+---
+title: OSS Filesystem
+keywords: [ hudi, hive, aliyun, oss, spark, presto]
+summary: In this page, we go over how to configure Hudi with OSS filesystem.
+last_modified_at: 2020-04-21T12:50:50-10:00
+language: cn
+---
+这个页面描述了如何让你的Hudi spark任务使用Aliyun OSS存储。
+
+## Aliyun OSS 部署
+
+为了让Hudi使用OSS，需要增加两部分的配置:
+
+- 为Hudi增加Aliyun OSS的相关配置
+- 增加Jar包的MVN依赖
+
+### Aliyun OSS 相关的配置
+
+新增下面的配置到你的Hudi能访问的core-site.xml文件。使用你的OSS bucket name替换掉`fs.defaultFS`，使用OSS endpoint地址替换`fs.oss.endpoint`，使用OSS的key和secret分别替换`fs.oss.accessKeyId`和`fs.oss.accessKeySecret`。这样Hudi就能读写相应的bucket。
+
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>oss://bucketname/</value>
+ </property>
+
+ <property>
+ <name>fs.oss.endpoint</name>
+ <value>oss-endpoint-address</value>
+ <description>Aliyun OSS endpoint to connect to.</description>
+ </property>
+
+ <property>
+ <name>fs.oss.accessKeyId</name>
+ <value>oss_key</value>
+ <description>Aliyun access key ID</description>
+ </property>
+
+ <property>
+ <name>fs.oss.accessKeySecret</name>
+ <value>oss-secret</value>
+ <description>Aliyun access key secret</description>
+ </property>
+
+ <property>
+ <name>fs.oss.impl</name>
+ <value>org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem</value>
+ </property>
+```
+
+### Aliyun OSS Libs
+
+新增Aliyun hadoop的jar包的MVN依赖到pom.xml文件。由于hadoop-aliyun依赖hadoop 2.9.1+，因此你需要使用hadoop 2.9.1或更新的版本。
+
+```xml
+<dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-aliyun</artifactId>
+ <version>3.2.1</version>
+</dependency>
+<dependency>
+ <groupId>com.aliyun.oss</groupId>
+ <artifactId>aliyun-sdk-oss</artifactId>
+ <version>3.8.1</version>
+</dependency>
+<dependency>
+ <groupId>org.jdom</groupId>
+ <artifactId>jdom</artifactId>
+ <version>1.1</version>
+</dependency>
+```
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/overview.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/overview.md
new file mode 100644
index 00000000000..2ae90fe37bd
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/overview.md
@@ -0,0 +1,155 @@
+---
+version: 0.8.0
+title: 概念
+keywords: [ hudi, design, storage, views, timeline]
+summary: "Here we introduce some basic concepts & give a broad technical overview of Hudi"
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+Apache Hudi(发音为“Hudi”)在DFS的数据集上提供以下流原语
+
+ * 插入更新 (如何改变数据集?)
+ * 增量拉取 (如何获取变更的数据?)
+
+在本节中，我们将讨论重要的概念和术语，这些概念和术语有助于理解并有效使用这些原语。
+
+## 时间轴
+在它的核心，Hudi维护一条包含在不同的`即时`时间所有对数据集操作的`时间轴`，从而提供，从不同时间点出发得到不同的视图下的数据集。Hudi即时包含以下组件
+
+ * `操作类型` : 对数据集执行的操作类型
+ * `即时时间` : 即时时间通常是一个时间戳(例如：20190117010349)，该时间戳按操作开始时间的顺序单调增加。
+ * `状态` : 即时的状态
+
+Hudi保证在时间轴上执行的操作的原子性和基于即时时间的时间轴一致性。
+
+执行的关键操作包括
+
+ * `COMMITS` - 一次提交表示将一组记录**原子写入**到数据集中。
+ * `CLEANS` - 删除数据集中不再需要的旧文件版本的后台活动。
+ * `DELTA_COMMIT` - 增量提交是指将一批记录**原子写入**到MergeOnRead存储类型的数据集中，其中一些/所有数据都可以只写到增量日志中。
+ * `COMPACTION` - 协调Hudi中差异数据结构的后台活动，例如：将更新从基于行的日志文件变成列格式。在内部，压缩表现为时间轴上的特殊提交。
+ * `ROLLBACK` - 表示提交/增量提交不成功且已回滚，删除在写入过程中产生的所有部分文件。
+ * `SAVEPOINT` - 将某些文件组标记为"已保存"，以便清理程序不会将其删除。在发生灾难/数据恢复的情况下，它有助于将数据集还原到时间轴上的某个点。
+
+任何给定的即时都可以处于以下状态之一
+
+ * `REQUESTED` - 表示已调度但尚未启动的操作。
+ * `INFLIGHT` - 表示当前正在执行该操作。
+ * `COMPLETED` - 表示在时间轴上完成了该操作。
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_timeline.png" alt="hudi_timeline.png" />
+</figure>
+
+上面的示例显示了在Hudi数据集上大约10:00到10:20之间发生的更新事件，大约每5分钟一次，将提交元数据以及其他后台清理/压缩保留在Hudi时间轴上。
+观察的关键点是：提交时间指示数据的`到达时间`（上午10:20），而实际数据组织则反映了实际时间或`事件时间`，即数据所反映的（从07:00开始的每小时时段）。在权衡数据延迟和完整性时，这是两个关键概念。
+
+如果有延迟到达的数据（事件时间为9:00的数据在10:20达到，延迟 >1 小时），我们可以看到upsert将新数据生成到更旧的时间段/文件夹中。
+在时间轴的帮助下，增量查询可以只提取10:00以后成功提交的新数据，并非常高效地只消费更改过的文件，且无需扫描更大的文件范围，例如07:00后的所有时间段。
+
+## 文件组织
+Hudi将DFS上的数据集组织到`基本路径`下的目录结构中。数据集分为多个分区，这些分区是包含该分区的数据文件的文件夹，这与Hive表非常相似。
+每个分区被相对于基本路径的特定`分区路径`区分开来。
+
+在每个分区内，文件被组织为`文件组`，由`文件id`唯一标识。
+每个文件组包含多个`文件切片`，其中每个切片包含在某个提交/压缩即时时间生成的基本列文件（`*.parquet`）以及一组日志文件（`*.log*`），该文件包含自生成基本文件以来对基本文件的插入/更新。
+Hudi采用MVCC设计，其中压缩操作将日志和基本文件合并以产生新的文件片，而清理操作则将未使用的/较旧的文件片删除以回收DFS上的空间。
+
+Hudi通过索引机制将给定的hoodie键（记录键+分区路径）映射到文件组，从而提供了高效的Upsert。
+一旦将记录的第一个版本写入文件，记录键和文件组/文件id之间的映射就永远不会改变。 简而言之，映射的文件组包含一组记录的所有版本。
+
+## 存储类型和视图
+Hudi存储类型定义了如何在DFS上对数据进行索引和布局以及如何在这种组织之上实现上述原语和时间轴活动（即如何写入数据）。
+反过来，`视图`定义了基础数据如何暴露给查询（即如何读取数据）。
+
+| 存储类型 | 支持的视图 |
+|-------------- |------------------|
+| 写时复制 | 近实时 + 增量 |
+| 读时合并 | 近实时 + 增量 + 读优化 |
+
+### 存储类型
+Hudi支持以下存储类型。
+
+ - [写时复制](#copy-on-write-storage) : 仅使用列文件格式（例如parquet）存储数据。通过在写入过程中执行同步合并以更新版本并重写文件。
+
+ - [读时合并](#merge-on-read-storage) : 使用列式（例如parquet）+ 基于行（例如avro）的文件格式组合来存储数据。 更新记录到增量文件中，然后进行同步或异步压缩以生成列文件的新版本。
+
+下表总结了这两种存储类型之间的权衡
+
+| 权衡 | 写时复制 | 读时合并 |
+|-------------- |------------------| ------------------|
+| 数据延迟 | 更高 | 更低 |
+| 更新代价(I/O) | 更高（重写整个parquet文件） | 更低（追加到增量日志） |
+| Parquet文件大小 | 更小（高更新代价（I/o）） | 更大（低更新代价） |
+| 写放大 | 更高 | 更低（取决于压缩策略） |
+
+
+### 视图
+Hudi支持以下存储数据的视图
+
+ - **读优化视图** : 在此视图上的查询将查看给定提交或压缩操作中数据集的最新快照。
+ 该视图仅将最新文件切片中的基本/列文件暴露给查询，并保证与非Hudi列式数据集相比，具有相同的列式查询性能。
+ - **增量视图** : 对该视图的查询只能看到从某个提交/压缩后写入数据集的新数据。该视图有效地提供了更改流，来支持增量数据管道。
+ - **实时视图** : 在此视图上的查询将查看某个增量提交操作中数据集的最新快照。该视图通过动态合并最新的基本文件(例如parquet)和增量文件(例如avro)来提供近实时数据集（几分钟的延迟）。
+
+
+下表总结了不同视图之间的权衡。
+
+| 权衡 | 读优化 | 实时 |
+|-------------- |------------------| ------------------|
+| 数据延迟 | 更高 | 更低 |
+| 查询延迟 | 更低（原始列式性能）| 更高（合并列式 + 基于行的增量） |
+
+
+## 写时复制存储 {#copy-on-write-storage}
+
+写时复制存储中的文件片仅包含基本/列文件，并且每次提交都会生成新版本的基本文件。
+换句话说，我们压缩每个提交，从而所有的数据都是以列数据的形式储存。在这种情况下，写入数据非常昂贵（我们需要重写整个列数据文件，即使只有一个字节的新数据被提交），而读取数据的成本则没有增加。
+这种视图有利于读取繁重的分析工作。
+
+以下内容说明了将数据写入写时复制存储并在其上运行两个查询时，它是如何工作的。
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_cow.png" alt="hudi_cow.png" />
+</figure>
+
+
+随着数据的写入，对现有文件组的更新将为该文件组生成一个带有提交即时时间标记的新切片，而插入分配一个新文件组并写入该文件组的第一个切片。
+这些文件切片及其提交即时时间在上面用颜色编码。
+针对这样的数据集运行SQL查询（例如：`select count(*)`统计该分区中的记录数目），首先检查时间轴上的最新提交并过滤每个文件组中除最新文件片以外的所有文件片。
+如您所见，旧查询不会看到以粉红色标记的当前进行中的提交的文件，但是在该提交后的新查询会获取新数据。因此，查询不受任何写入失败/部分写入的影响，仅运行在已提交数据上。
+
+写时复制存储的目的是从根本上改善当前管理数据集的方式，通过以下方法来实现
+
+ - 优先支持在文件级原子更新数据，而无需重写整个表/分区
+ - 能够只读取更新的部分，而不是进行低效的扫描或搜索
+ - 严格控制文件大小来保持出色的查询性能（小的文件会严重损害查询性能）。
+
+## 读时合并存储 {#merge-on-read-storage}
+
+读时合并存储是写时复制的升级版，从某种意义上说，它仍然可以通过读优化表提供数据集的读取优化视图（写时复制的功能）。
+此外，它将每个文件组的更新插入存储到基于行的增量日志中，通过文件id，将增量日志和最新版本的基本文件进行合并，从而提供近实时的数据查询。因此，此存储类型智能地平衡了读和写的成本，以提供近乎实时的查询。
+这里最重要的一点是压缩器，它现在可以仔细挑选需要压缩到其列式基础文件中的增量日志（根据增量日志的文件大小），以保持查询性能（较大的增量日志将会提升近实时的查询时间，并同时需要更长的合并时间）。
+
+以下内容说明了存储的工作方式，并显示了对近实时表和读优化表的查询。
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_mor.png" alt="hudi_mor.png" />
+</figure>
+
+此示例中发生了很多有趣的事情，这些带出了该方法的微妙之处。
+
+ - 现在，我们每1分钟左右就有一次提交，这是其他存储类型无法做到的。
+ - 现在，在每个文件id组中，都有一个增量日志，其中包含对基础列文件中记录的更新。
+ 在示例中，增量日志包含10:05至10:10的所有数据。与以前一样，基本列式文件仍使用提交进行版本控制。
+ 因此，如果只看一眼基本文件，那么存储布局看起来就像是写时复制表的副本。
+ - 定期压缩过程会从增量日志中合并这些更改，并生成基础文件的新版本，就像示例中10:05发生的情况一样。
+ - 有两种查询同一存储的方式：读优化（RO）表和近实时（RT）表，具体取决于我们选择查询性能还是数据新鲜度。
+ - 对于RO表来说，提交数据在何时可用于查询将有些许不同。 请注意，以10:10运行的（在RO表上的）此类查询将不会看到10:05之后的数据，而在RT表上的查询总会看到最新的数据。
+ - 何时触发压缩以及压缩什么是解决这些难题的关键。
+ 通过实施压缩策略，在该策略中，与较旧的分区相比，我们会积极地压缩最新的分区，从而确保RO表能够以一致的方式看到几分钟内发布的数据。
+
+读时合并存储上的目的是直接在DFS上启用近实时处理，而不是将数据复制到专用系统，后者可能无法处理大数据量。
+该存储还有一些其他方面的好处，例如通过避免数据的同步合并来减少写放大，即批量数据中每1字节数据需要的写入数据量。
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/performance.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/performance.md
new file mode 100644
index 00000000000..25a1c7589ef
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/performance.md
@@ -0,0 +1,62 @@
+---
+title: 性能
+keywords: [ hudi, index, storage, compaction, cleaning, implementation]
+toc: false
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+在本节中，我们将介绍一些有关Hudi插入更新、增量提取的实际性能数据，并将其与实现这些任务的其它传统工具进行比较。
+
+## 插入更新
+
+下面显示了从NoSQL数据库摄取获得的速度提升，这些速度提升数据是通过在写入时复制存储上的Hudi数据集上插入更新而获得的，
+数据集包括5个从小到大的表（相对于批量加载表）。
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_upsert_perf1.png").default} alt="hudi_upsert_perf1.png" />
+</figure>
+
+由于Hudi可以通过增量构建数据集，它也为更频繁地调度摄取提供了可能性，从而减少了延迟，并显著节省了总体计算成本。
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_upsert_perf2.png").default} alt="hudi_upsert_perf2.png" />
+</figure>
+
+Hudi插入更新在t1表的一次提交中就进行了高达4TB的压力测试。
+有关一些调优技巧，请参见[这里](https://cwiki.apache.org/confluence/display/HUDI/Tuning+Guide)。
+
+## 索引
+
+为了有效地插入更新数据，Hudi需要将要写入的批量数据中的记录分类为插入和更新（并标记它所属的文件组）。
+为了加快此操作的速度，Hudi采用了可插拔索引机制，该机制存储了recordKey和它所属的文件组ID之间的映射。
+默认情况下，Hudi使用内置索引，该索引使用文件范围和布隆过滤器来完成此任务，相比于Spark Join，其速度最高可提高10倍。
+
+当您将recordKey建模为单调递增时（例如时间戳前缀），Hudi提供了最佳的索引性能，从而进行范围过滤来避免与许多文件进行比较。
+即使对于基于UUID的键，也有[已知技术](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/)来达到同样目的。
+例如，在具有80B键、3个分区、11416个文件、10TB数据的事件表上使用100M个时间戳前缀的键（5％的更新，95％的插入）时，
+相比于原始Spark Join，Hudi索引速度的提升**约为7倍（440秒相比于2880秒）**。
+即使对于具有挑战性的工作负载，如使用300个核对3.25B UUID键、30个分区、6180个文件的“100％更新”的数据库摄取工作负载，Hudi索引也可以提供**80-100％的加速**。
+
+## 读优化查询
+
+读优化视图的主要设计目标是在不影响查询的情况下实现上一节中提到的延迟减少和效率提高。
+下图比较了对Hudi和非Hudi数据集的Hive、Presto、Spark查询，并对此进行说明。
+
+**Hive**
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_query_perf_hive.png").default} alt="hudi_query_perf_hive.png" />
+</figure>
+
+**Spark**
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_query_perf_spark.png").default} alt="hudi_query_perf_spark.png" />
+</figure>
+
+**Presto**
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_query_perf_presto.png").default} alt="hudi_query_perf_presto.png" />
+</figure>
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/powered_by.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/powered_by.md
new file mode 100644
index 00000000000..3d0c3c3a653
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/powered_by.md
@@ -0,0 +1,88 @@
+---
+title: 演讲 & Hudi 用户
+keywords: [ hudi, talks, presentation]
+last_modified_at: 2019-12-31T15:59:57-04:00
+language: cn
+---
+
+## 已使用
+
+### Uber
+
+Hudi最初由[Uber](https://uber.com)开发，用于实现[低延迟、高效率的数据库摄取](http://www.slideshare.net/vinothchandar/hadoop-strata-talk-uber-your-hadoop-has-arrived/32)。
+Hudi自2016年8月开始在生产环境上线，在Hadoop上驱动约100个非常关键的业务表，支撑约几百TB的数据规模(前10名包括行程、乘客、司机)。
+Hudi还支持几个增量的Hive ETL管道，并且目前已集成到Uber的数据分发系统中。
+
+### EMIS Health
+
+[EMIS Health](https://www.emishealth.com/)是英国最大的初级保健IT软件提供商，其数据集包括超过5000亿的医疗保健记录。HUDI用于管理生产中的分析数据集，并使其与上游源保持同步。Presto用于查询以HUDI格式写入的数据。
+
+### Yields.io
+
+[Yields.io](https://www.yields.io/Blog/Apache-Hudi-at-Yields)是第一个使用AI在企业范围内进行自动模型验证和实时监控的金融科技平台。他们的数据湖由Hudi管理，他们还积极使用Hudi为增量式、跨语言/平台机器学习构建基础架构。
+
+### Yotpo
+
+Hudi在Yotpo有不少用途。首先，在他们的[开源ETL框架](https://github.com/YotpoLtd/metorikku)中集成了Hudi作为CDC管道的输出写入程序，即从数据库binlog生成的事件流到Kafka然后再写入S3。
+
+## 演讲 & 报告
+
+1. ["Hoodie: Incremental processing on Hadoop at Uber"](https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/56511) - By Vinoth Chandar & Prasanna Rajaperumal
+ Mar 2017, Strata + Hadoop World, San Jose, CA
+
+2. ["Hoodie: An Open Source Incremental Processing Framework From Uber"](http://www.dataengconf.com/hoodie-an-open-source-incremental-processing-framework-from-uber) - By Vinoth Chandar.
+ Apr 2017, DataEngConf, San Francisco, CA [Slides](https://www.slideshare.net/vinothchandar/hoodie-dataengconf-2017) [Video](https://www.youtube.com/watch?v=7Wudjc-v7CA)
+
+3. ["Incremental Processing on Large Analytical Datasets"](https://spark-summit.org/2017/events/incremental-processing-on-large-analytical-datasets/) - By Prasanna Rajaperumal
+ June 2017, Spark Summit 2017, San Francisco, CA. [Slides](https://www.slideshare.net/databricks/incremental-processing-on-large-analytical-datasets-with-prasanna-rajaperumal-and-vinoth-chandar) [Video](https://www.youtube.com/watch?v=3HS0lQX-cgo&feature=youtu.be)
+
+4. ["Hudi: Unifying storage and serving for batch and near-real-time analytics"](https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/70937) - By Nishith Agarwal & Balaji Vardarajan
+ September 2018, Strata Data Conference, New York, NY
+
+5. ["Hudi: Large-Scale, Near Real-Time Pipelines at Uber"](https://databricks.com/session/hudi-near-real-time-spark-pipelines-at-petabyte-scale) - By Vinoth Chandar & Nishith Agarwal
+ October 2018, Spark+AI Summit Europe, London, UK
+
+6. ["Powering Uber's global network analytics pipelines in real-time with Apache Hudi"](https://www.youtube.com/watch?v=1w3IpavhSWA) - By Ethan Guo & Nishith Agarwal, April 2019, Data Council SF19, San Francisco, CA.
+
+7. ["Building highly efficient data lakes using Apache Hudi (Incubating)"](https://www.slideshare.net/ChesterChen/sf-big-analytics-20190612-building-highly-efficient-data-lakes-using-apache-hudi) - By Vinoth Chandar
+ June 2019, SF Big Analytics Meetup, San Mateo, CA
+
+8. ["Apache Hudi (Incubating) - The Past, Present and Future Of Efficient Data Lake Architectures"](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM) - By Vinoth Chandar & Balaji Varadarajan
+ September 2019, ApacheCon NA 19, Las Vegas, NV, USA
+
+9. ["Insert, upsert, and delete data in Amazon S3 using Amazon EMR"](https://www.portal.reinvent.awsevents.com/connect/sessionDetail.ww?SESSION_ID=98662&csrftkn=YS67-AG7B-QIAV-ZZBK-E6TT-MD4Q-1HEP-747P) - By Paul Codding & Vinoth Chandar
+ December 2019, AWS re:Invent 2019, Las Vegas, NV, USA
+
+10. ["Building Robust CDC Pipeline With Apache Hudi And Debezium"](https://www.slideshare.net/SyedKather/building-robust-cdc-pipeline-with-apache-hudi-and-debezium) - By Pratyaksh, Purushotham, Syed and Shaik December 2019, Hadoop Summit Bangalore, India
+
+11. ["Using Apache Hudi to build the next-generation data lake and its application in medical big data"](https://drive.google.com/open?id=1dmH2kWJF69PNdifPp37QBgjivOHaSLDn) - By JingHuang & Leesf March 2020, Apache Hudi & Apache Kylin Online Meetup, China
+
+12. ["Building a near real-time, high-performance data warehouse based on Apache Hudi and Apache Kylin"](https://drive.google.com/open?id=1Pk_WdFxfEZxMMfAOn0R8-m3ALkcN6G9e) - By ShaoFeng Shi March 2020, Apache Hudi & Apache Kylin Online Meetup, China
+
+13. ["Building large scale, transactional data lakes using Apache Hudi"](https://berlinbuzzwords.de/session/building-large-scale-transactional-data-lakes-using-apache-hudi) - By Nishith Agarwal, June 2020, Berlin Buzzwords 2020.
+
+14. ["Apache Hudi - Design/Code Walkthrough Session for Contributors"](https://www.youtube.com/watch?v=N2eDfU_rQ_U) - By Vinoth Chandar, July 2020, Hudi community.
+
+15. ["PrestoDB and Apache Hudi"](https://youtu.be/nA3rwOdmm3A) - By Bhavani Sudha Saktheeswaran and Brandon Scheller, Aug 2020, PrestoDB Community Meetup.
+
+16. ["Panel Discussion on Presto Ecosystem"](https://www.youtube.com/watch?v=lsFSM2Z4kPs) - By Vinoth Chandar, Sep 2020, PrestoCon ["panel"](https://prestocon2020.sched.com/event/dgyw).
+
+17. ["Next Generation Data lakes using Apache Hudi"](https://docs.google.com/presentation/d/1y-ryRwCdTbqQHGr_bn3lxM_B8L1L5nsZOIXlJsDl_wU/edit?usp=sharing) - By Balaji Varadarajan and Sivabalan Narayanan, Sep 2020, ["ApacheCon"](https://www.apachecon.com/)
+
+18. ["Landing practice of Apache Hudi in T3go"](https://drive.google.com/file/d/1ULVPkjynaw-07wsutLcZm-4rVXf8E8N8/view?usp=sharing) - By VinoYang and XianghuWang, November 2020, Qcon.
+## 文章
+
+You can check out [our blog pages](https://hudi.apache.org/blog) for content written by our committers/contributors.
+
+1. ["The Case for incremental processing on Hadoop"](https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop) - O'reilly Ideas article by Vinoth Chandar
+2. ["Hoodie: Uber Engineering's Incremental Processing Framework on Hadoop"](https://eng.uber.com/hoodie/) - Engineering Blog By Prasanna Rajaperumal
+3. ["New – Insert, Update, Delete Data on S3 with Amazon EMR and Apache Hudi"](https://aws.amazon.com/blogs/aws/new-insert-update-delete-data-on-s3-with-amazon-emr-and-apache-hudi/) - AWS Blog by Danilo Poccia
+4. ["The Apache Software Foundation Announces Apache® Hudi™ as a Top-Level Project"](https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces64) - ASF Graduation announcement
+5. ["Apache Hudi grows cloud data lake maturity"](https://searchdatamanagement.techtarget.com/news/252484740/Apache-Hudi-grows-cloud-data-lake-maturity)
+6. ["Building a Large-scale Transactional Data Lake at Uber Using Apache Hudi"](https://eng.uber.com/apache-hudi-graduation/) - Uber eng blog by Nishith Agarwal
+7. ["Hudi On Hops"](https://www.diva-portal.org/smash/get/diva2:1413103/FULLTEXT01.pdf) - By NETSANET GEBRETSADKAN KIDANE
+8. ["开源数据湖存储框架 Apache Hudi 如何玩转增量处理"](https://www.infoq.cn/article/CAgIDpfJBVcJHKJLSbhe) - InfoQ CN article by Yanghua
+9. ["Origins of Data Lake at Grofers"](https://lambda.grofers.com/origins-of-data-lake-at-grofers-6c011f94b86c) - by Akshay Agarwal
+10. ["Data Lake Change Capture using Apache Hudi & Amazon AMS/EMR"](https://towardsdatascience.com/data-lake-change-data-capture-cdc-using-apache-hudi-on-amazon-emr-part-2-process-65e4662d7b4b) - Towards DataScience article, Oct 20
+11. ["How nClouds Helps Accelerate Data Delivery with Apache Hudi on Amazon EMR"](https://aws.amazon.com/blogs/apn/how-nclouds-helps-accelerate-data-delivery-with-apache-hudi-on-amazon-emr/) - published by nClouds in partnership with AWS
+12. ["Apply record level changes from relational databases to Amazon S3 data lake using Apache Hudi on Amazon EMR and AWS Database Migration Service"](https://aws.amazon.com/blogs/big-data/apply-record-level-changes-from-relational-databases-to-amazon-s3-data-lake-using-apache-hudi-on-amazon-emr-and-aws-database-migration-service/) - AWS blog
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/privacy.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/privacy.md
new file mode 100644
index 00000000000..afa167d37ab
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/privacy.md
@@ -0,0 +1,23 @@
+---
+title: 隐私协议
+keywords: [ hudi, privacy, 隐私]
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+关于你使用本网站的信息，将通过服务器访问日志和 Cookie 跟踪被收集。
+收集的信息由以下内容构成：
+
+* 你访问网站使用的 IP 地址；
+* 你访问我们的网站时使用的浏览器和操作系统；
+* 你访问我们的网站的日期和时间；
+* 你浏览的页面；
+* 引导你链接到我们的网站的页面地址；
+
+这些信息中的一部分将使用由 [Google Analytics](http://www.google.com/analytics) 服务设置的 Cookie 跟踪进行收集，并由 Google 按照在他们的 [隐私协议](http://www.google.com/privacy) 中描述的方式进行处理。如果你不希望与 Google 分享这些数据，请参考你的浏览器文档中关于如何禁用 Cookie 的说明。
+
+我们使用收集的数据来帮助让我们的网站对访问者更有用，并更好地了解我们的网站是如何、在何时被使用的。我们不跟踪也不收集个人隐私信息，同时也不与任何收集包含个人隐私数据的数据源合作。
+
+使用本网站，即代表你许可以上述的方式和目的收集这些数据。
+
+Hudi 开发者社区欢迎你提出关于本隐私协议的问题或评论。请将他们发送至 dev@hudi.apache.org 。
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/querying_data.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/querying_data.md
new file mode 100644
index 00000000000..cd78c331558
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/querying_data.md
@@ -0,0 +1,224 @@
+---
+title: 查询 Hudi 数据集
+keywords: [ hudi, hive, spark, sql, presto]
+summary: In this page, we go over how to enable SQL queries on Hudi built tables.
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+从概念上讲，Hudi物理存储一次数据到DFS上，同时在其上提供三个逻辑视图，如[之前](/cn/docs/concepts#views)所述。
+数据集同步到Hive Metastore后，它将提供由Hudi的自定义输入格式支持的Hive外部表。一旦提供了适当的Hudi捆绑包，
+就可以通过Hive、Spark和Presto之类的常用查询引擎来查询数据集。
+
+具体来说，在写入过程中传递了两个由[table name](/cn/docs/configurations#TABLE_NAME_OPT_KEY)命名的Hive表。
+例如，如果`table name = hudi_tbl`，我们得到
+
+ - `hudi_tbl` 实现了由 `HoodieParquetInputFormat` 支持的数据集的读优化视图，从而提供了纯列式数据。
+ - `hudi_tbl_rt` 实现了由 `HoodieParquetRealtimeInputFormat` 支持的数据集的实时视图，从而提供了基础数据和日志数据的合并视图。
+
+如概念部分所述，[增量处理](https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop)所需要的
+一个关键原语是`增量拉取`（以从数据集中获取更改流/日志）。您可以增量提取Hudi数据集，这意味着自指定的即时时间起，
+您可以只获得全部更新和新行。 这与插入更新一起使用，对于构建某些数据管道尤其有用，包括将1个或多个源Hudi表（数据流/事实）以增量方式拉出（流/事实）
+并与其他表（数据集/维度）结合以[写出增量](/cn/docs/writing_data)到目标Hudi数据集。增量视图是通过查询上表之一实现的，并具有特殊配置，
+该特殊配置指示查询计划仅需要从数据集中获取增量数据。
+
+
+## 查询引擎支持列表
+
+下面的表格展示了各查询引擎是否支持Hudi格式
+
+### 读优化表
+
+|查询引擎|实时视图|增量拉取|
+|------------|--------|-----------|
+|**Hive**|Y|Y|
+|**Spark SQL**|Y|Y|
+|**Spark Datasource**|Y|Y|
+|**Flink SQL**|Y|N|
+|**PrestoDB**|Y|N|
+|**Impala**|Y|N|
+
+
+### 实时表
+
+|查询引擎|实时视图|增量拉取|读优化表|
+|------------|--------|-----------|--------------|
+|**Hive**|Y|Y|Y|
+|**Spark SQL**|Y|Y|Y|
+|**Spark Datasource**|Y|Y|Y|
+|**Flink SQL**|Y|Y|Y|
+|**PrestoDB**|Y|N|Y|
+|**Impala**|N|N|Y|
+
+
+接下来，我们将详细讨论在每个查询引擎上如何访问所有三个视图。
+
+## Hive
+
+为了使Hive能够识别Hudi数据集并正确查询，
+HiveServer2需要在其[辅助jars路径](https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_mc_hive_udf#concept_nc3_mms_lr)中提供`hudi-hadoop-mr-bundle-x.y.z-SNAPSHOT.jar`。
+这将确保输入格式类及其依赖项可用于查询计划和执行。
+
+### 读优化表 {#hive-ro-view}
+除了上述设置之外，对于beeline cli访问，还需要将`hive.input.format`变量设置为`org.apache.hudi.hadoop.HoodieParquetInputFormat`输入格式的完全限定路径名。
+对于Tez，还需要将`hive.tez.input.format`设置为`org.apache.hadoop.hive.ql.io.HiveInputFormat`。
+
+### 实时表 {#hive-rt-view}
+除了在HiveServer2上安装Hive捆绑jars之外，还需要将其放在整个集群的hadoop/hive安装中，这样查询也可以使用自定义RecordReader。
+
+### 增量拉取 {#hive-incr-pull}
+
+`HiveIncrementalPuller`允许通过HiveQL从大型事实/维表中增量提取更改，
+结合了Hive（可靠地处理复杂的SQL查询）和增量原语的好处（通过增量拉取而不是完全扫描来加快查询速度）。
+该工具使用Hive JDBC运行hive查询并将其结果保存在临时表中，这个表可以被插入更新。
+Upsert实用程序（`HoodieDeltaStreamer`）具有目录结构所需的所有状态，以了解目标表上的提交时间应为多少。
+例如：`/app/incremental-hql/intermediate/{source_table_name}_temp/{last_commit_included}`。
+已注册的Delta Hive表的格式为`{tmpdb}.{source_table}_{last_commit_included}`。
+
+以下是HiveIncrementalPuller的配置选项
+
+| **配置** | **描述** | **默认值** |
+|-------|--------|--------|
+|hiveUrl| 要连接的Hive Server 2的URL | |
+|hiveUser| Hive Server 2 用户名 | |
+|hivePass| Hive Server 2 密码 | |
+|queue| YARN 队列名称 | |
+|tmp| DFS中存储临时增量数据的目录。目录结构将遵循约定。请参阅以下部分。 | |
+|extractSQLFile| 在源表上要执行的提取数据的SQL。提取的数据将是自特定时间点以来已更改的所有行。 | |
+|sourceTable| 源表名称。在Hive环境属性中需要设置。 | |
+|targetTable| 目标表名称。中间存储目录结构需要。 | |
+|sourceDataPath| 源DFS基本路径。这是读取Hudi元数据的地方。 | |
+|targetDataPath| 目标DFS基本路径。 这是计算fromCommitTime所必需的。 如果显式指定了fromCommitTime，则不需要设置这个参数。 | |
+|tmpdb| 用来创建中间临时增量表的数据库 | hoodie_temp |
+|fromCommitTime| 这是最重要的参数。 这是从中提取更改的记录的时间点。 | |
+|maxCommits| 要包含在拉取中的提交数。将此设置为-1将包括从fromCommitTime开始的所有提交。将此设置为大于0的值，将包括在fromCommitTime之后仅更改指定提交次数的记录。如果您需要一次赶上两次提交，则可能需要这样做。| 3 |
+|help| 实用程序帮助 | |
+
+
+设置fromCommitTime=0和maxCommits=-1将提取整个源数据集，可用于启动Backfill。
+如果目标数据集是Hudi数据集，则该实用程序可以确定目标数据集是否没有提交或延迟超过24小时（这是可配置的），
+它将自动使用Backfill配置，因为增量应用最近24小时的更改会比Backfill花费更多的时间。
+该工具当前的局限性在于缺乏在混合模式（正常模式和增量模式）下自联接同一表的支持。
+
+**关于使用Fetch任务执行的Hive查询的说明：**
+由于Fetch任务为每个分区调用InputFormat.listStatus()，每个listStatus()调用都会列出Hoodie元数据。
+为了避免这种情况，如下操作可能是有用的，即使用Hive session属性对增量查询禁用Fetch任务：
+`set hive.fetch.task.conversion = none;`。这将确保Hive查询使用Map Reduce执行，
+合并分区（用逗号分隔），并且对所有这些分区仅调用一次InputFormat.listStatus()。
+
+## Spark
+
+Spark可将Hudi jars和捆绑包轻松部署和管理到作业/笔记本中。简而言之，通过Spark有两种方法可以访问Hudi数据集。
+
+ - **Hudi DataSource**：支持实时视图，读取优化和增量拉取，类似于标准数据源（例如：`spark.read.parquet`）的工作方式。
+ - **以Hive表读取**：支持所有三个视图，包括实时视图，依赖于自定义的Hudi输入格式（再次类似Hive）。
+
+通常，您的spark作业需要依赖`hudi-spark`或`hudi-spark-bundle-x.y.z.jar`，
+它们必须位于驱动程序和执行程序的类路径上（提示：使用`--jars`参数）。
+
+### 读优化表 {#spark-ro-view}
+
+要使用SparkSQL将RO表读取为Hive表，只需按如下所示将路径过滤器推入sparkContext。
+对于Hudi表，该方法保留了Spark内置的读取Parquet文件的优化功能，例如进行矢量化读取。
+
+```scala
+spark.sparkContext.hadoopConfiguration.setClass("mapreduce.input.pathFilter.class", classOf[org.apache.hudi.hadoop.HoodieROTablePathFilter], classOf[org.apache.hadoop.fs.PathFilter]);
+```
+
+如果您希望通过数据源在DFS上使用全局路径，则只需执行以下类似操作即可得到Spark DataFrame。
+
+```scala
+Dataset<Row> hoodieROViewDF = spark.read().format("org.apache.hudi")
+// pass any path glob, can include hudi & non-hudi datasets
+.load("/glob/path/pattern");
+```
+
+### 实时表 {#spark-rt-view}
+将实时表在Spark中作为Hive表进行查询，设置`spark.sql.hive.convertMetastoreParquet = false`，
+迫使Spark回退到使用Hive Serde读取数据（计划/执行仍然是Spark）。
+
+```scala
+$ spark-shell --jars hudi-spark-bundle-x.y.z-SNAPSHOT.jar --driver-class-path /etc/hive/conf --packages com.databricks:spark-avro_2.11:4.0.0 --conf spark.sql.hive.convertMetastoreParquet=false --num-executors 10 --driver-memory 7g --executor-memory 2g --master yarn-client
+
+scala> sqlContext.sql("select count(*) from hudi_rt where datestr = '2016-10-02'").show()
+```
+
+如果您希望通过数据源在DFS上使用全局路径，则只需执行以下类似操作即可得到Spark DataFrame。
+
+```scala
+Dataset<Row> hoodieRealtimeViewDF = spark.read().format("org.apache.hudi")
+// pass any path glob, can include hudi & non-hudi datasets
+.load("/glob/path/pattern");
+```
+
+如果您希望只查询实时表的读优化视图
+
+```scala
+Dataset<Row> hoodieRealtimeViewDF = spark.read().format("org.apache.hudi")
+.option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_READ_OPTIMIZED_OPT_VAL)
+// pass any path glob, can include hudi & non-hudi datasets
+.load("/glob/path/pattern");
+```
+
+### 增量拉取 {#spark-incr-pull}
+`hudi-spark`模块提供了DataSource API，这是一种从Hudi数据集中提取数据并通过Spark处理数据的更优雅的方法。
+如下所示是一个示例增量拉取，它将获取自`beginInstantTime`以来写入的所有记录。
+
+```java
+ Dataset<Row> hoodieIncViewDF = spark.read()
+ .format("org.apache.hudi")
+ .option(DataSourceReadOptions.VIEW_TYPE_OPT_KEY(),
+ DataSourceReadOptions.VIEW_TYPE_INCREMENTAL_OPT_VAL())
+ .option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY(),
+ <beginInstantTime>)
+ .option(DataSourceReadOptions.INCR_PATH_GLOB_OPT_KEY(),
+ "/year=2020/month=*/day=*") // 可选，从指定的分区增量拉取
+ .load(tablePath); // 用数据集的最底层路径
+```
+
+请参阅[设置](/cn/docs/configurations#spark-datasource)部分，以查看所有数据源选项。
+
+另外，`HoodieReadClient`通过Hudi的隐式索引提供了以下功能。
+
+| **API** | **描述** |
+|-------|--------|
+| read(keys) | 使用Hudi自己的索通过快速查找将与键对应的数据作为DataFrame读出 |
+| filterExists() | 从提供的RDD[HoodieRecord]中过滤出已经存在的记录。对删除重复数据有用 |
+| checkExists(keys) | 检查提供的键是否存在于Hudi数据集中 |
+
+
+## PrestoDB
+
+PrestoDB是一种常用的查询引擎，可提供交互式查询性能。 Hudi RO表可以在Presto中无缝查询。
+这需要在整个安装过程中将`hudi-presto-bundle` jar放入`<presto_install>/plugin/hive-hadoop2/`中。
+
+## Impala (3.4 or later)
+
+### 读优化表
+
+Impala可以在HDFS上查询Hudi读优化表，作为一种 [EXTERNAL TABLE](https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_tables#external_tables) 的形式。
+可以通过以下方式在Impala上建立Hudi读优化表:
+```
+CREATE EXTERNAL TABLE database.table_name
+LIKE PARQUET '/path/to/load/xxx.parquet'
+STORED AS HUDIPARQUET
+LOCATION '/path/to/load';
+```
+Impala可以利用合理的文件分区来提高查询的效率。
+如果想要建立分区的表，文件夹命名需要根据此种方式`year=2020/month=1`.
+Impala使用`=`来区分分区名和分区值.
+可以通过以下方式在Impala上建立分区Hudi读优化表:
+```
+CREATE EXTERNAL TABLE database.table_name
+LIKE PARQUET '/path/to/load/xxx.parquet'
+PARTITION BY (year int, month int, day int)
+STORED AS HUDIPARQUET
+LOCATION '/path/to/load';
+ALTER TABLE database.table_name RECOVER PARTITIONS;
+```
+在Hudi成功写入一个新的提交后, 刷新Impala表来得到最新的结果.
+```
+REFRESH database.table_name
+```
+
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/s3_hoodie.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/s3_hoodie.md
new file mode 100644
index 00000000000..1e0b3293c2e
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/s3_hoodie.md
@@ -0,0 +1,81 @@
+---
+title: S3 文件系统
+keywords: [ hudi, hive, aws, s3, spark, presto]
+summary: 在本页中，我们将讨论如何在 S3 文件系统中配置 Hudi 。
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+在本页中，我们将解释如何让你的 Hudi Spark 作业存储到 AWS S3 。
+
+## AWS 配置
+
+
+Hudi 与 S3 的适配需要两项配置：
+
+- 为 Hudi 加 AWS 凭证
+- 将需要的 jar 包添加到类路径
+
+### AWS 凭证
+
+在 S3 上使用 Hudi 的最简单的办法，是为你的 `SparkSession` 或 `SparkContext` 设置 S3 凭证。 Hudi 将自动拾取并通知 S3 。
+
+或者，将需要的配置添加到你的 core-site.xml 文件中， Hudi 可以从那里获取它们。用你的 S3 Bucket 名称替换 `fs.defaultFS` ，之后 Hudi 应该能够从 Bucket 中读取/写入.
+
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>s3://ysharma</value>
+ </property>
+
+ <property>
+ <name>fs.s3.impl</name>
+ <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
+ </property>
+
+ <property>
+ <name>fs.s3.awsAccessKeyId</name>
+ <value>AWS_KEY</value>
+ </property>
+
+ <property>
+ <name>fs.s3.awsSecretAccessKey</name>
+ <value>AWS_SECRET</value>
+ </property>
+
+ <property>
+ <name>fs.s3n.awsAccessKeyId</name>
+ <value>AWS_KEY</value>
+ </property>
+
+ <property>
+ <name>fs.s3n.awsSecretAccessKey</name>
+ <value>AWS_SECRET</value>
+ </property>
+```
+
+
+`hudi-cli` 或 DeltaStreamer 这些工具集能通过 `HOODIE_ENV_` 前缀的环境变量拾取。以下是一个作为示例的基础代码片段，它设置了这些变量并让 CLI 能够在保存在 S3 上的数据集上工作。
+
+```java
+export HOODIE_ENV_fs_DOT_s3a_DOT_access_DOT_key=$accessKey
+export HOODIE_ENV_fs_DOT_s3a_DOT_secret_DOT_key=$secretKey
+export HOODIE_ENV_fs_DOT_s3_DOT_awsAccessKeyId=$accessKey
+export HOODIE_ENV_fs_DOT_s3_DOT_awsSecretAccessKey=$secretKey
+export HOODIE_ENV_fs_DOT_s3n_DOT_awsAccessKeyId=$accessKey
+export HOODIE_ENV_fs_DOT_s3n_DOT_awsSecretAccessKey=$secretKey
+export HOODIE_ENV_fs_DOT_s3n_DOT_impl=org.apache.hadoop.fs.s3a.S3AFileSystem
+```
+
+
+
+### AWS 库
+
+将 AWS Hadoop 库添加到我们的类路径。
+
+ - com.amazonaws:aws-java-sdk:1.10.34
+ - org.apache.hadoop:hadoop-aws:2.7.3
+
+如果使用了 AWS Glue 的数据，则需要 AWS Glue 库。
+
+ - com.amazonaws.glue:aws-glue-datacatalog-hive2-client:1.11.0
+ - com.amazonaws:aws-java-sdk-glue:1.11.475
\ No newline at end of file
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/spark_quick-start-guide.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/spark_quick-start-guide.md
new file mode 100644
index 00000000000..7ced36bf08c
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/spark_quick-start-guide.md
@@ -0,0 +1,449 @@
+---
+title: "Spark Guide"
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+本指南通过使用spark-shell简要介绍了Hudi功能。使用Spark数据源，我们将通过代码段展示如何插入和更新Hudi的默认存储类型数据集：
+[写时复制](/cn/docs/concepts#copy-on-write-storage)。每次写操作之后，我们还将展示如何读取快照和增量数据。
+
+## 设置spark-shell
+Hudi适用于Spark-2.4.3+ & Spark 3.x 版本。您可以按照[此处](https://spark.apache.org/downloads)的说明设置spark。
+在提取的目录中，使用spark-shell运行Hudi：
+
+```scala
+// spark-shell for spark 3
+spark-shell \
+ --packages org.apache.hudi:hudi-spark3-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:3.0.1 \
+ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
+// spark-shell for spark 2 with scala 2.12
+spark-shell \
+ --packages org.apache.hudi:hudi-spark-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:2.4.4 \
+ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
+// spark-shell for spark 2 with scala 2.11
+spark-shell \
+ --packages org.apache.hudi:hudi-spark-bundle_2.11:0.8.0,org.apache.spark:spark-avro_2.11:2.4.4 \
+ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
+```
+
+
+设置表名、基本路径和数据生成器来为本指南生成记录。
+
+```scala
+import org.apache.hudi.QuickstartUtils._
+import scala.collection.JavaConversions._
+import org.apache.spark.sql.SaveMode._
+import org.apache.hudi.DataSourceReadOptions._
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.config.HoodieWriteConfig._
+
+val tableName = "hudi_cow_table"
+val basePath = "file:///tmp/hudi_cow_table"
+val dataGen = new DataGenerator
+```
+
+[数据生成器](https://github.com/apache/hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L50)
+可以基于[行程样本模式](https://github.com/apache/hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L57)
+生成插入和更新的样本。
+
+## 插入数据 {#inserts}
+生成一些新的行程样本，将其加载到DataFrame中，然后将DataFrame写入Hudi数据集中，如下所示。
+
+```scala
+val inserts = convertToStringList(dataGen.generateInserts(10))
+val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
+df.write.format("org.apache.hudi").
+ options(getQuickstartWriteConfigs).
+ option(PRECOMBINE_FIELD_OPT_KEY, "ts").
+ option(RECORDKEY_FIELD_OPT_KEY, "uuid").
+ option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
+ option(TABLE_NAME, tableName).
+ mode(Overwrite).
+ save(basePath);
+```
+
+`mode(Overwrite)`覆盖并重新创建数据集(如果已经存在)。
+您可以检查在`/tmp/hudi_cow_table/<region>/<country>/<city>/`下生成的数据。我们提供了一个记录键
+([schema](#sample-schema)中的`uuid`)，分区字段(`region/country/city`）和组合逻辑([schema](#sample-schema)中的`ts`)
+以确保行程记录在每个分区中都是唯一的。更多信息请参阅
+[对Hudi中的数据进行建模](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)，
+有关将数据提取到Hudi中的方法的信息，请参阅[写入Hudi数据集](/cn/docs/writing_data)。
+这里我们使用默认的写操作：`插入更新`。 如果您的工作负载没有`更新`，也可以使用更快的`插入`或`批量插入`操作。
+想了解更多信息，请参阅[写操作](/cn/docs/writing_data#write-operations)
+
+## 查询数据 {#query}
+
+将数据文件加载到DataFrame中。
+
+```scala
+val roViewDF = spark.
+ read.
+ format("org.apache.hudi").
+ load(basePath + "/*/*/*/*")
+ //load(basePath) 如果使用 "/partitionKey=partitionValue" 文件夹命名格式，Spark将自动识别分区信息
+
+roViewDF.registerTempTable("hudi_ro_table")
+spark.sql("select fare, begin_lon, begin_lat, ts from hudi_ro_table where fare > 20.0").show()
+spark.sql("select _hoodie_commit_time, _hoodie_record_key, _hoodie_partition_path, rider, driver, fare from hudi_ro_table").show()
+```
+
+该查询提供已提取数据的读取优化视图。由于我们的分区路径(`region/country/city`)是嵌套的3个级别
+从基本路径开始，我们使用了`load(basePath + "/*/*/*/*")`。
+有关支持的所有存储类型和视图的更多信息，请参考[存储类型和视图](/cn/docs/concepts#storage-types--views)。
+
+## 更新数据 {#updates}
+
+这类似于插入新数据。使用数据生成器生成对现有行程的更新，加载到DataFrame中并将DataFrame写入hudi数据集。
+
+```scala
+val updates = convertToStringList(dataGen.generateUpdates(10))
+val df = spark.read.json(spark.sparkContext.parallelize(updates, 2));
+df.write.format("org.apache.hudi").
+ options(getQuickstartWriteConfigs).
+ option(PRECOMBINE_FIELD_OPT_KEY, "ts").
+ option(RECORDKEY_FIELD_OPT_KEY, "uuid").
+ option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
+ option(TABLE_NAME, tableName).
+ mode(Append).
+ save(basePath);
+```
+
+注意，保存模式现在为`追加`。通常，除非您是第一次尝试创建数据集，否则请始终使用追加模式。
+[查询](#query)现在再次查询数据将显示更新的行程。每个写操作都会生成一个新的由时间戳表示的[commit](/cn/docs/concepts)
+。在之前提交的相同的`_hoodie_record_key`中寻找`_hoodie_commit_time`, `rider`, `driver`字段变更。
+
+## 增量查询
+
+Hudi还提供了获取给定提交时间戳以来已更改的记录流的功能。
+这可以通过使用Hudi的增量视图并提供所需更改的开始时间来实现。
+如果我们需要给定提交之后的所有更改(这是常见的情况)，则无需指定结束时间。
+
+```scala
+// reload data
+spark.
+ read.
+ format("org.apache.hudi").
+ load(basePath + "/*/*/*/*").
+ createOrReplaceTempView("hudi_ro_table")
+
+val commits = spark.sql("select distinct(_hoodie_commit_time) as commitTime from hudi_ro_table order by commitTime").map(k => k.getString(0)).take(50)
+val beginTime = commits(commits.length - 2) // commit time we are interested in
+
+// 增量查询数据
+val incViewDF = spark.
+ read.
+ format("org.apache.hudi").
+ option(VIEW_TYPE_OPT_KEY, VIEW_TYPE_INCREMENTAL_OPT_VAL).
+ option(BEGIN_INSTANTTIME_OPT_KEY, beginTime).
+ load(basePath);
+incViewDF.registerTempTable("hudi_incr_table")
+spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_incr_table where fare > 20.0").show()
+```
+
+这将提供在开始时间提交之后发生的所有更改，其中包含票价大于20.0的过滤器。关于此功能的独特之处在于，它现在使您可以在批量数据上创作流式管道。
+
+## 特定时间点查询
+
+让我们看一下如何查询特定时间的数据。可以通过将结束时间指向特定的提交时间，将开始时间指向"000"(表示最早的提交时间)来表示特定时间。
+
+```scala
+val beginTime = "000" // Represents all commits > this time.
+val endTime = commits(commits.length - 2) // commit time we are interested in
+
+// 增量查询数据
+val incViewDF = spark.read.format("org.apache.hudi").
+ option(VIEW_TYPE_OPT_KEY, VIEW_TYPE_INCREMENTAL_OPT_VAL).
+ option(BEGIN_INSTANTTIME_OPT_KEY, beginTime).
+ option(END_INSTANTTIME_OPT_KEY, endTime).
+ load(basePath);
+incViewDF.registerTempTable("hudi_incr_table")
+spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_incr_table where fare > 20.0").show()
+```
+
+## 删除数据 {#deletes}
+删除传入的 HoodieKeys 的记录。
+
+```scala
+// spark-shell
+// 获取记录总数
+spark.sql("select uuid, partitionpath from hudi_trips_snapshot").count()
+// 拿到两条将要删除的数据
+val ds = spark.sql("select uuid, partitionpath from hudi_trips_snapshot").limit(2)
+
+// 执行删除
+val deletes = dataGen.generateDeletes(ds.collectAsList())
+val df = spark.read.json(spark.sparkContext.parallelize(deletes, 2))
+
+df.write.format("hudi").
+ options(getQuickstartWriteConfigs).
+ option(OPERATION_OPT_KEY,"delete").
+ option(PRECOMBINE_FIELD_OPT_KEY, "ts").
+ option(RECORDKEY_FIELD_OPT_KEY, "uuid").
+ option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
+ option(TABLE_NAME, tableName).
+ mode(Append).
+ save(basePath)
+
+// 向之前一样运行查询
+val roAfterDeleteViewDF = spark.
+ read.
+ format("hudi").
+ load(basePath + "/*/*/*/*")
+
+roAfterDeleteViewDF.registerTempTable("hudi_trips_snapshot")
+// 应返回 (total - 2) 条记录
+spark.sql("select uuid, partitionpath from hudi_trips_snapshot").count()
+```
+注意: 删除操作只支持 `Append` 模式。
+
+
+
+请查阅写数据页的[删除部分](/cn/docs/writing_data#deletes) 查看更多信息.
+
+
+# Pyspark 示例
+## 设置spark-shell
+
+Hudi适用于Spark-2.4.3+ & Spark 3.x 版本。您可以按照[此处](https://spark.apache.org/downloads)的说明设置spark。
+在提取的目录中，使用spark-shell运行Hudi：
+
+```python
+# pyspark
+export PYSPARK_PYTHON=$(which python3)
+# for spark3
+pyspark \
+ --packages org.apache.hudi:hudi-spark3-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:3.0.1 \
+ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
+# for spark2 with scala 2.12
+pyspark \
+ --packages org.apache.hudi:hudi-spark-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:2.4.4 \
+ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
+# for spark2 with scala 2.11
+pyspark \
+ --packages org.apache.hudi:hudi-spark-bundle_2.11:0.8.0,org.apache.spark:spark-avro_2.11:2.4.4 \
+ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
+```
+
+<div className="notice--info">
+ <h4>请注意以下事项: </h4>
+<ul>
+ <li>需要通过 --packages 指定 spark-avro, 因为默认情况下 spark-shell 不包含该模块</li>
+ <li>spark-avro 和 spark 的版本必须匹配 (上面两个我们都使用了2.4.4)</li>
+ <li>我们使用了基于 scala 2.11 构建的 hudi-spark-bundle, 因为使用的 spark-avro 也是基于 scala 2.11的.
+ 如果使用了 spark-avro_2.12, 相应的, 需要使用 hudi-spark-bundle_2.12. </li>
+</ul>
+</div>
+
+设置表名、基本路径和数据生成器来为本指南生成记录。
+
+```python
+# pyspark
+tableName = "hudi_trips_cow"
+basePath = "file:///tmp/hudi_trips_cow"
+dataGen = sc._jvm.org.apache.hudi.QuickstartUtils.DataGenerator()
+```
+
+[数据生成器](https://github.com/apache/hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L50)
+可以基于[行程样本模式](https://github.com/apache/hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L57)
+生成插入和更新的样本。
+\{: .notice--info}
+
+
+## 插入数据 {#inserts}
+
+生成一些新的行程样本，将其加载到DataFrame中，然后将DataFrame写入Hudi数据集中，如下所示。
+
+```python
+# pyspark
+inserts = sc._jvm.org.apache.hudi.QuickstartUtils.convertToStringList(dataGen.generateInserts(10))
+df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
+
+hudi_options = {
+ 'hoodie.table.name': tableName,
+ 'hoodie.datasource.write.recordkey.field': 'uuid',
+ 'hoodie.datasource.write.partitionpath.field': 'partitionpath',
+ 'hoodie.datasource.write.table.name': tableName,
+ 'hoodie.datasource.write.operation': 'insert',
+ 'hoodie.datasource.write.precombine.field': 'ts',
+ 'hoodie.upsert.shuffle.parallelism': 2,
+ 'hoodie.insert.shuffle.parallelism': 2
+}
+
+df.write.format("hudi"). \
+ options(**hudi_options). \
+ mode("overwrite"). \
+ save(basePath)
+```
+
+`mode(Overwrite)` 覆盖并重新创建数据集(如果已经存在)。
+您可以检查在`/tmp/hudi_cow_table/<region>/<country>/<city>/`下生成的数据。我们提供了一个记录键
+([schema](#sample-schema)中的`uuid`)，分区字段(`region/country/city`）和组合逻辑([schema](#sample-schema)中的`ts`)
+以确保行程记录在每个分区中都是唯一的。更多信息请参阅
+[对Hudi中的数据进行建模](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)，
+有关将数据提取到Hudi中的方法的信息，请参阅[写入Hudi数据集](/cn/docs/writing_data)。
+这里我们使用默认的写操作：`插入更新`。 如果您的工作负载没有`更新`，也可以使用更快的`插入`或`批量插入`操作。
+想了解更多信息，请参阅[写操作](/cn/docs/writing_data#write-operations)
+\{: .notice--info}
+
+## 查询数据 {#query}
+
+将数据文件加载到DataFrame中。
+
+```python
+# pyspark
+tripsSnapshotDF = spark. \
+ read. \
+ format("hudi"). \
+ load(basePath + "/*/*/*/*")
+# load(basePath) use "/partitionKey=partitionValue" folder structure for Spark auto partition discovery
+
+tripsSnapshotDF.createOrReplaceTempView("hudi_trips_snapshot")
+
+spark.sql("select fare, begin_lon, begin_lat, ts from hudi_trips_snapshot where fare > 20.0").show()
+spark.sql("select _hoodie_commit_time, _hoodie_record_key, _hoodie_partition_path, rider, driver, fare from hudi_trips_snapshot").show()
+```
+
+该查询提供已提取数据的读取优化视图。由于我们的分区路径(`region/country/city`)是嵌套的3个级别
+从基本路径开始，我们使用了`load(basePath + "/*/*/*/*")`。
+有关支持的所有存储类型和视图的更多信息，请参考[存储类型和视图](/cn/docs/concepts#storage-types--views)。
+\{: .notice--info}
+
+## 更新数据 {#updates}
+
+这类似于插入新数据。使用数据生成器生成对现有行程的更新，加载到DataFrame中并将DataFrame写入hudi数据集。
+
+```python
+# pyspark
+updates = sc._jvm.org.apache.hudi.QuickstartUtils.convertToStringList(dataGen.generateUpdates(10))
+df = spark.read.json(spark.sparkContext.parallelize(updates, 2))
+df.write.format("hudi"). \
+ options(**hudi_options). \
+ mode("append"). \
+ save(basePath)
+```
+
+Notice that the save mode is now `Append`. In general, always use append mode unless you are trying to create the table for the first time.
+[Querying](#query-data) the data again will now show updated trips. Each write operation generates a new [commit](/docs/concepts)
+denoted by the timestamp. Look for changes in `_hoodie_commit_time`, `rider`, `driver` fields for the same `_hoodie_record_key`s in previous commit.
+
+注意，保存模式现在为`追加`。通常，除非您是第一次尝试创建数据集，否则请始终使用追加模式。
+[查询](#query)现在再次查询数据将显示更新的行程。每个写操作都会生成一个新的由时间戳表示的[commit](/cn/docs/concepts)
+。在之前提交的相同的`_hoodie_record_key`中寻找`_hoodie_commit_time`, `rider`, `driver`字段变更。
+\{: .notice--info}
+
+## 增量查询
+
+Hudi还提供了获取给定提交时间戳以来已更改的记录流的功能。
+这可以通过使用Hudi的增量视图并提供所需更改的开始时间来实现。
+如果我们需要给定提交之后的所有更改(这是常见的情况)，则无需指定结束时间。
+
+```python
+# pyspark
+# 加载数据
+spark. \
+ read. \
+ format("hudi"). \
+ load(basePath + "/*/*/*/*"). \
+ createOrReplaceTempView("hudi_trips_snapshot")
+
+commits = list(map(lambda row: row[0], spark.sql("select distinct(_hoodie_commit_time) as commitTime from hudi_trips_snapshot order by commitTime").limit(50).collect()))
+beginTime = commits[len(commits) - 2] # commit time we are interested in
+
+# 增量的查询数据
+incremental_read_options = {
+ 'hoodie.datasource.query.type': 'incremental',
+ 'hoodie.datasource.read.begin.instanttime': beginTime,
+}
+
+tripsIncrementalDF = spark.read.format("hudi"). \
+ options(**incremental_read_options). \
+ load(basePath)
+tripsIncrementalDF.createOrReplaceTempView("hudi_trips_incremental")
+
+spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
+```
+
+这将提供在开始时间提交之后发生的所有更改，其中包含票价大于20.0的过滤器。关于此功能的独特之处在于，它现在使您可以在批量数据上创作流式管道。
+\{: .notice--info}
+
+## 特定时间点查询
+
+让我们看一下如何查询特定时间的数据。可以通过将结束时间指向特定的提交时间，将开始时间指向"000"(表示最早的提交时间)来表示特定时间。
+
+```python
+# pyspark
+beginTime = "000" # 代表所有大于该时间的 commits.
+endTime = commits[len(commits) - 2]
+
+# 特定时间查询
+point_in_time_read_options = {
+ 'hoodie.datasource.query.type': 'incremental',
+ 'hoodie.datasource.read.end.instanttime': endTime,
+ 'hoodie.datasource.read.begin.instanttime': beginTime
+}
+
+tripsPointInTimeDF = spark.read.format("hudi"). \
+ options(**point_in_time_read_options). \
+ load(basePath)
+
+tripsPointInTimeDF.createOrReplaceTempView("hudi_trips_point_in_time")
+spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_point_in_time where fare > 20.0").show()
+```
+
+## 删除数据 {#deletes}
+删除传入的 HoodieKeys 的记录。
+
+注意: 删除操作只支持 `Append` 模式。
+
+```python
+# pyspark
+# 获取记录总数
+spark.sql("select uuid, partitionpath from hudi_trips_snapshot").count()
+# 拿到两条将被删除的记录
+ds = spark.sql("select uuid, partitionpath from hudi_trips_snapshot").limit(2)
+
+# 执行删除
+hudi_delete_options = {
+ 'hoodie.table.name': tableName,
+ 'hoodie.datasource.write.recordkey.field': 'uuid',
+ 'hoodie.datasource.write.partitionpath.field': 'partitionpath',
+ 'hoodie.datasource.write.table.name': tableName,
+ 'hoodie.datasource.write.operation': 'delete',
+ 'hoodie.datasource.write.precombine.field': 'ts',
+ 'hoodie.upsert.shuffle.parallelism': 2,
+ 'hoodie.insert.shuffle.parallelism': 2
+}
+
+from pyspark.sql.functions import lit
+deletes = list(map(lambda row: (row[0], row[1]), ds.collect()))
+df = spark.sparkContext.parallelize(deletes).toDF(['uuid', 'partitionpath']).withColumn('ts', lit(0.0))
+df.write.format("hudi"). \
+ options(**hudi_delete_options). \
+ mode("append"). \
+ save(basePath)
+
+# 向之前一样运行查询
+roAfterDeleteViewDF = spark. \
+ read. \
+ format("hudi"). \
+ load(basePath + "/*/*/*/*")
+roAfterDeleteViewDF.registerTempTable("hudi_trips_snapshot")
+# 应返回 (total - 2) 条记录
+spark.sql("select uuid, partitionpath from hudi_trips_snapshot").count()
+```
+
+请查阅写数据页的[删除部分](/cn/docs/writing_data#deletes) 查看更多信息.
+
+
+## 从这开始下一步？
+
+您也可以通过[自己构建hudi](https://github.com/apache/hudi#building-apache-hudi-from-source)来快速开始，
+并在spark-shell命令中使用`--jars <path to hudi_code>/packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.1?-*.*.*-SNAPSHOT.jar`，
+而不是`--packages org.apache.hudi:hudi-spark3-bundle_2.12:0.8.0`
+
+
+这里我们使用Spark演示了Hudi的功能。但是，Hudi可以支持多种存储类型/视图，并且可以从Hive，Spark，Presto等查询引擎中查询Hudi数据集。
+我们制作了一个基于Docker设置、所有依赖系统都在本地运行的[演示视频](https://www.youtube.com/watch?v=VhNgUsxdrD0)，
+我们建议您复制相同的设置然后按照[这里](/cn/docs/docker_demo)的步骤自己运行这个演示。
+另外，如果您正在寻找将现有数据迁移到Hudi的方法，请参考[迁移指南](/cn/docs/migration_guide)。
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/use_cases.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/use_cases.md
new file mode 100644
index 00000000000..8ddfc5cbf9d
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/use_cases.md
@@ -0,0 +1,67 @@
+---
+title: 使用案例
+keywords: [ hudi, data ingestion, etl, real time, use cases]
+summary: "Following are some sample use-cases for Hudi, which illustrate the benefits in terms of faster processing & increased efficiency"
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+以下是一些使用Hudi的示例，说明了加快处理速度和提高效率的好处
+
+## 近实时摄取
+
+将外部源(如事件日志、数据库、外部源)的数据摄取到[Hadoop数据湖](http://martinfowler.com/bliki/DataLake)是一个众所周知的问题。
+尽管这些数据对整个组织来说是最有价值的，但不幸的是，在大多数(如果不是全部)Hadoop部署中都使用零散的方式解决，即使用多个不同的摄取工具。
+
+
+对于RDBMS摄取，Hudi提供 __通过更新插入达到更快加载__，而不是昂贵且低效的批量加载。例如，您可以读取MySQL BIN日志或[Sqoop增量导入](https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide#_incremental_imports)并将其应用于
+DFS上的等效Hudi表。这比[批量合并任务](https://sqoop.apache.org/docs/1.4.0-incubating/SqoopUserGuide#id1770457)及[复杂的手工合并工作流](http://hortonworks.com/blog/four-step-strategy-incremental-updates-hive/)更快/更有效率。
+
+
+对于NoSQL数据存储，如[Cassandra](http://cassandra.apache.org/) / [Voldemort](http://www.project-voldemort.com/voldemort/) / [HBase](https://hbase.apache.org/)，即使是中等规模大小也会存储数十亿行。
+毫无疑问， __全量加载不可行__，如果摄取需要跟上较高的更新量，那么则需要更有效的方法。
+
+
+即使对于像[Kafka](http://kafka.apache.org)这样的不可变数据源，Hudi也可以 __强制在HDFS上使用最小文件大小__, 这采取了综合方式解决[HDFS小文件问题](https://blog.cloudera.com/blog/2009/02/the-small-files-problem/)来改善NameNode的健康状况。这对事件流来说更为重要，因为它通常具有较高容量(例如：点击流)，如果管理不当，可能会对Hadoop集群造成严重损害。
+
+在所有源中，通过`commits`这一概念，Hudi增加了以原子方式向消费者发布新数据的功能，这种功能十分必要。
+
+## 近实时分析
+
+通常，实时[数据集市](https://en.wikipedia.org/wiki/Data_mart)由专业(实时)数据分析存储提供支持，例如[Druid](http://druid.io/)或[Memsql](http://www.memsql.com/)或[OpenTSDB](http://opentsdb.net/)。
+这对于较小规模的数据量来说绝对是完美的([相比于这样安装Hadoop](https://blog.twitter.com/2015/hadoop-filesystem-at-twitter))，这种情况需要在亚秒级响应查询，例如系统监控或交互式实时分析。
+但是，由于Hadoop上的数据太陈旧了，通常这些系统会被滥用于非交互式查询，这导致利用率不足和硬件/许可证成本的浪费。
+
+另一方面，Hadoop上的交互式SQL解决方案(如Presto和SparkSQL)表现出色，在 __几秒钟内完成查询__。
+通过将 __数据新鲜度提高到几分钟__，Hudi可以提供一个更有效的替代方案，并支持存储在DFS中的 __数量级更大的数据集__ 的实时分析。
+此外，Hudi没有外部依赖(如专用于实时分析的HBase集群)，因此可以在更新的分析上实现更快的分析，而不会增加操作开销。
+
+
+## 增量处理管道
+
+Hadoop提供的一个基本能力是构建一系列数据集，这些数据集通过表示为工作流的DAG相互派生。
+工作流通常取决于多个上游工作流输出的新数据，新数据的可用性传统上由新的DFS文件夹/Hive分区指示。
+让我们举一个具体的例子来说明这点。上游工作流`U`可以每小时创建一个Hive分区，在每小时结束时(processing_time)使用该小时的数据(event_time)，提供1小时的有效新鲜度。
+然后，下游工作流`D`在`U`结束后立即启动，并在下一个小时内自行处理，将有效延迟时间增加到2小时。
+
+上面的示例忽略了迟到的数据，即`processing_time`和`event_time`分开时。
+不幸的是，在今天的后移动和前物联网世界中，__来自间歇性连接的移动设备和传感器的延迟数据是常态，而不是异常__。
+在这种情况下，保证正确性的唯一补救措施是[重新处理最后几个小时](https://falcon.apache.org/FalconDocumentation#Handling_late_input_data)的数据，
+每小时一遍又一遍，这可能会严重影响整个生态系统的效率。例如; 试想一下，在数百个工作流中每小时重新处理TB数据。
+
+Hudi通过以单个记录为粒度的方式(而不是文件夹/分区)从上游 Hudi数据集`HU`消费新数据(包括迟到数据)，来解决上面的问题。
+应用处理逻辑，并使用下游Hudi数据集`HD`高效更新/协调迟到数据。在这里，`HU`和`HD`可以以更频繁的时间被连续调度
+比如15分钟，并且`HD`提供端到端30分钟的延迟。
+
+为实现这一目标，Hudi采用了类似于[Spark Streaming](https://spark.apache.org/docs/latest/streaming-programming-guide#join-operations)、发布/订阅系统等流处理框架，以及像[Kafka](http://kafka.apache.org/documentation/#theconsumer)
+或[Oracle XStream](https://docs.oracle.com/cd/E11882_01/server.112/e16545/xstrm_cncpt.htm#XSTRM187)等数据库复制技术的类似概念。
+如果感兴趣，可以在[这里](https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop)找到有关增量处理(相比于流处理和批处理)好处的更详细解释。
+
+## DFS的数据分发
+
+一个常用场景是先在Hadoop上处理数据，然后将其分发回在线服务存储层，以供应用程序使用。
+例如，一个Spark管道可以[确定Hadoop上的紧急制动事件](https://eng.uber.com/telematics/)并将它们加载到服务存储层(如ElasticSearch)中，供Uber应用程序使用以增加安全驾驶。这种用例中，通常架构会在Hadoop和服务存储之间引入`队列`，以防止目标服务存储被压垮。
+对于队列的选择，一种流行的选择是Kafka，这个模型经常导致 __在DFS上存储相同数据的冗余(用于计算结果的离线分析)和Kafka(用于分发)__
+
+通过将每次运行的Spark管道更新插入的输出转换为Hudi数据集，Hudi可以再次有效地解决这个问题，然后可以以增量方式获取尾部数据(就像Kafka topic一样)然后写入服务存储层。
diff --git a/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/writing_data.md b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/writing_data.md
new file mode 100644
index 00000000000..f9c84bc1066
--- /dev/null
+++ b/website/i18n/cn/docusaurus-plugin-content-docs/version-1.0.0/writing_data.md
@@ -0,0 +1,222 @@
+---
+title: 写入 Hudi 数据集
+keywords: [ hudi, incremental, batch, stream, processing, Hive, ETL, Spark SQL]
+summary: In this page, we will discuss some available tools for incrementally ingesting & storing data.
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+language: cn
+---
+
+这一节我们将介绍使用[DeltaStreamer](#deltastreamer)工具从外部源甚至其他Hudi数据集摄取新更改的方法，
+以及通过使用[Hudi数据源](#datasource-writer)的upserts加快大型Spark作业的方法。
+对于此类数据集，我们可以使用各种查询引擎[查询](/cn/docs/querying_data)它们。
+
+## 写操作
+
+在此之前，了解Hudi数据源及delta streamer工具提供的三种不同的写操作以及如何最佳利用它们可能会有所帮助。
+这些操作可以在针对数据集发出的每个提交/增量提交中进行选择/更改。
+
+ - **UPSERT（插入更新）** ：这是默认操作，在该操作中，通过查找索引，首先将输入记录标记为插入或更新。
+ 在运行启发式方法以确定如何最好地将这些记录放到存储上，如优化文件大小之后，这些记录最终会被写入。
+ 对于诸如数据库更改捕获之类的用例，建议该操作，因为输入几乎肯定包含更新。
+ - **INSERT（插入）** ：就使用启发式方法确定文件大小而言，此操作与插入更新（UPSERT）非常相似，但此操作完全跳过了索引查找步骤。
+ 因此，对于日志重复数据删除等用例（结合下面提到的过滤重复项的选项），它可以比插入更新快得多。
+ 插入也适用于这种用例，这种情况数据集可以允许重复项，但只需要Hudi的事务写/增量提取/存储管理功能。
+ - **BULK_INSERT（批插入）** ：插入更新和插入操作都将输入记录保存在内存中，以加快存储优化启发式计算的速度（以及其它未提及的方面）。
+ 所以对Hudi数据集进行初始加载/引导时这两种操作会很低效。批量插入提供与插入相同的语义，但同时实现了基于排序的数据写入算法，
+ 该算法可以很好地扩展数百TB的初始负载。但是，相比于插入和插入更新能保证文件大小，批插入在调整文件大小上只能尽力而为。
+
+## DeltaStreamer
+
+`HoodieDeltaStreamer`实用工具 (hudi-utilities-bundle中的一部分) 提供了从DFS或Kafka等不同来源进行摄取的方式，并具有以下功能。
+
+ - 从Kafka单次摄取新事件，从Sqoop、HiveIncrementalPuller输出或DFS文件夹中的多个文件
+ [增量导入](https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide#_incremental_imports)
+ - 支持json、avro或自定义记录类型的传入数据
+ - 管理检查点，回滚和恢复
+ - 利用DFS或Confluent [schema注册表](https://github.com/confluentinc/schema-registry)的Avro模式。
+ - 支持自定义转换操作
+
+命令行选项更详细地描述了这些功能：
+
+```java
+[hoodie]$ spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` --help
+Usage: <main class> [options]
+ Options:
+ --commit-on-errors
+ Commit even when some records failed to be written
+ Default: false
+ --enable-hive-sync
+ Enable syncing to hive
+ Default: false
+ --filter-dupes
+ Should duplicate records from source be dropped/filtered outbefore
+ insert/bulk-insert
+ Default: false
+ --help, -h
+ --hudi-conf
+ Any configuration that can be set in the properties file (using the CLI
+ parameter "--propsFilePath") can also be passed command line using this
+ parameter
+ Default: []
+ --op
+ Takes one of these values : UPSERT (default), INSERT (use when input is
+ purely new data/inserts to gain speed)
+ Default: UPSERT
+ Possible Values: [UPSERT, INSERT, BULK_INSERT]
+ --payload-class
+ subclass of HoodieRecordPayload, that works off a GenericRecord.
+ Implement your own, if you want to do something other than overwriting
+ existing value
+ Default: org.apache.hudi.OverwriteWithLatestAvroPayload
+ --props
+ path to properties file on localfs or dfs, with configurations for
+ Hudi client, schema provider, key generator and data source. For
+ Hudi client props, sane defaults are used, but recommend use to
+ provide basic things like metrics endpoints, hive configs etc. For
+ sources, referto individual classes, for supported properties.
+ Default: file:///Users/vinoth/bin/hoodie/src/test/resources/delta-streamer-config/dfs-source.properties
+ --schemaprovider-class
+ subclass of org.apache.hudi.utilities.schema.SchemaProvider to attach
+ schemas to input & target table data, built in options:
+ FilebasedSchemaProvider
+ Default: org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+ --source-class
+ Subclass of org.apache.hudi.utilities.sources to read data. Built-in
+ options: org.apache.hudi.utilities.sources.{JsonDFSSource (default),
+ AvroDFSSource, JsonKafkaSource, AvroKafkaSource, HiveIncrPullSource}
+ Default: org.apache.hudi.utilities.sources.JsonDFSSource
+ --source-limit
+ Maximum amount of data to read from source. Default: No limit For e.g:
+ DFSSource => max bytes to read, KafkaSource => max events to read
+ Default: 9223372036854775807
+ --source-ordering-field
+ Field within source record to decide how to break ties between records
+ with same key in input data. Default: 'ts' holding unix timestamp of
+ record
+ Default: ts
+ --spark-master
+ spark master to use.
+ Default: local[2]
+ * --target-base-path
+ base path for the target Hudi dataset. (Will be created if did not
+ exist first time around. If exists, expected to be a Hudi dataset)
+ * --target-table
+ name of the target table in Hive
+ --transformer-class
+ subclass of org.apache.hudi.utilities.transform.Transformer. UDF to
+ transform raw source dataset to a target dataset (conforming to target
+ schema) before writing. Default : Not set. E:g -
+ org.apache.hudi.utilities.transform.SqlQueryBasedTransformer (which
+ allows a SQL query template to be passed as a transformation function)
+```
+
+该工具采用层次结构组成的属性文件，并具有可插拔的接口，用于提取数据、生成密钥和提供模式。
+从Kafka和DFS摄取数据的示例配置在这里：`hudi-utilities/src/test/resources/delta-streamer-config`。
+
+例如：当您让Confluent Kafka、Schema注册表启动并运行后，可以用这个命令产生一些测试数据
+（[impressions.avro](https://docs.confluent.io/current/ksql/docs/tutorials/generate-custom-test-data)，
+由schema-registry代码库提供）
+
+```java
+[confluent-5.0.0]$ bin/ksql-datagen schema=../impressions.avro format=avro topic=impressions key=impressionid
+```
+
+然后用如下命令摄取这些数据。
+
+```java
+[hoodie]$ spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` \
+ --props file://${PWD}/hudi-utilities/src/test/resources/delta-streamer-config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --target-base-path file:///tmp/hudi-deltastreamer-op --target-table uber.impressions \
+ --op BULK_INSERT
+```
+
+在某些情况下，您可能需要预先将现有数据集迁移到Hudi。 请参考[迁移指南](/cn/docs/migration_guide)。
+
+## Datasource Writer
+
+`hudi-spark`模块提供了DataSource API，可以将任何DataFrame写入（也可以读取）到Hudi数据集中。
+以下是在指定需要使用的字段名称的之后，如何插入更新DataFrame的方法，这些字段包括
+`recordKey => _row_key`、`partitionPath => partition`和`precombineKey => timestamp`
+
+```java
+inputDF.write()
+ .format("org.apache.hudi")
+ .options(clientOpts) // 可以传入任何Hudi客户端参数
+ .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")
+ .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")
+ .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")
+ .option(HoodieWriteConfig.TABLE_NAME, tableName)
+ .mode(SaveMode.Append)
+ .save(basePath);
+```
+
+## 与Hive同步
+
+上面的两个工具都支持将数据集的最新模式同步到Hive Metastore，以便查询新的列和分区。
+如果需要从命令行或在独立的JVM中运行它，Hudi提供了一个`HiveSyncTool`，
+在构建了hudi-hive模块之后，可以按以下方式调用它。
+
+```java
+cd hudi-hive
+./run_sync_tool.sh
+ [hudi-hive]$ ./run_sync_tool.sh --help
+Usage: <main class> [options]
+ Options:
+ * --base-path
+ Basepath of Hudi dataset to sync
+ * --database
+ name of the target database in Hive
+ --help, -h
+ Default: false
+ * --jdbc-url
+ Hive jdbc connect url
+ * --pass
+ Hive password
+ * --table
+ name of the target table in Hive
+ * --user
+ Hive username
+```
+
+## 删除数据
+
+通过允许用户指定不同的数据记录负载实现，Hudi支持对存储在Hudi数据集中的数据执行两种类型的删除。
+
+ - **Soft Deletes（软删除）** ：使用软删除时，用户希望保留键，但仅使所有其他字段的值都为空。
+ 通过确保适当的字段在数据集模式中可以为空，并在将这些字段设置为null之后直接向数据集插入更新这些记录，即可轻松实现这一点。
+ - **Hard Deletes（硬删除）** ：这种更强形式的删除是从数据集中彻底删除记录在存储上的任何痕迹。
+ 这可以通过触发一个带有自定义负载实现的插入更新来实现，这种实现可以使用总是返回Optional.Empty作为组合值的DataSource或DeltaStreamer。
+ Hudi附带了一个内置的`org.apache.hudi.EmptyHoodieRecordPayload`类，它就是实现了这一功能。
+
+```java
+ deleteDF // 仅包含要删除的记录的DataFrame
+ .write().format("org.apache.hudi")
+ .option(...) // 根据设置需要添加HUDI参数，例如记录键、分区路径和其他参数
+ // 指定record_key，partition_key，precombine_fieldkey和常规参数
+ .option(DataSourceWriteOptions.PAYLOAD_CLASS_OPT_KEY, "org.apache.hudi.EmptyHoodieRecordPayload")
+
+```
+
+## 存储管理
+
+Hudi还对存储在Hudi数据集中的数据执行几个关键的存储管理功能。在DFS上存储数据的关键方面是管理文件大小和数量以及回收存储空间。
+例如，HDFS在处理小文件上性能很差，这会对Name Node的内存及RPC施加很大的压力，并可能破坏整个集群的稳定性。
+通常，查询引擎可在较大的列文件上提供更好的性能，因为它们可以有效地摊销获得列统计信息等的成本。
+即使在某些云数据存储上，列出具有大量小文件的目录也常常比较慢。
+
+以下是一些有效管理Hudi数据集存储的方法。
+
+ - Hudi中的[小文件处理功能](/cn/docs/configurations#compactionSmallFileSize)，可以分析传入的工作负载并将插入内容分配到现有文件组中，
+ 而不是创建新文件组。新文件组会生成小文件。
+ - 可以[配置](/cn/docs/configurations#retainCommits)Cleaner来清理较旧的文件片，清理的程度可以调整，
+ 具体取决于查询所需的最长时间和增量拉取所需的回溯。
+ - 用户还可以调整[基础/parquet文件](/cn/docs/configurations#limitFileSize)、[日志文件](/cn/docs/configurations#logFileMaxSize)的大小
+ 和预期的[压缩率](/cn/docs/configurations#parquetCompressionRatio)，使足够数量的插入被分到同一个文件组中，最终产生大小合适的基础文件。
+ - 智能调整[批插入并行度](/cn/docs/configurations#withBulkInsertParallelism)，可以产生大小合适的初始文件组。
+ 实际上，正确执行此操作非常关键，因为文件组一旦创建后就不能删除，只能如前所述对其进行扩展。
+ - 对于具有大量更新的工作负载，[读取时合并存储](/cn/docs/concepts#merge-on-read-storage)提供了一种很好的机制，
+ 可以快速将其摄取到较小的文件中，之后通过压缩将它们合并为较大的基础文件。
diff --git a/website/releases/download.md b/website/releases/download.md
index 6259ffaa587..f93d0e93e00 100644
--- a/website/releases/download.md
+++ b/website/releases/download.md
@@ -6,6 +6,10 @@ toc: true
last_modified_at: 2022-12-27T15:59:57-04:00
---
+### Release 1.0.0
+* Source Release : [Apache Hudi 1.0.0 Source Release](https://downloads.apache.org/hudi/1.0.0/hudi-1.0.0.src.tgz) ([asc](https://downloads.apache.org/hudi/1.0.0/hudi-1.0.0.src.tgz.asc), [sha512](https://downloads.apache.org/hudi/1.0.0/hudi-1.0.0.src.tgz.sha512))
+* Release Note : ([Release Note for Apache Hudi 1.0.0](/releases/release-1.0.0))
+
### Release 1.0.0-beta2
* Source Release : [Apache Hudi 1.0.0-beta2 Source Release](https://downloads.apache.org/hudi/1.0.0-beta2/hudi-1.0.0-beta2.src.tgz) ([asc](https://downloads.apache.org/hudi/1.0.0-beta2/hudi-1.0.0-beta2.src.tgz.asc), [sha512](https://downloads.apache.org/hudi/1.0.0-beta2/hudi-1.0.0-beta2.src.tgz.sha512))
* Release Note : ([Release Note for Apache Hudi 1.0.0-beta2](/releases/release-1.0.0-beta2))
diff --git a/website/src/components/HomepageFeatures/index.js b/website/src/components/HomepageFeatures/index.js
index 83ca35a8591..89307ec1fa8 100644
--- a/website/src/components/HomepageFeatures/index.js
+++ b/website/src/components/HomepageFeatures/index.js
@@ -21,7 +21,7 @@ const HomepageFeatures = () => {
title: "Mutability support for all workload shapes & sizes",
description:
"Quickly update & delete data with fast, pluggable indexing. This includes database CDC and high-scale streaming data, with best-in-class support for out-of-order records, bursty traffic & data deduplication.",
- link: "/docs/indexing",
+ link: "/docs/indexes",
},
{
icon: IncrementalProcessing,
diff --git a/website/src/components/HomepageHeader/index.js b/website/src/components/HomepageHeader/index.js
index fba104fed57..3100dc17cf5 100644
--- a/website/src/components/HomepageHeader/index.js
+++ b/website/src/components/HomepageHeader/index.js
@@ -19,7 +19,7 @@ function HomepageHeader() {
</h1>
<FeatureRender />
<div className={styles.buttons}>
- <LinkButton to="/releases/release-0.15.0">
+ <LinkButton to="/releases/release-1.0.0">
Latest releases
</LinkButton>
<LinkButton type="secondary" to="/docs/quick-start-guide">
diff --git a/website/versioned_docs/version-0.10.0/file_layouts.md b/website/versioned_docs/version-0.10.0/file_layouts.md
index 41ddca2a6be..dc669b6c249 100644
--- a/website/versioned_docs/version-0.10.0/file_layouts.md
+++ b/website/versioned_docs/version-0.10.0/file_layouts.md
@@ -10,7 +10,7 @@ The following describes the general file layout structure for Apache Hudi
* Each file group contains several file slices
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
-Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.10.0/file_sizing.md b/website/versioned_docs/version-0.10.0/file_sizing.md
index 58831e4b299..08f4a314ef0 100644
--- a/website/versioned_docs/version-0.10.0/file_sizing.md
+++ b/website/versioned_docs/version-0.10.0/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.10.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.10.0/flink-quick-start-guide.md
index 1bbf61f73f2..a1779904348 100644
--- a/website/versioned_docs/version-0.10.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.10.0/flink-quick-start-guide.md
@@ -169,7 +169,7 @@ Check out the [Flink Setup](/docs/0.10.0/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.10.0/overview.mdx b/website/versioned_docs/version-0.10.0/overview.mdx
index 1567d99e56b..f827943c34b 100644
--- a/website/versioned_docs/version-0.10.0/overview.mdx
+++ b/website/versioned_docs/version-0.10.0/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.10.0/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.10.0/use_cases.md b/website/versioned_docs/version-0.10.0/use_cases.md
index 89f322a1eb7..7054048d2d6 100644
--- a/website/versioned_docs/version-0.10.0/use_cases.md
+++ b/website/versioned_docs/version-0.10.0/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.10.0/write_operations.md b/website/versioned_docs/version-0.10.0/write_operations.md
index 952fe3b1196..f3e3e0e1a38 100644
--- a/website/versioned_docs/version-0.10.0/write_operations.md
+++ b/website/versioned_docs/version-0.10.0/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](/docs/file_sizing)
+3. [File Sizing](file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.10.1/file_layouts.md b/website/versioned_docs/version-0.10.1/file_layouts.md
index 41ddca2a6be..7d2af539d57 100644
--- a/website/versioned_docs/version-0.10.1/file_layouts.md
+++ b/website/versioned_docs/version-0.10.1/file_layouts.md
@@ -11,6 +11,6 @@ The following describes the general file layout structure for Apache Hudi
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.10.1/file_sizing.md b/website/versioned_docs/version-0.10.1/file_sizing.md
index 58831e4b299..08f4a314ef0 100644
--- a/website/versioned_docs/version-0.10.1/file_sizing.md
+++ b/website/versioned_docs/version-0.10.1/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.10.1/flink-quick-start-guide.md b/website/versioned_docs/version-0.10.1/flink-quick-start-guide.md
index b9afcc679a5..225ead873cd 100644
--- a/website/versioned_docs/version-0.10.1/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.10.1/flink-quick-start-guide.md
@@ -169,7 +169,7 @@ Check out the [Flink Setup](/docs/0.10.1/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.10.1/overview.mdx b/website/versioned_docs/version-0.10.1/overview.mdx
index 2040014979a..2a166cccd6a 100644
--- a/website/versioned_docs/version-0.10.1/overview.mdx
+++ b/website/versioned_docs/version-0.10.1/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.10.1/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.10.1/use_cases.md b/website/versioned_docs/version-0.10.1/use_cases.md
index 89f322a1eb7..989cde1bc3e 100644
--- a/website/versioned_docs/version-0.10.1/use_cases.md
+++ b/website/versioned_docs/version-0.10.1/use_cases.md
@@ -127,12 +127,12 @@ cost savings for your data lake.
Some examples of the Apache Hudi services that make this performance optimization easy include:
-- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
+- [Auto File Sizing](file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.10.1/write_operations.md b/website/versioned_docs/version-0.10.1/write_operations.md
index 952fe3b1196..f3e3e0e1a38 100644
--- a/website/versioned_docs/version-0.10.1/write_operations.md
+++ b/website/versioned_docs/version-0.10.1/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](/docs/file_sizing)
+3. [File Sizing](file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.11.0/file_layouts.md b/website/versioned_docs/version-0.11.0/file_layouts.md
index 41ddca2a6be..7d2af539d57 100644
--- a/website/versioned_docs/version-0.11.0/file_layouts.md
+++ b/website/versioned_docs/version-0.11.0/file_layouts.md
@@ -11,6 +11,6 @@ The following describes the general file layout structure for Apache Hudi
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.11.0/file_sizing.md b/website/versioned_docs/version-0.11.0/file_sizing.md
index 58831e4b299..08f4a314ef0 100644
--- a/website/versioned_docs/version-0.11.0/file_sizing.md
+++ b/website/versioned_docs/version-0.11.0/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.11.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.11.0/flink-quick-start-guide.md
index 9c04bff57d3..e8dd5d106a3 100644
--- a/website/versioned_docs/version-0.11.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.11.0/flink-quick-start-guide.md
@@ -176,7 +176,7 @@ Check out the [Flink Setup](/docs/0.11.0/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.11.0/overview.mdx b/website/versioned_docs/version-0.11.0/overview.mdx
index f17967539a9..f2ff5120a97 100644
--- a/website/versioned_docs/version-0.11.0/overview.mdx
+++ b/website/versioned_docs/version-0.11.0/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.11.0/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.11.0/s3_hoodie.md b/website/versioned_docs/version-0.11.0/s3_hoodie.md
index a67de25b821..0e550000d97 100644
--- a/website/versioned_docs/version-0.11.0/s3_hoodie.md
+++ b/website/versioned_docs/version-0.11.0/s3_hoodie.md
@@ -81,7 +81,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.11.0/use_cases.md b/website/versioned_docs/version-0.11.0/use_cases.md
index dcba917b9cc..926f84a69cc 100644
--- a/website/versioned_docs/version-0.11.0/use_cases.md
+++ b/website/versioned_docs/version-0.11.0/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.11.0/write_operations.md b/website/versioned_docs/version-0.11.0/write_operations.md
index 9ff8431384c..632230055fd 100644
--- a/website/versioned_docs/version-0.11.0/write_operations.md
+++ b/website/versioned_docs/version-0.11.0/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
3. [File Sizing](/docs/file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.11.1/file_layouts.md b/website/versioned_docs/version-0.11.1/file_layouts.md
index 41ddca2a6be..7d2af539d57 100644
--- a/website/versioned_docs/version-0.11.1/file_layouts.md
+++ b/website/versioned_docs/version-0.11.1/file_layouts.md
@@ -11,6 +11,6 @@ The following describes the general file layout structure for Apache Hudi
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.11.1/file_sizing.md b/website/versioned_docs/version-0.11.1/file_sizing.md
index 58831e4b299..08f4a314ef0 100644
--- a/website/versioned_docs/version-0.11.1/file_sizing.md
+++ b/website/versioned_docs/version-0.11.1/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.11.1/flink-quick-start-guide.md b/website/versioned_docs/version-0.11.1/flink-quick-start-guide.md
index 07f142efff0..420f1f5b711 100644
--- a/website/versioned_docs/version-0.11.1/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.11.1/flink-quick-start-guide.md
@@ -176,7 +176,7 @@ Check out the [Flink Setup](/docs/0.11.1/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.11.1/overview.mdx b/website/versioned_docs/version-0.11.1/overview.mdx
index c84fc036826..20536fccd5c 100644
--- a/website/versioned_docs/version-0.11.1/overview.mdx
+++ b/website/versioned_docs/version-0.11.1/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.11.1/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.11.1/s3_hoodie.md b/website/versioned_docs/version-0.11.1/s3_hoodie.md
index a67de25b821..0e550000d97 100644
--- a/website/versioned_docs/version-0.11.1/s3_hoodie.md
+++ b/website/versioned_docs/version-0.11.1/s3_hoodie.md
@@ -81,7 +81,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.11.1/use_cases.md b/website/versioned_docs/version-0.11.1/use_cases.md
index dcba917b9cc..926f84a69cc 100644
--- a/website/versioned_docs/version-0.11.1/use_cases.md
+++ b/website/versioned_docs/version-0.11.1/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.11.1/write_operations.md b/website/versioned_docs/version-0.11.1/write_operations.md
index 9ff8431384c..632230055fd 100644
--- a/website/versioned_docs/version-0.11.1/write_operations.md
+++ b/website/versioned_docs/version-0.11.1/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
3. [File Sizing](/docs/file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.12.0/file_layouts.md b/website/versioned_docs/version-0.12.0/file_layouts.md
index 41ddca2a6be..7d2af539d57 100644
--- a/website/versioned_docs/version-0.12.0/file_layouts.md
+++ b/website/versioned_docs/version-0.12.0/file_layouts.md
@@ -11,6 +11,6 @@ The following describes the general file layout structure for Apache Hudi
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.12.0/file_sizing.md b/website/versioned_docs/version-0.12.0/file_sizing.md
index 1c1c12fe207..b43ccdaa8b3 100644
--- a/website/versioned_docs/version-0.12.0/file_sizing.md
+++ b/website/versioned_docs/version-0.12.0/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
diff --git a/website/versioned_docs/version-0.12.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.12.0/flink-quick-start-guide.md
index 75bae79386e..49fa8d58213 100644
--- a/website/versioned_docs/version-0.12.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.12.0/flink-quick-start-guide.md
@@ -324,7 +324,7 @@ Check out the [Flink Setup](/docs/0.12.0/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.12.0/overview.mdx b/website/versioned_docs/version-0.12.0/overview.mdx
index 8318fad875f..228d4024727 100644
--- a/website/versioned_docs/version-0.12.0/overview.mdx
+++ b/website/versioned_docs/version-0.12.0/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.12.0/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.12.0/s3_hoodie.md b/website/versioned_docs/version-0.12.0/s3_hoodie.md
index 876afc04658..04b9ff75ccc 100644
--- a/website/versioned_docs/version-0.12.0/s3_hoodie.md
+++ b/website/versioned_docs/version-0.12.0/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
diff --git a/website/versioned_docs/version-0.12.0/use_cases.md b/website/versioned_docs/version-0.12.0/use_cases.md
index dcba917b9cc..926f84a69cc 100644
--- a/website/versioned_docs/version-0.12.0/use_cases.md
+++ b/website/versioned_docs/version-0.12.0/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.12.0/write_operations.md b/website/versioned_docs/version-0.12.0/write_operations.md
index 9ff8431384c..632230055fd 100644
--- a/website/versioned_docs/version-0.12.0/write_operations.md
+++ b/website/versioned_docs/version-0.12.0/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
3. [File Sizing](/docs/file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.12.1/file_layouts.md b/website/versioned_docs/version-0.12.1/file_layouts.md
index 41ddca2a6be..7d2af539d57 100644
--- a/website/versioned_docs/version-0.12.1/file_layouts.md
+++ b/website/versioned_docs/version-0.12.1/file_layouts.md
@@ -11,6 +11,6 @@ The following describes the general file layout structure for Apache Hudi
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.12.1/file_sizing.md b/website/versioned_docs/version-0.12.1/file_sizing.md
index 1c1c12fe207..b43ccdaa8b3 100644
--- a/website/versioned_docs/version-0.12.1/file_sizing.md
+++ b/website/versioned_docs/version-0.12.1/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
diff --git a/website/versioned_docs/version-0.12.1/flink-quick-start-guide.md b/website/versioned_docs/version-0.12.1/flink-quick-start-guide.md
index e31ad421667..754eb199271 100644
--- a/website/versioned_docs/version-0.12.1/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.12.1/flink-quick-start-guide.md
@@ -324,7 +324,7 @@ Check out the [Flink Setup](/docs/0.12.1/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.12.1/overview.mdx b/website/versioned_docs/version-0.12.1/overview.mdx
index 8819a5846f4..be15a8c626f 100644
--- a/website/versioned_docs/version-0.12.1/overview.mdx
+++ b/website/versioned_docs/version-0.12.1/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.12.1/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.12.1/s3_hoodie.md b/website/versioned_docs/version-0.12.1/s3_hoodie.md
index 876afc04658..04b9ff75ccc 100644
--- a/website/versioned_docs/version-0.12.1/s3_hoodie.md
+++ b/website/versioned_docs/version-0.12.1/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
diff --git a/website/versioned_docs/version-0.12.1/use_cases.md b/website/versioned_docs/version-0.12.1/use_cases.md
index dcba917b9cc..926f84a69cc 100644
--- a/website/versioned_docs/version-0.12.1/use_cases.md
+++ b/website/versioned_docs/version-0.12.1/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.12.1/write_operations.md b/website/versioned_docs/version-0.12.1/write_operations.md
index 9ff8431384c..632230055fd 100644
--- a/website/versioned_docs/version-0.12.1/write_operations.md
+++ b/website/versioned_docs/version-0.12.1/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
3. [File Sizing](/docs/file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.12.2/file_layouts.md b/website/versioned_docs/version-0.12.2/file_layouts.md
index 41ddca2a6be..7d2af539d57 100644
--- a/website/versioned_docs/version-0.12.2/file_layouts.md
+++ b/website/versioned_docs/version-0.12.2/file_layouts.md
@@ -11,6 +11,6 @@ The following describes the general file layout structure for Apache Hudi
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.12.2/file_sizing.md b/website/versioned_docs/version-0.12.2/file_sizing.md
index 58831e4b299..08f4a314ef0 100644
--- a/website/versioned_docs/version-0.12.2/file_sizing.md
+++ b/website/versioned_docs/version-0.12.2/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.12.2/flink-quick-start-guide.md b/website/versioned_docs/version-0.12.2/flink-quick-start-guide.md
index 3d0944ccd2b..1ca2c75f251 100644
--- a/website/versioned_docs/version-0.12.2/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.12.2/flink-quick-start-guide.md
@@ -329,7 +329,7 @@ Check out the [Flink Setup](/docs/0.12.2/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.12.2/overview.mdx b/website/versioned_docs/version-0.12.2/overview.mdx
index 3ecc28b9ff1..aec3c4f8eb9 100644
--- a/website/versioned_docs/version-0.12.2/overview.mdx
+++ b/website/versioned_docs/version-0.12.2/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.12.2/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.12.2/s3_hoodie.md b/website/versioned_docs/version-0.12.2/s3_hoodie.md
index 044b201c7cb..99e23f0d59f 100644
--- a/website/versioned_docs/version-0.12.2/s3_hoodie.md
+++ b/website/versioned_docs/version-0.12.2/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.12.2/use_cases.md b/website/versioned_docs/version-0.12.2/use_cases.md
index dcba917b9cc..926f84a69cc 100644
--- a/website/versioned_docs/version-0.12.2/use_cases.md
+++ b/website/versioned_docs/version-0.12.2/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.12.2/write_operations.md b/website/versioned_docs/version-0.12.2/write_operations.md
index 9ff8431384c..632230055fd 100644
--- a/website/versioned_docs/version-0.12.2/write_operations.md
+++ b/website/versioned_docs/version-0.12.2/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
3. [File Sizing](/docs/file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.12.3/file_layouts.md b/website/versioned_docs/version-0.12.3/file_layouts.md
index 41ddca2a6be..7d2af539d57 100644
--- a/website/versioned_docs/version-0.12.3/file_layouts.md
+++ b/website/versioned_docs/version-0.12.3/file_layouts.md
@@ -11,6 +11,6 @@ The following describes the general file layout structure for Apache Hudi
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.12.3/file_sizing.md b/website/versioned_docs/version-0.12.3/file_sizing.md
index 58831e4b299..08f4a314ef0 100644
--- a/website/versioned_docs/version-0.12.3/file_sizing.md
+++ b/website/versioned_docs/version-0.12.3/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.12.3/flink-quick-start-guide.md b/website/versioned_docs/version-0.12.3/flink-quick-start-guide.md
index 17951814522..59af3a4dc1c 100644
--- a/website/versioned_docs/version-0.12.3/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.12.3/flink-quick-start-guide.md
@@ -329,7 +329,7 @@ Check out the [Flink Setup](/docs/0.12.3/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.12.3/overview.mdx b/website/versioned_docs/version-0.12.3/overview.mdx
index d0ad5ed610b..964406bbf9d 100644
--- a/website/versioned_docs/version-0.12.3/overview.mdx
+++ b/website/versioned_docs/version-0.12.3/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.12.3/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.12.3/s3_hoodie.md b/website/versioned_docs/version-0.12.3/s3_hoodie.md
index 044b201c7cb..99e23f0d59f 100644
--- a/website/versioned_docs/version-0.12.3/s3_hoodie.md
+++ b/website/versioned_docs/version-0.12.3/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.12.3/use_cases.md b/website/versioned_docs/version-0.12.3/use_cases.md
index dcba917b9cc..926f84a69cc 100644
--- a/website/versioned_docs/version-0.12.3/use_cases.md
+++ b/website/versioned_docs/version-0.12.3/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.12.3/write_operations.md b/website/versioned_docs/version-0.12.3/write_operations.md
index 9ff8431384c..632230055fd 100644
--- a/website/versioned_docs/version-0.12.3/write_operations.md
+++ b/website/versioned_docs/version-0.12.3/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
3. [File Sizing](/docs/file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.13.0/file_layouts.md b/website/versioned_docs/version-0.13.0/file_layouts.md
index 41ddca2a6be..7d2af539d57 100644
--- a/website/versioned_docs/version-0.13.0/file_layouts.md
+++ b/website/versioned_docs/version-0.13.0/file_layouts.md
@@ -11,6 +11,6 @@ The following describes the general file layout structure for Apache Hudi
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.13.0/file_sizing.md b/website/versioned_docs/version-0.13.0/file_sizing.md
index 58831e4b299..08f4a314ef0 100644
--- a/website/versioned_docs/version-0.13.0/file_sizing.md
+++ b/website/versioned_docs/version-0.13.0/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.13.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.13.0/flink-quick-start-guide.md
index 8cae9919bc0..efc7e0a89c5 100644
--- a/website/versioned_docs/version-0.13.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.13.0/flink-quick-start-guide.md
@@ -339,7 +339,7 @@ Check out the [Flink Setup](/docs/0.13.0/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.13.0/overview.mdx b/website/versioned_docs/version-0.13.0/overview.mdx
index 5fd332d94b3..0afacbb4204 100644
--- a/website/versioned_docs/version-0.13.0/overview.mdx
+++ b/website/versioned_docs/version-0.13.0/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.13.0/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.13.0/s3_hoodie.md b/website/versioned_docs/version-0.13.0/s3_hoodie.md
index 044b201c7cb..99e23f0d59f 100644
--- a/website/versioned_docs/version-0.13.0/s3_hoodie.md
+++ b/website/versioned_docs/version-0.13.0/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.13.0/use_cases.md b/website/versioned_docs/version-0.13.0/use_cases.md
index dcba917b9cc..926f84a69cc 100644
--- a/website/versioned_docs/version-0.13.0/use_cases.md
+++ b/website/versioned_docs/version-0.13.0/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.13.0/write_operations.md b/website/versioned_docs/version-0.13.0/write_operations.md
index 9ff8431384c..632230055fd 100644
--- a/website/versioned_docs/version-0.13.0/write_operations.md
+++ b/website/versioned_docs/version-0.13.0/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
3. [File Sizing](/docs/file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.13.1/file_layouts.md b/website/versioned_docs/version-0.13.1/file_layouts.md
index 41ddca2a6be..7d2af539d57 100644
--- a/website/versioned_docs/version-0.13.1/file_layouts.md
+++ b/website/versioned_docs/version-0.13.1/file_layouts.md
@@ -11,6 +11,6 @@ The following describes the general file layout structure for Apache Hudi
* Each slice contains a base file (*.parquet) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.13.1/file_sizing.md b/website/versioned_docs/version-0.13.1/file_sizing.md
index 58831e4b299..08f4a314ef0 100644
--- a/website/versioned_docs/version-0.13.1/file_sizing.md
+++ b/website/versioned_docs/version-0.13.1/file_sizing.md
@@ -50,4 +50,4 @@ small files together in the background to generate larger files. NOTE that durin
*Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will
always create a newer version of the smaller file, resulting in 2 versions of the same file.
-The [cleaner service](/docs/hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
+The [cleaner service](hoodie_cleaner) will later kick in and delete the older version small file and keep the latest one.*
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.13.1/flink-quick-start-guide.md b/website/versioned_docs/version-0.13.1/flink-quick-start-guide.md
index e30598a9e23..621945b6e25 100644
--- a/website/versioned_docs/version-0.13.1/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.13.1/flink-quick-start-guide.md
@@ -340,7 +340,7 @@ Check out the [Flink Setup](/docs/0.13.1/flink_configuration) how-to page for de
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.13.1/overview.mdx b/website/versioned_docs/version-0.13.1/overview.mdx
index f0a63c4d5c7..79c0eec7cb3 100644
--- a/website/versioned_docs/version-0.13.1/overview.mdx
+++ b/website/versioned_docs/version-0.13.1/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/0.13.1/table_management),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.13.1/s3_hoodie.md b/website/versioned_docs/version-0.13.1/s3_hoodie.md
index 044b201c7cb..99e23f0d59f 100644
--- a/website/versioned_docs/version-0.13.1/s3_hoodie.md
+++ b/website/versioned_docs/version-0.13.1/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.13.1/use_cases.md b/website/versioned_docs/version-0.13.1/use_cases.md
index dcba917b9cc..926f84a69cc 100644
--- a/website/versioned_docs/version-0.13.1/use_cases.md
+++ b/website/versioned_docs/version-0.13.1/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.13.1/write_operations.md b/website/versioned_docs/version-0.13.1/write_operations.md
index 9ff8431384c..632230055fd 100644
--- a/website/versioned_docs/version-0.13.1/write_operations.md
+++ b/website/versioned_docs/version-0.13.1/write_operations.md
@@ -39,20 +39,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
3. [File Sizing](/docs/file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.14.0/file_layouts.md b/website/versioned_docs/version-0.14.0/file_layouts.md
index 81453786fe4..3a535b83b98 100644
--- a/website/versioned_docs/version-0.14.0/file_layouts.md
+++ b/website/versioned_docs/version-0.14.0/file_layouts.md
@@ -11,7 +11,7 @@ The following describes the general file layout structure for Apache Hudi. Pleas
* Each slice contains a base file (*.parquet/*.orc) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/configurations/#hoodietablebasefileformat) ) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
diff --git a/website/versioned_docs/version-0.14.0/file_sizing.md b/website/versioned_docs/version-0.14.0/file_sizing.md
index b0a1d281623..467c7f734a9 100644
--- a/website/versioned_docs/version-0.14.0/file_sizing.md
+++ b/website/versioned_docs/version-0.14.0/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](/docs/hoodie_cleaner)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md
index c64f84a4e60..555b5d85983 100644
--- a/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md
@@ -374,7 +374,7 @@ Check out the [Flink Setup](/docs/flink_tuning) how-to page for deeper dive into
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.14.0/overview.mdx b/website/versioned_docs/version-0.14.0/overview.mdx
index e615df6bc67..71a84591ce5 100644
--- a/website/versioned_docs/version-0.14.0/overview.mdx
+++ b/website/versioned_docs/version-0.14.0/overview.mdx
@@ -14,7 +14,7 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced “hoodie”) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](/docs/sql_ddl),
-[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](/docs/indexing),
+[transactions](/docs/timeline), [efficient upserts/deletes](/docs/write_operations), [advanced indexes](indexing),
[streaming ingestion services](/docs/hoodie_streaming_ingestion), data [clustering](/docs/clustering)/[compaction](/docs/compaction) optimizations,
and [concurrency](/docs/concurrency_control) all while keeping your data in open source file formats.
@@ -31,7 +31,7 @@ the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- [Hudi Timeline](/docs/timeline) – How Hudi manages transactions and other table services
-- [Hudi File Layout](/docs/file_layouts) - How the files are laid out on storage
+- [Hudi File Layout](file_layouts) - How the files are laid out on storage
- [Hudi Table Types](/docs/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- [Hudi Query Types](/docs/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
diff --git a/website/versioned_docs/version-0.14.0/s3_hoodie.md b/website/versioned_docs/version-0.14.0/s3_hoodie.md
index 37f79ae7534..b990add7d4b 100644
--- a/website/versioned_docs/version-0.14.0/s3_hoodie.md
+++ b/website/versioned_docs/version-0.14.0/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.14.0/use_cases.md b/website/versioned_docs/version-0.14.0/use_cases.md
index 0e6e38102ab..893aa653f5e 100644
--- a/website/versioned_docs/version-0.14.0/use_cases.md
+++ b/website/versioned_docs/version-0.14.0/use_cases.md
@@ -130,9 +130,9 @@ Some examples of the Apache Hudi services that make this performance optimizatio
- [Auto File Sizing](/docs/file_sizing) - to solve the "small files" problem.
- [Clustering](/docs/clustering) - to co-locate data next to each other.
- [Compaction](/docs/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/indexing) - for efficient upserts and deletes.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.14.0/write_operations.md b/website/versioned_docs/version-0.14.0/write_operations.md
index 29132a38f5a..3df12fd3b90 100644
--- a/website/versioned_docs/version-0.14.0/write_operations.md
+++ b/website/versioned_docs/version-0.14.0/write_operations.md
@@ -88,20 +88,20 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/indexing)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
3. [File Sizing](/docs/file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/file_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/indexing)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/hoodie_cleaner) (if needed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
diff --git a/website/versioned_docs/version-0.14.1/s3_hoodie.md b/website/versioned_docs/version-0.14.1/s3_hoodie.md
index 37f79ae7534..b990add7d4b 100644
--- a/website/versioned_docs/version-0.14.1/s3_hoodie.md
+++ b/website/versioned_docs/version-0.14.1/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.15.0/s3_hoodie.md b/website/versioned_docs/version-0.15.0/s3_hoodie.md
index 37f79ae7534..b990add7d4b 100644
--- a/website/versioned_docs/version-0.15.0/s3_hoodie.md
+++ b/website/versioned_docs/version-0.15.0/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](https://hudi.apache.orghoodie_cleaner) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-1.0.0/azure_hoodie.md b/website/versioned_docs/version-1.0.0/azure_hoodie.md
new file mode 100644
index 00000000000..f28ec609c70
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/azure_hoodie.md
@@ -0,0 +1,50 @@
+---
+title: Microsoft Azure
+keywords: [ hudi, hive, azure, spark, presto]
+summary: In this page, we go over how to configure Hudi with Azure filesystem.
+last_modified_at: 2020-05-25T19:00:57-04:00
+---
+In this page, we explain how to use Hudi on Microsoft Azure.
+
+## Disclaimer
+
+This page is maintained by the Hudi community.
+If the information is inaccurate or you have additional information to add.
+Please feel free to create a JIRA ticket. Contribution is highly appreciated.
+
+## Supported Storage System
+
+There are two storage systems support Hudi .
+
+- Azure Blob Storage
+- Azure Data Lake Gen 2
+
+## Verified Combination of Spark and storage system
+
+#### HDInsight Spark2.4 on Azure Data Lake Storage Gen 2
+This combination works out of the box. No extra config needed.
+
+#### Databricks Spark2.4 on Azure Data Lake Storage Gen 2
+- Import Hudi jar to databricks workspace
+
+- Mount the file system to dbutils.
+ ```scala
+ dbutils.fs.mount(
+ source = "abfss://xxx@xxx.dfs.core.windows.net",
+ mountPoint = "/mountpoint",
+ extraConfigs = configs)
+ ```
+- When writing Hudi dataset, use abfss URL
+ ```scala
+ inputDF.write
+ .format("org.apache.hudi")
+ .options(opts)
+ .mode(SaveMode.Append)
+ .save("abfss://<<storage-account>>.dfs.core.windows.net/hudi-tables/customer")
+ ```
+- When reading Hudi dataset, use the mounting point
+ ```scala
+ spark.read
+ .format("org.apache.hudi")
+ .load("/mountpoint/hudi-tables/customer")
+ ```
diff --git a/website/versioned_docs/version-1.0.0/basic_configurations.md b/website/versioned_docs/version-1.0.0/basic_configurations.md
new file mode 100644
index 00000000000..84f589d84db
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/basic_configurations.md
@@ -0,0 +1,797 @@
+---
+title: Basic Configurations
+summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](/docs/configurations) page.
+last_modified_at: 2024-12-06T17:38:05.854
+---
+
+
+This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](/docs/configurations) page.
+
+- [**Hudi Table Config**](#TABLE_CONFIG): Basic Hudi Table configuration parameters.
+- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
+- [**Flink Sql Configs**](#FLINK_SQL): These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
+- [**Write Client Configs**](#WRITE_CLIENT): Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.
+- [**Metastore and Catalog Sync Configs**](#META_SYNC): Configurations used by the Hudi to sync metadata to external metastores and catalogs.
+- [**Metrics Configs**](#METRICS): These set of configs are used to enable monitoring and reporting of key Hudi stats and metrics.
+- [**Kafka Connect Configs**](#KAFKA_CONNECT): These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables
+- [**Hudi Streamer Configs**](#HUDI_STREAMER): These set of configs are used for Hudi Streamer utility which provides the way to ingest from different sources such as DFS or Kafka.
+
+:::note
+In the tables below **(N/A)** means there is no default value set
+:::
+
+## Hudi Table Config {#TABLE_CONFIG}
+Basic Hudi Table configuration parameters.
+
+
+### Hudi Table Basic Configs {#Hudi-Table-Basic-Configs}
+Configurations of the Hudi Table like type of ingestion, storage formats, hive table name etc. Configurations are loaded from hoodie.properties, these properties are usually set during initializing a path as hoodie base path and never changes during the lifetime of a hoodie table.
+
+
+
+
+[**Basic Configs**](#Hudi-Table-Basic-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BOOTSTRAP_BASE_PATH` [...]
+| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | (N/A) | Payload class to use for performing merges, compactions, i.e merge delta logs with current base file and then produce a new base file.<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name. If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME` [...]
+| [hoodie.record.merge.strategy.id](#hoodierecordmergestrategyid) | (N/A) | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in `hoodie.write.record.merge.custom.implementation.classes` which has the same merger strategy id<br />`Config Param: RECORD_MERGE_STRATEGY_ID`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.<br />`Config Param: TABLE_CHECKSUM`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.create.schema](#hoodietablecreateschema) | (N/A) | Schema used when creating the table<br />`Config Param: CREATE_SCHEMA` [...]
+| [hoodie.table.index.defs.path](#hoodietableindexdefspath) | (N/A) | Relative path to table base path where the index definitions are stored<br />`Config Param: RELATIVE_INDEX_DEFINITION_PATH`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.keygenerator.class](#hoodietablekeygeneratorclass) | (N/A) | Key Generator class property for the hoodie table<br />`Config Param: KEY_GENERATOR_CLASS_NAME` [...]
+| [hoodie.table.keygenerator.type](#hoodietablekeygeneratortype) | (N/A) | Key Generator type to determine key generator class<br />`Config Param: KEY_GENERATOR_TYPE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.metadata.partitions](#hoodietablemetadatapartitions) | (N/A) | Comma-separated list of metadata partitions that have been completely built and in-sync with data table. These partitions are ready for use by the readers<br />`Config Param: TABLE_METADATA_PARTITIONS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.metadata.partitions.inflight](#hoodietablemetadatapartitionsinflight) | (N/A) | Comma-separated list of metadata partitions whose building is in progress. These partitions are not yet ready for use by the readers.<br />`Config Param: TABLE_METADATA_PARTITIONS_INFLIGHT`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: NAME` [...]
+| [hoodie.table.partition.fields](#hoodietablepartitionfields) | (N/A) | Comma separated field names used to partition the table. These field names also include the partition type which is used by custom key generators<br />`Config Param: PARTITION_FIELDS` [...]
+| [hoodie.table.precombine.field](#hoodietableprecombinefield) | (N/A) | Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.table.recordkey.fields](#hoodietablerecordkeyfields) | (N/A) | Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br />`Config Param: RECORDKEY_FIELDS` [...]
+| [hoodie.table.secondary.indexes.metadata](#hoodietablesecondaryindexesmetadata) | (N/A) | The metadata of secondary indexes<br />`Config Param: SECONDARY_INDEXES_METADATA`<br />`Since Version: 0.13.0` [...]
+| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | (N/A) | Version of timeline used, by the table.<br />`Config Param: TIMELINE_LAYOUT_VERSION` [...]
+| [hoodie.archivelog.folder](#hoodiearchivelogfolder) | archived | path under the meta folder, to store archived timeline instants at.<br />`Config Param: ARCHIVELOG_FOLDER` [...]
+| [hoodie.bootstrap.index.class](#hoodiebootstrapindexclass) | org.apache.hudi.common.bootstrap.index.hfile.HFileBootstrapIndex | Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_CLASS_NAME` [...]
+| [hoodie.bootstrap.index.enable](#hoodiebootstrapindexenable) | true | Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined, default true.<br />`Config Param: BOOTSTRAP_INDEX_ENABLE` [...]
+| [hoodie.bootstrap.index.type](#hoodiebootstrapindextype) | HFILE | Bootstrap index type determines which implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_TYPE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` [...]
+| [hoodie.partition.metafile.use.base.format](#hoodiepartitionmetafileusebaseformat) | false | If true, partition metafiles are saved in the same format as base-files for this dataset (e.g. Parquet / ORC). If false (default) partition metafiles are saved as properties files.<br />`Config Param: PARTITION_METAFILE_USE_BASE_FORMAT` [...]
+| [hoodie.populate.meta.fields](#hoodiepopulatemetafields) | true | When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br />`Config Param: POPULATE_META_FIELDS` [...]
+| [hoodie.record.merge.mode](#hoodierecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge reco [...]
+| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | Base file format to store all the base file data.<br />`Config Param: BASE_FILE_FORMAT` [...]
+| [hoodie.table.cdc.enabled](#hoodietablecdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode.<br />`Config Param: CDC_ENABLED`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.cdc.supplemental.logging.mode](#hoodietablecdcsupplementalloggingmode) | DATA_BEFORE_AFTER | org.apache.hudi.common.table.cdc.HoodieCDCSupplementalLoggingMode: Change log capture supplemental logging mode. The supplemental log is used for accelerating the generation of change log details. OP_KEY_ONLY: Only keeping record keys in the supplemental logs, so the reader needs to figure out the update before image [...]
+| [hoodie.table.initial.version](#hoodietableinitialversion) | EIGHT | Initial Version of table when the table was created. Used for upgrade/downgrade to identify what upgrade/downgrade paths happened on the table. This is only configured when the table is initially setup.<br />`Config Param: INITIAL_VERSION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.log.file.format](#hoodietablelogfileformat) | HOODIE_LOG | Log format used for the delta logs.<br />`Config Param: LOG_FILE_FORMAT` [...]
+| [hoodie.table.multiple.base.file.formats.enable](#hoodietablemultiplebasefileformatsenable) | false | When set to true, the table can support reading and writing multiple base file formats.<br />`Config Param: MULTIPLE_BASE_FILE_FORMATS_ENABLE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.timeline.timezone](#hoodietabletimelinetimezone) | LOCAL | User can set hoodie commit timeline timezone, such as utc, local and so on. local is default<br />`Config Param: TIMELINE_TIMEZONE` [...]
+| [hoodie.table.type](#hoodietabletype) | COPY_ON_WRITE | The table type for the underlying data.<br />`Config Param: TYPE` [...]
+| [hoodie.table.version](#hoodietableversion) | EIGHT | Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br />`Config Param: VERSION` [...]
+| [hoodie.timeline.history.path](#hoodietimelinehistorypath) | history | path under the meta folder, to store timeline history at.<br />`Config Param: TIMELINE_HISTORY_PATH` [...]
+| [hoodie.timeline.path](#hoodietimelinepath) | timeline | path under the meta folder, to store timeline instants at.<br />`Config Param: TIMELINE_PATH` [...]
+---
+
+## Spark Datasource Configs {#SPARK_DATASOURCE}
+These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
+
+
+### Read Options {#Read-Options}
+Options useful for reading tables via `read.format.option(...)`
+
+
+
+
+
+[**Basic Configs**](#Read-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| -------------------------------------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.read.begin.instanttime](#hoodiedatasourcereadbegininstanttime) | (N/A) | Required when `hoodie.datasource.query.type` is set to `incremental`. Represents the completion time to start incrementally pulling data from. The completion time here need not necessarily correspond to an instant on the timeline. New data written with completion_time >= START_COMMIT are fetched out. For e.g: ‘20170901080000’ will get all new data written on or after Sep 1, 201 [...]
+| [hoodie.datasource.read.end.instanttime](#hoodiedatasourcereadendinstanttime) | (N/A) | Used when `hoodie.datasource.query.type` is set to `incremental`. Represents the completion time to limit incrementally fetched data to. When not specified latest commit completion time from timeline is assumed by default. When specified, new data written with completion_time <= END_COMMIT are fetched out. Point in time type queries make more sense with begin and end completion [...]
+| [hoodie.datasource.read.incr.table.version](#hoodiedatasourcereadincrtableversion) | (N/A) | The table version assumed for incremental read<br />`Config Param: INCREMENTAL_READ_TABLE_VERSION` [...]
+| [hoodie.datasource.read.streaming.table.version](#hoodiedatasourcereadstreamingtableversion) | (N/A) | The table version assumed for streaming read<br />`Config Param: STREAMING_READ_TABLE_VERSION` [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files)<br />`Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: READ_PRE_COMBINE_FIELD` [...]
+---
+
+
+### Write Options {#Write-Options}
+You can pass down any of the WriteClient level configs directly using `options()` or `option(k,v)` methods.
+
+```java
+inputDF.write()
+.format("org.apache.hudi")
+.options(clientOpts) // any of the Hudi client opts can be passed in as well
+.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")
+.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")
+.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")
+.option(HoodieWriteConfig.TABLE_NAME, tableName)
+.mode(SaveMode.Append)
+.save(basePath);
+```
+
+Options useful for writing tables via `write.format.option(...)`
+
+
+
+
+
+[**Basic Configs**](#Write-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` [...]
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD` [...]
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD` [...]
+| [hoodie.datasource.write.secondarykey.column](#hoodiedatasourcewritesecondarykeycolumn) | (N/A) | Columns that constitute the secondary key component. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: SECONDARYKEY_COLUMN_NAME` [...]
+| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` [...]
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` [...]
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` [...]
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING` [...]
+| [hoodie.datasource.write.operation](#hoodiedatasourcewriteoperation) | upsert | Whether to do upsert, insert or bulk_insert for the write operation. Use bulk_insert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br />`Config Param: OPERATION` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.datasource.write.table.type](#hoodiedatasourcewritetabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TABLE_TYPE` [...]
+| [hoodie.write.record.merge.mode](#hoodiewriterecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge records, i.e., the record with the large [...]
+---
+
+## Flink Sql Configs {#FLINK_SQL}
+These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
+
+
+### Flink Options {#Flink-Options}
+Flink jobs using the SQL can be configured through the options in WITH clause. The actual datasource level configs are listed below.
+
+
+
+
+[**Basic Configs**](#Flink-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name to register to Hive metastore<br /> `Config Param: DATABASE_NAME` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name to register to Hive metastore<br /> `Config Param: TABLE_NAME` [...]
+| [path](#path) | (N/A) | Base path for the target hoodie table. The path would be created if it does not exist, otherwise a Hoodie table expects to be initialized successfully<br /> `Config Param: PATH` [...]
+| [read.commits.limit](#readcommitslimit) | (N/A) | The maximum number of commits allowed to read in each instant check, if it is streaming read, the avg read instants number per-second would be 'read.commits.limit'/'read.streaming.check-interval', by default no limit<br /> `Config Param: READ_COMMITS_LIMIT` [...]
+| [read.end-commit](#readend-commit) | (N/A) | End commit instant for reading, the commit time format should be 'yyyyMMddHHmmss'<br /> `Config Param: READ_END_COMMIT` [...]
+| [read.start-commit](#readstart-commit) | (N/A) | Start commit instant for reading, the commit time format should be 'yyyyMMddHHmmss', by default reading from the latest instant for streaming read<br /> `Config Param: READ_START_COMMIT` [...]
+| [archive.max_commits](#archivemax_commits) | 50 | Max number of commits to keep before archiving older commits into a sequential log, default 50<br /> `Config Param: ARCHIVE_MAX_COMMITS` [...]
+| [archive.min_commits](#archivemin_commits) | 40 | Min number of commits to keep before archiving older commits into a sequential log, default 40<br /> `Config Param: ARCHIVE_MIN_COMMITS` [...]
+| [cdc.enabled](#cdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode<br /> `Config Param: CDC_ENABLED` [...]
+| [cdc.supplemental.logging.mode](#cdcsupplementalloggingmode) | DATA_BEFORE_AFTER | Setting 'op_key_only' persists the 'op' and the record key only, setting 'data_before' persists the additional 'before' image, and setting 'data_before_after' persists the additional 'before' and 'after' images.<br /> `Config Param: SUPPLEMENTAL_LOGGING_MODE` [...]
+| [changelog.enabled](#changelogenabled) | false | Whether to keep all the intermediate changes, we try to keep all the changes of a record when enabled: 1). The sink accept the UPDATE_BEFORE message; 2). The source try to emit every changes of a record. The semantics is best effort because the compaction job would finally merge all changes of a record into one. default false to have UPSERT semantics<br /> [...]
+| [clean.async.enabled](#cleanasyncenabled) | true | Whether to cleanup the old commits immediately on new commits, enabled by default<br /> `Config Param: CLEAN_ASYNC_ENABLED` [...]
+| [clean.retain_commits](#cleanretain_commits) | 30 | Number of commits to retain. So data will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much you can incrementally pull on this table, default 30<br /> `Config Param: CLEAN_RETAIN_COMMITS` [...]
+| [clustering.async.enabled](#clusteringasyncenabled) | false | Async Clustering, default false<br /> `Config Param: CLUSTERING_ASYNC_ENABLED` [...]
+| [clustering.plan.strategy.small.file.limit](#clusteringplanstrategysmallfilelimit) | 600 | Files smaller than the size specified here are candidates for clustering, default 600 MB<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_SMALL_FILE_LIMIT` [...]
+| [clustering.plan.strategy.target.file.max.bytes](#clusteringplanstrategytargetfilemaxbytes) | 1073741824 | Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups, default 1 GB<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_TARGET_FILE_MAX_BYTES` [...]
+| [compaction.async.enabled](#compactionasyncenabled) | true | Async Compaction, enabled by default for MOR<br /> `Config Param: COMPACTION_ASYNC_ENABLED` [...]
+| [compaction.delta_commits](#compactiondelta_commits) | 5 | Max delta commits needed to trigger compaction, default 5 commits<br /> `Config Param: COMPACTION_DELTA_COMMITS` [...]
+| [hive_sync.enabled](#hive_syncenabled) | false | Asynchronously sync Hive meta to HMS, default false<br /> `Config Param: HIVE_SYNC_ENABLED` [...]
+| [hive_sync.jdbc_url](#hive_syncjdbc_url) | jdbc:hive2://localhost:10000 | Jdbc URL for hive sync, default 'jdbc:hive2://localhost:10000'<br /> `Config Param: HIVE_SYNC_JDBC_URL` [...]
+| [hive_sync.metastore.uris](#hive_syncmetastoreuris) | | Metastore uris for hive sync, default ''<br /> `Config Param: HIVE_SYNC_METASTORE_URIS` [...]
+| [hive_sync.mode](#hive_syncmode) | HMS | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql, default 'hms'<br /> `Config Param: HIVE_SYNC_MODE` [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Decides how data files need to be read, in 1) Snapshot mode (obtain latest view, based on row & columnar data); 2) incremental mode (new data since an instantTime); 3) Read Optimized mode (obtain latest view, based on columnar data) .Default: snapshot<br /> `Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br /> `Config Param: HIVE_STYLE_PARTITIONING` [...]
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | | Partition path field. Value to be used at the `partitionPath` component of `HoodieKey`. Actual value obtained by invoking .toString(), default ''<br /> `Config Param: PARTITION_PATH_FIELD` [...]
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | uuid | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br /> `Config Param: RECORD_KEY_FIELD` [...]
+| [index.type](#indextype) | FLINK_STATE | Index type of Flink write job, default is using state backed index.<br /> `Config Param: INDEX_TYPE` [...]
+| [lookup.join.cache.ttl](#lookupjoincachettl) | PT1H | The cache TTL (e.g. 10min) for the build table in lookup join.<br /> `Config Param: LOOKUP_JOIN_CACHE_TTL` [...]
+| [metadata.compaction.delta_commits](#metadatacompactiondelta_commits) | 10 | Max delta commits for metadata table to trigger compaction, default 10<br /> `Config Param: METADATA_COMPACTION_DELTA_COMMITS` [...]
+| [metadata.enabled](#metadataenabled) | true | Enable the internal metadata table which serves table metadata like level file listings, default enabled<br /> `Config Param: METADATA_ENABLED` [...]
+| [precombine.field](#precombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br /> `Config Param: PRECOMBINE_FIELD` [...]
+| [read.streaming.enabled](#readstreamingenabled) | false | Whether to read as streaming source, default false<br /> `Config Param: READ_AS_STREAMING` [...]
+| [read.streaming.skip_insertoverwrite](#readstreamingskip_insertoverwrite) | false | Whether to skip insert overwrite instants to avoid reading base files of insert overwrite operations for streaming read. In streaming scenarios, insert overwrite is usually used to repair data, here you can control the visibility of downstream streaming read.<br /> `Config Param: READ_STREAMING_SKIP_INSERT_OVERWRITE` [...]
+| [table.type](#tabletype) | COPY_ON_WRITE | Type of table to write. COPY_ON_WRITE (or) MERGE_ON_READ<br /> `Config Param: TABLE_TYPE` [...]
+| [write.operation](#writeoperation) | upsert | The write operation, that this write should do<br /> `Config Param: OPERATION` [...]
+| [write.parquet.max.file.size](#writeparquetmaxfilesize) | 120 | Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /> `Config Param: WRITE_PARQUET_MAX_FILE_SIZE` [...]
+---
+
+## Write Client Configs {#WRITE_CLIENT}
+Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.
+
+
+### Common Configurations {#Common-Configurations}
+The following set of configurations are common across Hudi.
+
+
+
+
+[**Basic Configs**](#Common-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` |
+---
+
+
+### Metadata Configs {#Metadata-Configs}
+Configurations used by the Hudi Metadata Table. This table maintains the metadata about a given Hudi table (e.g file listings) to avoid overhead of accessing cloud storage, during queries.
+
+
+
+
+[**Basic Configs**](#Metadata-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.metadata.enable](#hoodiemetadataenable) | true | Enable the internal metadata table which serves table metadata like level file listings<br />`Config Param: ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.metadata.index.bloom.filter.enable](#hoodiemetadataindexbloomfilterenable) | false | Enable indexing bloom filters of user data files under metadata table. When enabled, metadata table will have a partition to store the bloom filter index and will be used during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_BLOOM_FILTER`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.column.stats.enable](#hoodiemetadataindexcolumnstatsenable) | false | Enable indexing column ranges of user data files under metadata table key lookups. When enabled, metadata table will have a partition to store the column ranges and will be used for pruning files during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_COLUMN_STATS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.expression.enable](#hoodiemetadataindexexpressionenable) | false | Enable expression index within the metadata table. When this configuration property is enabled (`true`), the Hudi writer automatically keeps all expression indexes consistent with the data table. When disabled (`false`), all expression indexes are deleted. Note that individual expression index can only be created through a `CREATE INDEX` and deleted through a `DROP INDEX` statement [...]
+| [hoodie.metadata.index.partition.stats.enable](#hoodiemetadataindexpartitionstatsenable) | false | Enable aggregating stats for each column at the storage partition level.<br />`Config Param: ENABLE_METADATA_INDEX_PARTITION_STATS`<br />`Since Version: 1.0.0` [...]
+| [hoodie.metadata.index.secondary.enable](#hoodiemetadataindexsecondaryenable) | true | Enable secondary index within the metadata table. When this configuration property is enabled (`true`), the Hudi writer automatically keeps all secondary indexes consistent with the data table. When disabled (`false`), all secondary indexes are deleted. Note that individual secondary index can only be created through a `CREATE INDEX` and deleted through a `DROP INDEX` statement in [...]
+---
+
+
+### Storage Configs {#Storage-Configs}
+Configurations that control aspects around writing, sizing, reading base and log files.
+
+
+
+
+[**Basic Configs**](#Storage-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.parquet.compression.codec](#hoodieparquetcompressioncodec) | gzip | Compression Codec for parquet files<br />`Config Param: PARQUET_COMPRESSION_CODEC_NAME` |
+| [hoodie.parquet.max.file.size](#hoodieparquetmaxfilesize) | 125829120 | Target size in bytes for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br />`Config Param: PARQUET_MAX_FILE_SIZE` |
+---
+
+
+### Archival Configs {#Archival-Configs}
+Configurations that control archival.
+
+
+
+
+[**Basic Configs**](#Archival-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.keep.max.commits](#hoodiekeepmaxcommits) | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. <br />`Config Param: MAX_COMMITS_TO_KEEP` |
+| [hoodie.keep.min.commits](#hoodiekeepmincommits) | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline.<br />`Config Param: MIN_COMMITS_TO_KEEP` |
+---
+
+
+### Bootstrap Configs {#Bootstrap-Configs}
+Configurations that control how you want to bootstrap your existing tables for the first time into hudi. The bootstrap operation can flexibly avoid copying data over before you can use Hudi and support running the existing writers and new hudi writers in parallel, to validate the migration.
+
+
+
+
+[**Basic Configs**](#Bootstrap-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BASE_PATH`<br />`Since Version: 0.6.0` |
+---
+
+
+### Clean Configs {#Clean-Configs}
+Cleaning (reclamation of older/unused file groups/slices).
+
+
+
+
+[**Basic Configs**](#Clean-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------ | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.clean.async.enabled](#hoodiecleanasyncenabled) | false | Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_CLEAN` |
+| [hoodie.clean.commits.retained](#hoodiecleancommitsretained) | 10 | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br />`Config Param: CLEANER_COMMITS_RETAINED` |
+---
+
+
+### Clustering Configs {#Clustering-Configs}
+Configurations that control the clustering table service in hudi, which optimizes the storage layout for better query performance by sorting and sizing data files.
+
+
+
+
+[**Basic Configs**](#Clustering-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.plan.strategy.small.file.limit](#hoodieclusteringplanstrategysmallfilelimit) | 314572800 | Files smaller than the size in bytes specified here are candidates for clustering<br />`Config Param: PLAN_STRATEGY_SMALL_FILE_LIMIT`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.plan.strategy.target.file.max.bytes](#hoodieclusteringplanstrategytargetfilemaxbytes) | 1073741824 | Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups<br />`Config Param: PLAN_STRATEGY_TARGET_FILE_MAX_BYTES`<br />`Since Version: 0.7.0` |
+---
+
+
+### Compaction Configs {#Compaction-Configs}
+Configurations that control compaction (merging of log files onto a new base files).
+
+
+
+
+[**Basic Configs**](#Compaction-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.compact.inline](#hoodiecompactinline) | false | When set to true, compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br />`Config Param: INLINE_COMPACT` |
+| [hoodie.compact.inline.max.delta.commits](#hoodiecompactinlinemaxdeltacommits) | 5 | Number of delta commits after the last compaction, before scheduling of a new compaction is attempted. This config takes effect only for the compaction triggering strategy based on the number of commits, i.e., NUM_COMMITS, NUM_COMMITS_AFTER_LAST_REQUEST, NUM_AND_TIME, and NUM_OR_TIME.<br />`Config Param: INLINE_COMPACT_NUM_DELTA_COMMITS` |
+---
+
+
+### Error table Configs {#Error-table-Configs}
+Configurations that are required for Error table configs
+
+
+
+
+[**Basic Configs**](#Error-table-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.errortable.base.path](#hoodieerrortablebasepath) | (N/A) | Base path for error table under which all error records would be stored.<br />`Config Param: ERROR_TABLE_BASE_PATH` |
+| [hoodie.errortable.target.table.name](#hoodieerrortabletargettablename) | (N/A) | Table name to be used for the error table<br />`Config Param: ERROR_TARGET_TABLE` |
+| [hoodie.errortable.write.class](#hoodieerrortablewriteclass) | (N/A) | Class which handles the error table writes. This config is used to configure a custom implementation for Error Table Writer. Specify the full class name of the custom error table writer as a value for this config<br />`Config Param: ERROR_TABLE_WRITE_CLASS` |
+| [hoodie.errortable.enable](#hoodieerrortableenable) | false | Config to enable error table. If the config is enabled, all the records with processing error in DeltaStreamer are transferred to error table.<br />`Config Param: ERROR_TABLE_ENABLED` |
+| [hoodie.errortable.insert.shuffle.parallelism](#hoodieerrortableinsertshuffleparallelism) | 200 | Config to set insert shuffle parallelism. The config is similar to hoodie.insert.shuffle.parallelism config but applies to the error table.<br />`Config Param: ERROR_TABLE_INSERT_PARALLELISM_VALUE` |
+| [hoodie.errortable.upsert.shuffle.parallelism](#hoodieerrortableupsertshuffleparallelism) | 200 | Config to set upsert shuffle parallelism. The config is similar to hoodie.upsert.shuffle.parallelism config but applies to the error table.<br />`Config Param: ERROR_TABLE_UPSERT_PARALLELISM_VALUE` |
+| [hoodie.errortable.validate.recordcreation.enable](#hoodieerrortablevalidaterecordcreationenable) | true | Records that fail to be created due to keygeneration failure or other issues will be sent to the Error Table<br />`Config Param: ERROR_ENABLE_VALIDATE_RECORD_CREATION`<br />`Since Version: 0.15.0` |
+| [hoodie.errortable.validate.targetschema.enable](#hoodieerrortablevalidatetargetschemaenable) | false | Records with schema mismatch with Target Schema are sent to Error Table.<br />`Config Param: ERROR_ENABLE_VALIDATE_TARGET_SCHEMA` |
+| [hoodie.errortable.write.failure.strategy](#hoodieerrortablewritefailurestrategy) | ROLLBACK_COMMIT | The config specifies the failure strategy if error table write fails. Use one of - [ROLLBACK_COMMIT (Rollback the corresponding base table write commit for which the error events were triggered) , LOG_ERROR (Error is logged but the base table write succeeds) ]<br />`Config Param: ERROR_TABLE_WRITE_FAILURE_STRATEGY` |
+---
+
+
+### Write Configurations {#Write-Configurations}
+Configurations that control write behavior on Hudi tables. These can be directly passed down from even higher level frameworks (e.g Spark datasources, Flink sink) and utilities (e.g Hudi Streamer).
+
+
+
+
+[**Basic Configs**](#Write-Configurations-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br />`Config Param: TBL_NAME` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD_NAME` [...]
+| [hoodie.instant_state.timeline_server_based.enabled](#hoodieinstant_statetimeline_server_basedenabled) | false | If enabled, writers get instant state from timeline server rather than requesting DFS directly<br />`Config Param: INSTANT_STATE_TIMELINE_SERVER_BASED`<br />`Since Version: 1.0.0` [...]
+| [hoodie.instant_state.timeline_server_based.force_refresh.request.number](#hoodieinstant_statetimeline_server_basedforce_refreshrequestnumber) | 100 | Number of requests to trigger instant state cache refreshing<br />`Config Param: INSTANT_STATE_TIMELINE_SERVER_BASED_FORCE_REFRESH_REQUEST_NUMBER`<br />`Since Version: 1.0.0` [...]
+| [hoodie.write.auto.upgrade](#hoodiewriteautoupgrade) | true | If enabled, writers automatically migrate the table to the specified write table version if the current table version is lower.<br />`Config Param: AUTO_UPGRADE_VERSION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.write.concurrency.mode](#hoodiewriteconcurrencymode) | SINGLE_WRITER | org.apache.hudi.common.model.WriteConcurrencyMode: Concurrency modes for write operations. SINGLE_WRITER(default): Only one active writer to the table. Maximizes throughput. OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table with lazy conflict resolution using locks. This means that only on [...]
+| [hoodie.write.record.merge.mode](#hoodiewriterecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge rec [...]
+| [hoodie.write.table.version](#hoodiewritetableversion) | 8 | The table version this writer is storing the table in. This should match the current table version.<br />`Config Param: WRITE_TABLE_VERSION`<br />`Since Version: 1.0.0` [...]
+---
+
+
+### Lock Configs {#LOCK}
+Configurations that control locking mechanisms required for concurrency control between writers to a Hudi table. Concurrency between Hudi's own table services are auto managed internally.
+
+
+#### Common Lock Configurations {#Common-Lock-Configurations}
+
+
+
+
+
+[**Basic Configs**](#Common-Lock-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.write.lock.heartbeat_interval_ms](#hoodiewritelockheartbeat_interval_ms) | 60000 | Heartbeat interval in ms, to send a heartbeat to indicate that hive client holding locks.<br />`Config Param: LOCK_HEARTBEAT_INTERVAL_MS`<br />`Since Version: 0.15.0` |
+---
+
+
+### Key Generator Configs {#KEY_GENERATOR}
+Hudi maintains keys (record key + partition path) for uniquely identifying a particular record. These configs allow developers to setup the Key generator class that extracts these out of incoming records.
+
+
+#### Key Generator Options {#Key-Generator-Options}
+
+
+
+
+
+[**Basic Configs**](#Key-Generator-Options-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD_NAME` |
+| [hoodie.datasource.write.secondarykey.column](#hoodiedatasourcewritesecondarykeycolumn) | (N/A) | Columns that constitute the secondary key component. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: SECONDARYKEY_COLUMN_NAME` |
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` |
+---
+
+
+### Index Configs {#INDEX}
+Configurations that control indexing behavior, which tags incoming records as either inserts or updates to older records.
+
+
+#### Common Index Configs {#Common-Index-Configs}
+
+
+
+
+
+[**Basic Configs**](#Common-Index-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------ | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.expression.index.function](#hoodieexpressionindexfunction) | (N/A) | Function to be used for building the expression index.<br />`Config Param: INDEX_FUNCTION`<br />`Since Version: 1.0.0` |
+| [hoodie.expression.index.name](#hoodieexpressionindexname) | (N/A) | Name of the expression index. This is also used for the partition name in the metadata table.<br />`Config Param: INDEX_NAME`<br />`Since Version: 1.0.0` |
+| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Index definition checksum is used to guard against partial writes in HDFS. It is added as the last entry in index.properties and then used to validate while reading table config.<br />`Config Param: INDEX_DEFINITION_CHECKSUM`<br />`Since Version: 1.0.0` |
+| [hoodie.expression.index.type](#hoodieexpressionindextype) | COLUMN_STATS | Type of the expression index. Default is `column_stats` if there are no functions and expressions in the command. Valid options could be BITMAP, COLUMN_STATS, LUCENE, etc. If index_type is not provided, and there are functions or expressions in the command then a expression index using column stats will be created.<br />`Config Param: INDEX_TYPE`<br />`Since Version: 1.0.0` |
+---
+
+
+#### Common Index Configs {#Common-Index-Configs}
+
+
+
+
+
+[**Basic Configs**](#Common-Index-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.index.type](#hoodieindextype) | (N/A) | org.apache.hudi.index.HoodieIndex$IndexType: Determines how input records are indexed, i.e., looked up based on the key for the location in the existing table. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java engines. HBASE: uses an external managed Apache HBase table to store record key to location mapping. HBase index is a global index, enforcing key uniqueness across all partitions i [...]
+| [hoodie.bucket.index.query.pruning](#hoodiebucketindexquerypruning) | true | Control if table with bucket index use bucket query or not<br />`Config Param: BUCKET_QUERY_INDEX` [...]
+---
+
+## Metastore and Catalog Sync Configs {#META_SYNC}
+Configurations used by the Hudi to sync metadata to external metastores and catalogs.
+
+
+### Common Metadata Sync Configs {#Common-Metadata-Sync-Configs}
+
+
+
+
+
+[**Basic Configs**](#Common-Metadata-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+
+### Glue catalog sync based client Configurations {#Glue-catalog-sync-based-client-Configurations}
+Configs that control Glue catalog sync based client.
+
+
+
+
+[**Basic Configs**](#Glue-catalog-sync-based-client-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.glue.partition_index_fields](#hoodiedatasourcemetasyncgluepartition_index_fields) | | Specify the partitions fields to index on aws glue. Separate the fields by semicolon. By default, when the feature is enabled, all the partition will be indexed. You can create up to three indexes, separate them by comma. Eg: col1;col2;col3,col2,col3<br />`Config Param: META_SYNC_PARTITION_INDEX_FIELDS`<br />`Since Version: 0.15.0` |
+| [hoodie.datasource.meta.sync.glue.partition_index_fields.enable](#hoodiedatasourcemetasyncgluepartition_index_fieldsenable) | false | Enable aws glue partition index feature, to speedup partition based query pattern<br />`Config Param: META_SYNC_PARTITION_INDEX_FIELDS_ENABLE`<br />`Since Version: 0.15.0` |
+---
+
+
+### BigQuery Sync Configs {#BigQuery-Sync-Configs}
+Configurations used by the Hudi to sync metadata to Google BigQuery.
+
+
+
+
+[**Basic Configs**](#BigQuery-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+
+### Hive Sync Configs {#Hive-Sync-Configs}
+Configurations used by the Hudi to sync metadata to Hive Metastore.
+
+
+
+
+[**Basic Configs**](#Hive-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+
+### Global Hive Sync Configs {#Global-Hive-Sync-Configs}
+Global replication configurations used by the Hudi to sync metadata to Hive Metastore.
+
+
+
+
+[**Basic Configs**](#Global-Hive-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+
+### DataHub Sync Configs {#DataHub-Sync-Configs}
+Configurations used by the Hudi to sync metadata to DataHub.
+
+
+
+
+[**Basic Configs**](#DataHub-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+## Metrics Configs {#METRICS}
+These set of configs are used to enable monitoring and reporting of key Hudi stats and metrics.
+
+
+### Metrics Configurations {#Metrics-Configurations}
+Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean, rollback etc. The following sections list the supported reporters.
+
+
+
+
+[**Basic Configs**](#Metrics-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.on](#hoodiemetricson) | false | Turn on/off metrics reporting. off by default.<br />`Config Param: TURN_METRICS_ON`<br />`Since Version: 0.5.0` |
+| [hoodie.metrics.reporter.type](#hoodiemetricsreportertype) | GRAPHITE | Type of metrics reporter.<br />`Config Param: METRICS_REPORTER_TYPE_VALUE`<br />`Since Version: 0.5.0` |
+| [hoodie.metricscompaction.log.blocks.on](#hoodiemetricscompactionlogblockson) | false | Turn on/off metrics reporting for log blocks with compaction commit. off by default.<br />`Config Param: TURN_METRICS_COMPACTION_LOG_BLOCKS_ON`<br />`Since Version: 0.14.0` |
+---
+
+
+### Metrics Configurations for M3 {#Metrics-Configurations-for-M3}
+Enables reporting on Hudi metrics using M3. Hudi publishes metrics on every commit, clean, rollback etc.
+
+
+
+
+[**Basic Configs**](#Metrics-Configurations-for-M3-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.m3.env](#hoodiemetricsm3env) | production | M3 tag to label the environment (defaults to 'production'), applied to all metrics.<br />`Config Param: M3_ENV`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.host](#hoodiemetricsm3host) | localhost | M3 host to connect to.<br />`Config Param: M3_SERVER_HOST_NAME`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.port](#hoodiemetricsm3port) | 9052 | M3 port to connect to.<br />`Config Param: M3_SERVER_PORT_NUM`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.service](#hoodiemetricsm3service) | hoodie | M3 tag to label the service name (defaults to 'hoodie'), applied to all metrics.<br />`Config Param: M3_SERVICE`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.tags](#hoodiemetricsm3tags) | | Optional M3 tags applied to all metrics.<br />`Config Param: M3_TAGS`<br />`Since Version: 0.15.0` |
+---
+
+## Kafka Connect Configs {#KAFKA_CONNECT}
+These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables
+
+
+### Kafka Sink Connect Configurations {#Kafka-Sink-Connect-Configurations}
+Configurations for Kafka Connect Sink Connector for Hudi.
+
+
+
+
+[**Basic Configs**](#Kafka-Sink-Connect-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------- | --------------- | ----------------------------------------------------------------------------------------- |
+| [bootstrap.servers](#bootstrapservers) | localhost:9092 | The bootstrap servers for the Kafka Cluster.<br />`Config Param: KAFKA_BOOTSTRAP_SERVERS` |
+---
+
+## Hudi Streamer Configs {#HUDI_STREAMER}
+These set of configs are used for Hudi Streamer utility which provides the way to ingest from different sources such as DFS or Kafka.
+
+
+### Hudi Streamer Configs {#Hudi-Streamer-Configs}
+
+
+
+
+
+[**Basic Configs**](#Hudi-Streamer-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.kafka.topic](#hoodiestreamersourcekafkatopic) | (N/A) | Kafka topic name. The config is specific to HoodieMultiTableStreamer<br />`Config Param: KAFKA_TOPIC` |
+---
+
+
+### Hudi Streamer SQL Transformer Configs {#Hudi-Streamer-SQL-Transformer-Configs}
+Configurations controlling the behavior of SQL transformer in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#Hudi-Streamer-SQL-Transformer-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.transformer.sql](#hoodiestreamertransformersql) | (N/A) | SQL Query to be executed during write<br />`Config Param: TRANSFORMER_SQL` |
+| [hoodie.streamer.transformer.sql.file](#hoodiestreamertransformersqlfile) | (N/A) | File with a SQL script to be executed during write<br />`Config Param: TRANSFORMER_SQL_FILE` |
+---
+
+
+### Hudi Streamer Source Configs {#DELTA_STREAMER_SOURCE}
+Configurations controlling the behavior of reading source data.
+
+
+#### DFS Path Selector Configs {#DFS-Path-Selector-Configs}
+Configurations controlling the behavior of path selector for DFS source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#DFS-Path-Selector-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------- | ------- | ------------------------------------------------------------------- |
+| [hoodie.streamer.source.dfs.root](#hoodiestreamersourcedfsroot) | (N/A) | Root path of the source on DFS<br />`Config Param: ROOT_INPUT_PATH` |
+---
+
+
+#### Hudi Incremental Source Configs {#Hudi-Incremental-Source-Configs}
+Configurations controlling the behavior of incremental pulling from a Hudi table as a source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#Hudi-Incremental-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------- |
+| [hoodie.streamer.source.hoodieincr.path](#hoodiestreamersourcehoodieincrpath) | (N/A) | Base-path for the source Hudi table<br />`Config Param: HOODIE_SRC_BASE_PATH` |
+---
+
+
+#### Kafka Source Configs {#Kafka-Source-Configs}
+Configurations controlling the behavior of Kafka source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#Kafka-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.kafka.topic](#hoodiestreamersourcekafkatopic) | (N/A) | Kafka topic name.<br />`Config Param: KAFKA_TOPIC_NAME` |
+| [hoodie.streamer.source.kafka.proto.value.deserializer.class](#hoodiestreamersourcekafkaprotovaluedeserializerclass) | org.apache.kafka.common.serialization.ByteArrayDeserializer | Kafka Proto Payload Deserializer Class<br />`Config Param: KAFKA_PROTO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.15.0` |
+---
+
+
+#### Pulsar Source Configs {#Pulsar-Source-Configs}
+Configurations controlling the behavior of Pulsar source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#Pulsar-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.pulsar.topic](#hoodiestreamersourcepulsartopic) | (N/A) | Name of the target Pulsar topic to source data from<br />`Config Param: PULSAR_SOURCE_TOPIC_NAME` |
+| [hoodie.streamer.source.pulsar.endpoint.admin.url](#hoodiestreamersourcepulsarendpointadminurl) | http://localhost:8080 | URL of the target Pulsar endpoint (of the form 'pulsar://host:port'<br />`Config Param: PULSAR_SOURCE_ADMIN_ENDPOINT_URL` |
+| [hoodie.streamer.source.pulsar.endpoint.service.url](#hoodiestreamersourcepulsarendpointserviceurl) | pulsar://localhost:6650 | URL of the target Pulsar endpoint (of the form 'pulsar://host:port'<br />`Config Param: PULSAR_SOURCE_SERVICE_ENDPOINT_URL` |
+---
+
+
+#### S3 Source Configs {#S3-Source-Configs}
+Configurations controlling the behavior of S3 source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#S3-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------- |
+| [hoodie.streamer.s3.source.queue.url](#hoodiestreamers3sourcequeueurl) | (N/A) | Queue url for cloud object events<br />`Config Param: S3_SOURCE_QUEUE_URL` |
+---
+
+
+#### File-based SQL Source Configs {#File-based-SQL-Source-Configs}
+Configurations controlling the behavior of File-based SQL Source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#File-based-SQL-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.sql.file](#hoodiestreamersourcesqlfile) | (N/A) | SQL file path containing the SQL query to read source data.<br />`Config Param: SOURCE_SQL_FILE`<br />`Since Version: 0.14.0` |
+---
+
+
+#### SQL Source Configs {#SQL-Source-Configs}
+Configurations controlling the behavior of SQL source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#SQL-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------ | ------- | ------------------------------------------------------------------- |
+| [hoodie.streamer.source.sql.sql.query](#hoodiestreamersourcesqlsqlquery) | (N/A) | SQL query for fetching source data.<br />`Config Param: SOURCE_SQL` |
+---
+
+
+### Hudi Streamer Schema Provider Configs {#SCHEMA_PROVIDER}
+Configurations that control the schema provider for Hudi Streamer.
+
+
+#### Hudi Streamer Schema Provider Configs {#Hudi-Streamer-Schema-Provider-Configs}
+
+
+
+
+
+[**Basic Configs**](#Hudi-Streamer-Schema-Provider-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.registry.targetUrl](#hoodiestreamerschemaproviderregistrytargetUrl) | (N/A) | The schema of the target you are writing to e.g. https://foo:bar@schemaregistry.org<br />`Config Param: TARGET_SCHEMA_REGISTRY_URL` |
+| [hoodie.streamer.schemaprovider.registry.url](#hoodiestreamerschemaproviderregistryurl) | (N/A) | The schema of the source you are reading from e.g. https://foo:bar@schemaregistry.org<br />`Config Param: SRC_SCHEMA_REGISTRY_URL` |
+---
+
+
+#### File-based Schema Provider Configs {#File-based-Schema-Provider-Configs}
+Configurations for file-based schema provider.
+
+
+
+
+[**Basic Configs**](#File-based-Schema-Provider-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.source.schema.file](#hoodiestreamerschemaprovidersourceschemafile) | (N/A) | The schema of the source you are reading from<br />`Config Param: SOURCE_SCHEMA_FILE` |
+| [hoodie.streamer.schemaprovider.target.schema.file](#hoodiestreamerschemaprovidertargetschemafile) | (N/A) | The schema of the target you are writing to<br />`Config Param: TARGET_SCHEMA_FILE` |
+---
+
diff --git a/website/versioned_docs/version-1.0.0/bos_hoodie.md b/website/versioned_docs/version-1.0.0/bos_hoodie.md
new file mode 100644
index 00000000000..2a6cde81c86
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/bos_hoodie.md
@@ -0,0 +1,57 @@
+---
+title: Baidu Cloud
+keywords: [ hudi, hive, baidu, bos, spark, presto]
+summary: In this page, we go over how to configure Hudi with bos filesystem.
+last_modified_at: 2021-06-09T11:38:24-10:00
+---
+In this page, we explain how to get your Hudi job to store into Baidu BOS.
+
+## Baidu BOS configs
+
+There are two configurations required for Hudi-BOS compatibility:
+
+- Adding Baidu BOS Credentials for Hudi
+- Adding required Jars to classpath
+
+### Baidu BOS Credentials
+
+Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the `fs.defaultFS` with your BOS bucket name, replace `fs.bos.endpoint` with your bos endpoint, replace `fs.bos.access.key` with your bos key, replace `fs.bos.secret.access.key` with your bos secret key. Hudi should be able to read/write from the bucket.
+
+```xml
+<property>
+ <name>fs.defaultFS</name>
+ <value>bos://bucketname/</value>
+</property>
+
+<property>
+ <name>fs.bos.endpoint</name>
+ <value>bos-endpoint-address</value>
+ <description>Baidu bos endpoint to connect to,for example : http://bj.bcebos.com</description>
+</property>
+
+<property>
+ <name>fs.bos.access.key</name>
+ <value>bos-key</value>
+ <description>Baidu access key</description>
+</property>
+
+<property>
+ <name>fs.bos.secret.access.key</name>
+ <value>bos-secret-key</value>
+ <description>Baidu secret key.</description>
+</property>
+
+<property>
+ <name>fs.bos.impl</name>
+ <value>org.apache.hadoop.fs.bos.BaiduBosFileSystem</value>
+</property>
+```
+
+### Baidu bos Libs
+
+Baidu hadoop libraries jars to add to our classpath
+
+- com.baidubce:bce-java-sdk:0.10.165
+- bos-hdfs-sdk-1.0.2-community.jar
+
+You can download the bos-hdfs-sdk jar from [here](https://sdk.bce.baidu.com/console-sdk/bos-hdfs-sdk-1.0.2-community.jar.zip) , and then unzip it.
\ No newline at end of file
diff --git a/website/versioned_docs/version-1.0.0/cleaning.md b/website/versioned_docs/version-1.0.0/cleaning.md
new file mode 100644
index 00000000000..5f6ea4b3697
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/cleaning.md
@@ -0,0 +1,155 @@
+---
+title: Cleaning
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+---
+## Background
+Cleaning is a table service employed by Hudi to reclaim space occupied by older versions of data and keep storage costs
+in check. Apache Hudi provides snapshot isolation between writers and readers by managing multiple versioned files with **MVCC**
+concurrency. These file versions provide history and enable time travel and rollbacks, but it is important to manage
+how much history you keep to balance your costs. Cleaning service plays a crucial role in manging the tradeoff between
+retaining long history of data and the associated storage costs.
+
+Hudi enables [Automatic Hudi cleaning](/docs/configurations/#hoodiecleanautomatic) by default. Cleaning is invoked
+immediately after each commit, to delete older file slices. It's recommended to leave this enabled to ensure metadata
+and data storage growth is bounded. Cleaner can also be scheduled after every few commits instead of after every commit by
+configuring [hoodie.clean.max.commits](https://hudi.apache.org/docs/configurations#hoodiecleanmaxcommits).
+
+### Cleaning Retention Policies
+When cleaning old files, you should be careful not to remove files that are being actively used by long running queries.
+
+For spark based:
+
+| Config Name | Default | Description |
+|----------------------------------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
+| hoodie.cleaner.policy | KEEP_LATEST_COMMITS (Optional) | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. <br /><br />`Config Param: CLEANER_POLICY` |
+
+The corresponding config for Flink based engine is [`clean.policy`](https://hudi.apache.org/docs/configurations/#cleanpolicy).
+
+Hudi cleaner currently supports the below cleaning policies to keep a certain number of commits or file versions:
+
+- **KEEP_LATEST_COMMITS**: This is the default policy. This is a temporal cleaning policy that ensures the effect of
+ having lookback into all the changes that happened in the last X commits. Suppose a writer is ingesting data
+ into a Hudi dataset every 30 minutes and the longest running query can take 5 hours to finish, then the user should
+ retain atleast the last 10 commits. With such a configuration, we ensure that the oldest version of a file is kept on
+ disk for at least 5 hours, thereby preventing the longest running query from failing at any point in time. Incremental
+ cleaning is also possible using this policy.
+ Number of commits to retain can be configured by [`hoodie.cleaner.commits.retained`](https://analytics.google.com/analytics/web/#/p300324801/reports/intelligenthome).
+ The corresponding Flink related config is [`clean.retain_commits`](https://hudi.apache.org/docs/configurations/#cleanretain_commits).
+
+- **KEEP_LATEST_FILE_VERSIONS**: This policy has the effect of keeping N number of file versions irrespective of time.
+ This policy is useful when it is known how many MAX versions of the file does one want to keep at any given time.
+ To achieve the same behaviour as before of preventing long running queries from failing, one should do their calculations
+ based on data patterns. Alternatively, this policy is also useful if a user just wants to maintain 1 latest version of the file.
+ Number of file versions to retain can be configured by [`hoodie.cleaner.fileversions.retained`](https://hudi.apache.org/docs/configurations/#hoodiecleanerfileversionsretained).
+ The corresponding Flink related config is [`clean.retain_file_versions`](https://hudi.apache.org/docs/configurations/#cleanretain_file_versions).
+
+- **KEEP_LATEST_BY_HOURS**: This policy clean up based on hours.It is simple and useful when knowing that you want to
+ keep files at any given time. Corresponding to commits with commit times older than the configured number of hours to
+ be retained are cleaned. Currently you can configure by parameter [`hoodie.cleaner.hours.retained`](https://hudi.apache.org/docs/configurations/#hoodiecleanerhoursretained).
+ The corresponding Flink related config is [`clean.retain_hours`](https://hudi.apache.org/docs/configurations/#cleanretain_hours).
+
+### Configs
+For details about all possible configurations and their default values see the [configuration docs](https://hudi.apache.org/docs/next/configurations/#Clean-Configs).
+For Flink related configs refer [here](https://hudi.apache.org/docs/next/configurations/#FLINK_SQL).
+
+### Ways to trigger Cleaning
+
+#### Inline
+
+By default, in Spark based writing, cleaning is run inline after every commit using the default policy of `KEEP_LATEST_COMMITS`. It's recommended
+to keep this enabled, to ensure metadata and data storage growth is bounded. To enable this, users do not have to set any configs. Following are the relevant basic configs.
+
+| Config Name | Default | Description |
+|----------------------------------| -----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.clean.automatic | true (Optional) | When enabled, the cleaner table service is invoked immediately after each commit, to delete older file slices. It's recommended to enable this, to ensure metadata and data storage growth is bounded.<br /><br />`Config Param: AUTO_CLEAN` |
+| hoodie.cleaner.commits.retained | 10 (Optional) | Number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br /><br />`Config Param: CLEANER_COMMITS_RETAINED` |
+
+
+#### Async
+In case you wish to run the cleaner service asynchronously along with writing, please enable the [`hoodie.clean.async`](https://hudi.apache.org/docs/configurations#hoodiecleanasync) as shown below:
+```java
+hoodie.clean.automatic=true
+hoodie.clean.async=true
+```
+
+For Flink based writing, this is the default mode of cleaning. Please refer to [`clean.async.enabled`](https://hudi.apache.org/docs/configurations/#cleanasyncenabled) for details.
+
+#### Run independently
+Hoodie Cleaner can also be run as a separate process. Following is the command for running the cleaner independently:
+```
+spark-submit --master local \
+ --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
+ --class org.apache.hudi.utilities.HoodieCleaner `ls packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle-*.jar` --help
+ Usage: <main class> [options]
+ Options:
+ --help, -h
+
+ --hoodie-conf
+ Any configuration that can be set in the properties file (using the CLI
+ parameter "--props") can also be passed command line using this
+ parameter. This can be repeated
+ Default: []
+ --props
+ path to properties file on localfs or dfs, with configurations for
+ hoodie client for cleaning
+ --spark-master
+ spark master to use.
+ Default: local[2]
+ * --target-base-path
+ base path for the hoodie table to be cleaner.
+```
+Some examples to run the cleaner.
+Keep the latest 10 commits
+```
+spark-submit --master local \
+ --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
+ --class org.apache.hudi.utilities.HoodieCleaner `ls packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle-*.jar` \
+ --target-base-path /path/to/hoodie_table \
+ --hoodie-conf hoodie.cleaner.policy=KEEP_LATEST_COMMITS \
+ --hoodie-conf hoodie.cleaner.commits.retained=10 \
+ --hoodie-conf hoodie.cleaner.parallelism=200
+```
+Keep the latest 3 file versions
+```
+spark-submit --master local \
+ --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
+ --class org.apache.hudi.utilities.HoodieCleaner `ls packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle-*.jar` \
+ --hoodie-conf hoodie.cleaner.policy=KEEP_LATEST_FILE_VERSIONS \
+ --hoodie-conf hoodie.cleaner.fileversions.retained=3 \
+ --hoodie-conf hoodie.cleaner.parallelism=200
+```
+Clean commits older than 24 hours
+```
+spark-submit --master local \
+ --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
+ --class org.apache.hudi.utilities.HoodieCleaner `ls packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle-*.jar` \
+ --target-base-path /path/to/hoodie_table \
+ --hoodie-conf hoodie.cleaner.policy=KEEP_LATEST_BY_HOURS \
+ --hoodie-conf hoodie.cleaner.hours.retained=24 \
+ --hoodie-conf hoodie.cleaner.parallelism=200
+```
+Note: The parallelism takes the min value of number of partitions to clean and `hoodie.cleaner.parallelism`.
+
+#### CLI
+You can also use [Hudi CLI](/docs/cli) to run Hoodie Cleaner.
+
+CLI provides the below commands for cleaner service:
+- `cleans show`
+- `clean showpartitions`
+- `cleans run`
+
+Example of cleaner keeping the latest 10 commits
+```
+cleans run --sparkMaster local --hoodieConfigs hoodie.cleaner.policy=KEEP_LATEST_COMMITS hoodie.cleaner.commits.retained=3 hoodie.cleaner.parallelism=200
+```
+
+You can find more details and the relevant code for these commands in [`org.apache.hudi.cli.commands.CleansCommand`](https://github.com/apache/hudi/blob/master/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CleansCommand.java) class.
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Cleaner Service: Save up to 40% on data lake storage costs | Hudi Labs](https://youtu.be/mUvRhJDoO3w)
+* [Efficient Data Lake Management with Apache Hudi Cleaner: Benefits of Scheduling Data Cleaning #1](https://www.youtube.com/watch?v=CEzgFtmVjx4)
+* [Efficient Data Lake Management with Apache Hudi Cleaner: Benefits of Scheduling Data Cleaning #2](https://www.youtube.com/watch?v=RbBF9Ys2GqM)
\ No newline at end of file
diff --git a/website/versioned_docs/version-1.0.0/cli.md b/website/versioned_docs/version-1.0.0/cli.md
new file mode 100644
index 00000000000..def32b11a8e
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/cli.md
@@ -0,0 +1,755 @@
+---
+title: CLI
+keywords: [hudi, cli]
+last_modified_at: 2021-08-18T15:59:57-04:00
+---
+
+### Local set up
+Once hudi has been built, the shell can be fired by via `cd hudi-cli && ./hudi-cli.sh`.
+
+### Hudi CLI Bundle setup
+In release `0.13.0` we have now added another way of launching the `hudi cli`, which is using the `hudi-cli-bundle`.
+
+There are a couple of requirements when using this approach such as having `spark` installed locally on your machine.
+It is required to use a spark distribution with hadoop dependencies packaged such as `spark-3.3.1-bin-hadoop2.tgz` from https://archive.apache.org/dist/spark/.
+We also recommend you set an env variable `$SPARK_HOME` to the path of where spark is installed on your machine.
+One important thing to note is that the `hudi-spark-bundle` should also be present when using the `hudi-cli-bundle`.
+To provide the locations of these bundle jars you can set them in your shell like so:
+`export CLI_BUNDLE_JAR=<path-to-cli-bundle-jar-to-use>` , `export SPARK_BUNDLE_JAR=<path-to-spark-bundle-jar-to-use>`.
+
+For steps see below if you are not compiling the project and downloading the jars:
+
+1. Create an empty folder as a new directory
+2. Copy the hudi-cli-bundle jars and hudi-spark*-bundle jars to this directory
+3. Copy the following script and folder to this directory
+```
+packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+packaging/hudi-cli-bundle/conf . the `conf` folder should be in this directory.
+```
+
+4. Start Hudi CLI shell with environment variables set
+```
+export SPARK_HOME=<spark-home-folder>
+export CLI_BUNDLE_JAR=<cli-bundle-jar-to-use>
+export SPARK_BUNDLE_JAR=<spark-bundle-jar-to-use>
+
+./hudi-cli-with-bundle.sh
+
+```
+
+### Base path
+A hudi table resides on DFS, in a location referred to as the `basePath` and
+we would need this location in order to connect to a Hudi table. Hudi library effectively manages this table internally, using `.hoodie` subfolder to track all metadata.
+
+
+
+
+### Using Hudi-cli in S3
+If you are using hudi that comes packaged with AWS EMR, you can find instructions to use hudi-cli [here](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-cli.html).
+If you are not using EMR, or would like to use latest hudi-cli from master, you can follow the below steps to access S3 dataset in your local environment (laptop).
+
+Build Hudi with corresponding Spark version, for eg, -Dspark3.1.x
+
+Set the following environment variables.
+```
+export AWS_REGION=us-east-2
+export AWS_ACCESS_KEY_ID=<key_id>
+export AWS_SECRET_ACCESS_KEY=<secret_key>
+export SPARK_HOME=<spark_home>
+```
+Ensure you set the SPARK_HOME to your local spark home compatible to compiled hudi spark version above.
+
+Apart from these, we might need to add aws jars to class path so that accessing S3 is feasible from local.
+We need two jars, namely, aws-java-sdk-bundle jar and hadoop-aws jar which you can find online.
+For eg:
+```
+wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.0/hadoop-aws-3.2.0.jar -o /lib/spark-3.2.0-bin-hadoop3.2/jars/hadoop-aws-3.2.0.jar
+wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.375/aws-java-sdk-bundle-1.11.375.jar -o /lib/spark-3.2.0-bin-hadoop3.2/jars/aws-java-sdk-bundle-1.11.375.jar
+```
+
+#### Note: These AWS jar versions below are specific to Spark 3.2.0
+```
+export CLIENT_JAR=/lib/spark-3.2.0-bin-hadoop3.2/jars/aws-java-sdk-bundle-1.12.48.jar:/lib/spark-3.2.0-bin-hadoop3.2/jars/hadoop-aws-3.3.1.jar
+```
+Once these are set, you are good to launch hudi-cli and access S3 dataset.
+```
+./hudi-cli/hudi-cli.sh
+```
+### Using hudi-cli on Google Dataproc
+[Dataproc](https://cloud.google.com/dataproc) is Google's managed service for running Apache Hadoop, Apache Spark,
+Apache Flink, Presto and many other frameworks, including Hudi. If you want to run the Hudi CLI on a Dataproc node
+which has not been launched with Hudi support enabled, you can use the steps below:
+
+These steps use Hudi version 0.13.0. If you want to use a different version you will have to edit the below commands
+appropriately:
+1. Once you've started the Dataproc cluster, you can ssh into it as follows:
+```
+$ gcloud compute ssh --zone "YOUR_ZONE" "HOSTNAME_OF_MASTER_NODE" --project "YOUR_PROJECT"
+```
+
+2. Download the Hudi CLI bundle
+```
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-cli-bundle_2.12/0.13.0/hudi-cli-bundle_2.12-0.13.0.jar
+```
+
+3. Download the Hudi Spark bundle
+```
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark-bundle_2.12/0.13.0/hudi-spark-bundle_2.12-0.13.0.jar
+```
+
+4. Download the shell script that launches Hudi CLI bundle
+```
+wget https://raw.githubusercontent.com/apache/hudi/release-0.13.0/packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+```
+
+5. Launch Hudi CLI bundle with appropriate environment variables as follows:
+```
+CLIENT_JAR=$DATAPROC_DIR/lib/gcs-connector.jar CLI_BUNDLE_JAR=hudi-cli-bundle_2.12-0.13.0.jar SPARK_BUNDLE_JAR=hudi-spark-bundle_2.12-0.13.0.jar ./hudi-cli-with-bundle.sh
+```
+
+6. hudi->connect --path gs://path_to_some_table
+Metadata for table some_table loaded
+
+7. hudi:some_table->commits show --limit 5
+This command should show the recent commits, if the above steps work correctly.
+
+## Connect to a Kerberized cluster
+
+Before connecting to a Kerberized cluster, you can use **kerberos kinit** command. Following is the usage of this command.
+
+```shell
+hudi->help kerberos kinit
+NAME
+ kerberos kinit - Perform Kerberos authentication
+
+SYNOPSIS
+ kerberos kinit --krb5conf String [--principal String] [--keytab String]
+
+OPTIONS
+ --krb5conf String
+ Path to krb5.conf
+ [Optional, default = /etc/krb5.conf]
+
+ --principal String
+ Kerberos principal
+ [Mandatory]
+
+ --keytab String
+ Path to keytab
+ [Mandatory]
+```
+
+For example:
+
+```shell
+hudi->kerberos kinit --principal user/host@DOMAIN --keytab /etc/security/keytabs/user.keytab
+Perform Kerberos authentication
+Parameters:
+--krb5conf: /etc/krb5.conf
+--principal: user/host@DOMAIN
+--keytab: /etc/security/keytabs/user.keytab
+Kerberos current user: user/host@DOMAIN (auth:KERBEROS)
+Kerberos login user: user/host@DOMAIN (auth:KERBEROS)
+Kerberos authentication success
+```
+
+If you see "Kerberos authentication success" in the command output, it means Kerberos authentication has been successful.
+
+
+## Using hudi-cli
+
+To initialize a hudi table, use the following command.
+
+```java
+===================================================================
+* ___ ___ *
+* /\__\ ___ /\ \ ___ *
+* / / / /\__\ / \ \ /\ \ *
+* / /__/ / / / / /\ \ \ \ \ \ *
+* / \ \ ___ / / / / / \ \__\ / \__\ *
+* / /\ \ /\__\ / /__/ ___ / /__/ \ |__| / /\/__/ *
+* \/ \ \/ / / \ \ \ /\__\ \ \ \ / / / /\/ / / *
+* \ / / \ \ / / / \ \ / / / \ /__/ *
+* / / / \ \/ / / \ \/ / / \ \__\ *
+* / / / \ / / \ / / \/__/ *
+* \/__/ \/__/ \/__/ Apache Hudi CLI *
+* *
+===================================================================
+
+hudi->create --path /user/hive/warehouse/table1 --tableName hoodie_table_1 --tableType COPY_ON_WRITE
+.....
+```
+
+To see the description of hudi table, use the command:
+
+```java
+hudi:hoodie_table_1->desc
+18/09/06 15:57:19 INFO timeline.HoodieActiveTimeline: Loaded instants []
+ _________________________________________________________
+ | Property | Value |
+ |========================================================|
+ | basePath | ... |
+ | metaPath | ... |
+ | fileSystem | hdfs |
+ | hoodie.table.name | hoodie_table_1 |
+ | hoodie.table.type | COPY_ON_WRITE |
+ | hoodie.archivelog.folder| |
+```
+
+Following is a sample command to connect to a Hudi table contains uber trips.
+
+```java
+hudi:trips->connect --path /app/uber/trips
+
+16/10/05 23:20:37 INFO model.HoodieTableMetadata: All commits :HoodieCommits{commitList=[20161002045850, 20161002052915, 20161002055918, 20161002065317, 20161002075932, 20161002082904, 20161002085949, 20161002092936, 20161002105903, 20161002112938, 20161002123005, 20161002133002, 20161002155940, 20161002165924, 20161002172907, 20161002175905, 20161002190016, 20161002192954, 20161002195925, 20161002205935, 20161002215928, 20161002222938, 20161002225915, 20161002232906, 20161003003028, 201 [...]
+Metadata for table trips loaded
+```
+
+Once connected to the table, a lot of other commands become available. The shell has contextual autocomplete help (press TAB) and below is a list of all commands, few of which are reviewed in this section
+
+```shell
+hudi:trips->help
+* ! - Allows execution of operating system (OS) commands
+* // - Inline comment markers (start of line only)
+* ; - Inline comment markers (start of line only)
+* bootstrap index showmapping - Show bootstrap index mapping
+* bootstrap index showpartitions - Show bootstrap indexed partitions
+* bootstrap run - Run a bootstrap action for current Hudi table
+* clean showpartitions - Show partition level details of a clean
+* cleans refresh - Refresh table metadata
+* cleans run - run clean
+* cleans show - Show the cleans
+* clear - Clears the console
+* cls - Clears the console
+* clustering run - Run Clustering
+* clustering schedule - Schedule Clustering
+* clustering scheduleAndExecute - Run Clustering. Make a cluster plan first and execute that plan immediately
+* commit rollback - Rollback a commit
+* commits compare - Compare commits with another Hoodie table
+* commit show_write_stats - Show write stats of a commit
+* commit showfiles - Show file level details of a commit
+* commit showpartitions - Show partition level details of a commit
+* commits refresh - Refresh table metadata
+* commits show - Show the commits
+* commits showarchived - Show the archived commits
+* commits sync - Sync commits with another Hoodie table
+* compaction repair - Renames the files to make them consistent with the timeline as dictated by Hoodie metadata. Use when compaction unschedule fails partially.
+* compaction run - Run Compaction for given instant time
+* compaction schedule - Schedule Compaction
+* compaction scheduleAndExecute - Schedule compaction plan and execute this plan
+* compaction show - Shows compaction details for a specific compaction instant
+* compaction showarchived - Shows compaction details for a specific compaction instant
+* compactions show all - Shows all compactions that are in active timeline
+* compactions showarchived - Shows compaction details for specified time window
+* compaction unschedule - Unschedule Compaction
+* compaction unscheduleFileId - UnSchedule Compaction for a fileId
+* compaction validate - Validate Compaction
+* connect - Connect to a hoodie table
+* create - Create a hoodie table if not present
+* date - Displays the local date and time
+* desc - Describe Hoodie Table properties
+* downgrade table - Downgrades a table
+* exit - Exits the shell
+* export instants - Export Instants and their metadata from the Timeline
+* fetch table schema - Fetches latest table schema
+* hdfsparquetimport - Imports Parquet table to a hoodie table
+* help - List all commands usage
+* marker delete - Delete the marker
+* metadata create - Create the Metadata Table if it does not exist
+* metadata delete - Remove the Metadata Table
+* metadata init - Update the metadata table from commits since the creation
+* metadata list-files - Print a list of all files in a partition from the metadata
+* metadata list-partitions - List all partitions from metadata
+* metadata refresh - Refresh table metadata
+* metadata set - Set options for Metadata Table
+* metadata stats - Print stats about the metadata
+* metadata validate-files - Validate all files in all partitions from the metadata
+* quit - Exits the shell
+* refresh - Refresh table metadata
+* repair addpartitionmeta - Add partition metadata to a table, if not present
+* repair corrupted clean files - repair corrupted clean files
+* repair deduplicate - De-duplicate a partition path contains duplicates & produce repaired files to replace with
+* repair migrate-partition-meta - Migrate all partition meta file currently stored in text format to be stored in base file format. See HoodieTableConfig#PARTITION_METAFILE_USE_DATA_FORMAT.
+* repair overwrite-hoodie-props - Overwrite hoodie.properties with provided file. Risky operation. Proceed with caution!
+* savepoint create - Savepoint a commit
+* savepoint delete - Delete the savepoint
+* savepoint rollback - Savepoint a commit
+* savepoints refresh - Refresh table metadata
+* savepoints show - Show the savepoints
+* script - Parses the specified resource file and executes its commands
+* set - Set spark launcher env to cli
+* show archived commits - Read commits from archived files and show details
+* show archived commit stats - Read commits from archived files and show details
+* show env - Show spark launcher env by key
+* show envs all - Show spark launcher envs
+* show fsview all - Show entire file-system view
+* show fsview latest - Show latest file-system view
+* show logfile metadata - Read commit metadata from log files
+* show logfile records - Read records from log files
+* show rollback - Show details of a rollback instant
+* show rollbacks - List all rollback instants
+* stats filesizes - File Sizes. Display summary stats on sizes of files
+* stats wa - Write Amplification. Ratio of how many records were upserted to how many records were actually written
+* sync validate - Validate the sync by counting the number of records
+* system properties - Shows the shell's properties
+* table delete-configs - Delete the supplied table configs from the table.
+* table recover-configs - Recover table configs, from update/delete that failed midway.
+* table update-configs - Update the table configs with configs with provided file.
+* temp_delete - Delete view name
+* temp_query - query against created temp view
+* temp delete - Delete view name
+* temp query - query against created temp view
+* temps_show - Show all views name
+* temps show - Show all views name
+* upgrade table - Upgrades a table
+* utils loadClass - Load a class
+* version - Displays shell version
+
+hudi:trips->
+```
+
+
+### Inspecting Commits
+
+The task of upserting or inserting a batch of incoming records is known as a **commit** in Hudi. A commit provides basic atomicity guarantees such that only committed data is available for querying.
+Each commit has a monotonically increasing string/number called the **commit number**. Typically, this is the time at which we started the commit.
+
+To view some basic information about the last 10 commits,
+
+
+```java
+hudi:trips->commits show --sortBy "Total Bytes Written" --desc true --limit 10
+ ________________________________________________________________________________________________________________________________________________________________________
+ | CommitTime | Total Bytes Written| Total Files Added| Total Files Updated| Total Partitions Written| Total Records Written| Total Update Records Written| Total Errors|
+ |=======================================================================================================================================================================|
+ ....
+ ....
+ ....
+```
+
+At the start of each write, Hudi also writes a .inflight commit to the .hoodie folder. You can use the timestamp there to estimate how long the commit has been inflight
+
+
+```java
+$ hdfs dfs -ls /app/uber/trips/.hoodie/*.inflight
+-rw-r--r-- 3 vinoth supergroup 321984 2016-10-05 23:18 /app/uber/trips/.hoodie/20161005225920.inflight
+```
+
+
+### Drilling Down to a specific Commit
+
+To understand how the writes spread across specific partiions,
+
+
+```java
+hudi:trips->commit showpartitions --commit 20161005165855 --sortBy "Total Bytes Written" --desc true --limit 10
+ __________________________________________________________________________________________________________________________________________
+ | Partition Path| Total Files Added| Total Files Updated| Total Records Inserted| Total Records Updated| Total Bytes Written| Total Errors|
+ |=========================================================================================================================================|
+ ....
+ ....
+```
+
+If you need file level granularity , we can do the following
+
+
+```java
+hudi:trips->commit showfiles --commit 20161005165855 --sortBy "Partition Path"
+ ________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File ID | Previous Commit| Total Records Updated| Total Records Written| Total Bytes Written| Total Errors|
+ |=======================================================================================================================================================|
+ ....
+ ....
+```
+
+
+### FileSystem View
+
+Hudi views each partition as a collection of file-groups with each file-group containing a list of file-slices in commit order (See concepts).
+The below commands allow users to view the file-slices for a data-set.
+
+```java
+hudi:stock_ticks_mor->show fsview all
+ ....
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta File Size| Delta Files |
+ |==============================================================================================================================================================================================================================================================================================================================================================================================================|
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]|
+
+
+
+hudi:stock_ticks_mor->show fsview latest --partitionPath "2018/08/31"
+ ......
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ [...]
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta Size| Delta Size - compaction scheduled| Delta Size - compaction unscheduled| Delta To Base Ratio - compaction scheduled| Delta To Base Ratio - compaction unscheduled| Delta Files - compaction scheduled | Delta Files - compaction unscheduled|
+ |========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== [...]
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | 20.8 KB | 0.0 B | 0.0 B | 0.0 B | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]| [] |
+
+```
+
+
+### Statistics
+
+Since Hudi directly manages file sizes for DFS table, it might be good to get an overall picture
+
+
+```java
+hudi:trips->stats filesizes --partitionPath 2016/09/01 --sortBy "95th" --desc true --limit 10
+ ________________________________________________________________________________________________
+ | CommitTime | Min | 10th | 50th | avg | 95th | Max | NumFiles| StdDev |
+ |===============================================================================================|
+ | <COMMIT_ID> | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 2 | 2.3 KB |
+ ....
+ ....
+```
+
+In case of Hudi write taking much longer, it might be good to see the write amplification for any sudden increases
+
+
+```java
+hudi:trips->stats wa
+ __________________________________________________________________________
+ | CommitTime | Total Upserted| Total Written| Write Amplifiation Factor|
+ |=========================================================================|
+ ....
+ ....
+```
+
+
+### Archived Commits
+
+In order to limit the amount of growth of .commit files on DFS, Hudi archives older .commit files (with due respect to the cleaner policy) into a commits.archived file.
+This is a sequence file that contains a mapping from commitNumber => json with raw information about the commit (same that is nicely rolled up above).
+
+
+### Compactions
+
+To get an idea of the lag between compaction and writer applications, use the below command to list down all
+pending compactions.
+
+```java
+hudi:trips->compactions show all
+ ___________________________________________________________________
+ | Compaction Instant Time| State | Total FileIds to be Compacted|
+ |==================================================================|
+ | <INSTANT_1> | REQUESTED| 35 |
+ | <INSTANT_2> | INFLIGHT | 27 |
+```
+
+To inspect a specific compaction plan, use
+
+```java
+hudi:trips->compaction show --instant <INSTANT_1>
+ _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File Id | Base Instant | Data File Path | Total Delta Files| getMetrics |
+ |================================================================================================================================================================================================================================================
+ | 2018/07/17 | <UUID> | <INSTANT_1> | viewfs://ns-default/.../../UUID_<INSTANT>.parquet | 1 | {TOTAL_LOG_FILES=1.0, TOTAL_IO_READ_MB=1230.0, TOTAL_LOG_FILES_SIZE=2.51255751E8, TOTAL_IO_WRITE_MB=991.0, TOTAL_IO_MB=2221.0}|
+
+```
+
+To manually schedule or run a compaction, use the below command. This command uses spark launcher to perform compaction
+operations.
+
+**NOTE:** Make sure no other application is scheduling compaction for this table concurrently
+\{: .notice--info}
+
+```java
+hudi:trips->help compaction schedule
+Keyword: compaction schedule
+Description: Schedule Compaction
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: false
+ Default if specified: '__NULL__'
+ Default if unspecified: '1G'
+
+* compaction schedule - Schedule Compaction
+```
+
+```java
+hudi:trips->help compaction run
+Keyword: compaction run
+Description: Run Compaction for given instant time
+ Keyword: tableName
+ Help: Table name
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: parallelism
+ Help: Parallelism for hoodie compaction
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: schemaFilePath
+ Help: Path for Avro schema file
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: retry
+ Help: Number of retries
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: compactionInstant
+ Help: Base path for the target hoodie table
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+* compaction run - Run Compaction for given instant time
+```
+
+### Validate Compaction
+
+Validating a compaction plan : Check if all the files necessary for compactions are present and are valid
+
+```java
+hudi:stock_ticks_mor->compaction validate --instant 20181005222611
+...
+
+ COMPACTION PLAN VALID
+
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error|
+ |==========================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | true | |
+
+
+
+hudi:stock_ticks_mor->compaction validate --instant 20181005222601
+
+ COMPACTION PLAN INVALID
+
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error |
+ |=====================================================================================================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | false| All log files specified in compaction operation is not present. Missing .... |
+```
+
+**NOTE:** The following commands must be executed without any other writer/ingestion application running.
+\{: .notice--warning}
+
+Sometimes, it becomes necessary to remove a fileId from a compaction-plan inorder to speed-up or unblock compaction
+operation. Any new log-files that happened on this file after the compaction got scheduled will be safely renamed
+so that are preserved. Hudi provides the following CLI to support it
+
+
+### Unscheduling Compaction
+
+```java
+hudi:trips->compaction unscheduleFileId --fileId <FileUUID>
+....
+No File renames needed to unschedule file from pending compaction. Operation successful.
+```
+
+In other cases, an entire compaction plan needs to be reverted. This is supported by the following CLI
+
+```java
+hudi:trips->compaction unschedule --instant <compactionInstant>
+.....
+No File renames needed to unschedule pending compaction. Operation successful.
+```
+
+### Repair Compaction
+
+The above compaction unscheduling operations could sometimes fail partially (e:g -> DFS temporarily unavailable). With
+partial failures, the compaction operation could become inconsistent with the state of file-slices. When you run
+`compaction validate`, you can notice invalid compaction operations if there is one. In these cases, the repair
+command comes to the rescue, it will rearrange the file-slices so that there is no loss and the file-slices are
+consistent with the compaction plan
+
+```java
+hudi:stock_ticks_mor->compaction repair --instant 20181005222611
+......
+Compaction successfully repaired
+.....
+```
+
+### Savepoint and Restore
+As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
+
+To trigger savepoint for a hudi table
+```java
+connect --path /tmp/hudi_trips_cow/
+commits show
+set --conf SPARK_HOME=<SPARK_HOME>
+savepoint create --commit 20220128160245447 --sparkMaster local[2]
+```
+
+To restore the table to one of the savepointed commit:
+
+```java
+connect --path /tmp/hudi_trips_cow/
+commits show
+set --conf SPARK_HOME=<SPARK_HOME>
+savepoints show
+╔═══════════════════╗
+║ SavepointTime ║
+╠═══════════════════╣
+║ 20220128160245447 ║
+╚═══════════════════╝
+savepoint rollback --savepoint 20220128160245447 --sparkMaster local[2]
+```
+
+### Upgrade and Downgrade Table
+In case the user needs to downgrade the version of Hudi library used, the Hudi table needs to be manually downgraded
+on the newer version of Hudi CLI before library downgrade. To downgrade a Hudi table through CLI, user needs to specify
+the target Hudi table version as follows:
+
+```shell
+connect --path <table_path>
+downgrade table --toVersion <target_version>
+```
+
+The following table shows the Hudi table versions corresponding to the Hudi release versions:
+
+| Hudi Table Version | Hudi Release Version(s) |
+|:-------------------|:------------------------|
+| `FIVE` or `5` | 0.12.x |
+| `FOUR` or `4` | 0.11.x |
+| `THREE` or `3` | 0.10.x |
+| `TWO` or `2` | 0.9.x |
+| `ONE` or `1` | 0.6.x - 0.8.x |
+| `ZERO` or `0` | 0.5.x and below |
+
+For example, to downgrade a table from version `FIVE`(`5`) (current version) to `TWO`(`2`), you should run (use proper Spark master based
+on your environment)
+
+```shell
+downgrade table --toVersion TWO --sparkMaster local[2]
+```
+
+or
+
+```shell
+downgrade table --toVersion 2 --sparkMaster local[2]
+```
+
+You can verify the table version by looking at the `hoodie.table.version` property in `.hoodie/hoodie.properties` under
+the table path:
+
+```properties
+hoodie.table.version=2
+```
+
+Hudi CLI also provides the ability to manually upgrade a Hudi table. To upgrade a Hudi table through CLI:
+
+```shell
+upgrade table --toVersion <target_version>
+```
+
+:::note
+Table upgrade is automatically handled by the Hudi write client in different deployment modes such as DeltaStreamer
+after upgrading the Hudi library so that the user does not have to do manual upgrade. Such automatic table upgrade
+is the **recommended** way in general, instead of using `upgrade` CLI command.
+
+Table upgrade from table version ONE to TWO requires key generator related configs such as
+"hoodie.datasource.write.recordkey.field", which is only available when user configures the write job. So the table
+upgrade from version ONE to TWO through CLI is not supported, and user should rely on the automatic upgrade in the write
+client instead.
+:::
+
+You may also run the upgrade command without specifying the target version. In such a case, the latest table version
+corresponding to the library release version is used:
+
+```shell
+upgrade table
+```
+
+### Change Hudi Table Type
+There are cases we want to change the hudi table type. For example, change COW table to MOR for more efficient and
+lower latency ingestion; change MOR to COW for better read performance and compatibility with downstream engines.
+So we offer the table command to perform this modification conveniently.
+
+Changing **COW to MOR**, we can simply modify the `hoodie.table.type` in `hoodie.properties` to MERGE_ON_READ.
+
+While changing **MOR to COW**, we must make sure all the log files are compacted before modifying the table type,
+or it will cause data loss.
+
+```shell
+connect --path <table_path>
+table change-table-type <target_table_type>
+```
+
+The parameter `target_table_type` candidates are below:
+
+| target table type | comment [...]
+|:------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| MOR | Change COW table to MERGE_ON_READ. [...]
+| COW | Change MOR table to COPY_ON_WRITE. <br/>By default, changing to COW will **execute all pending compactions** and **perform a full compaction** if any log file left. Set `--enable-compaction=false` will disable the default compaction. <br/>There are params can be set for the compaction operation:<br/>`--parallelism`: Default `3`. Parallelism for hoodie compaction<br/>`--sparkMaster`: Default `local`. Spark Master<br/>`--sparkMemory`: Default `4G`. Spark executor memo [...]
+
+
+Example below is changing MOR table to COW:
+```shell
+connect --path /var/dataset/test_table_mor2cow
+desc
+╔════════════════════════════════════════════════╤═════════════════════════════════════════╗
+║ Property │ Value ║
+╠════════════════════════════════════════════════╪═════════════════════════════════════════╣
+║ basePath │ /var/dataset/test_table_mor2cow ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ metaPath │ /var/dataset/test_table_mor2cow/.hoodie ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ fileSystem │ file ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.name │ test_table ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.compaction.record.merger.strategy │ eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.metadata.partitions │ files ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.type │ MERGE_ON_READ ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.metadata.partitions.inflight │ ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.archivelog.folder │ archived ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.timeline.layout.version │ 1 ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.checksum │ 2702201862 ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.compaction.payload.type │ HOODIE_AVRO ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.version │ 6 ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.datasource.write.drop.partition.columns │ false ║
+╚════════════════════════════════════════════════╧═════════════════════════════════════════╝
+
+table change-table-type COW
+╔════════════════════════════════════════════════╤══════════════════════════════════════╤══════════════════════════════════════╗
+║ Property │ Old Value │ New Value ║
+╠════════════════════════════════════════════════╪══════════════════════════════════════╪══════════════════════════════════════╣
+║ hoodie.archivelog.folder │ archived │ archived ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.compaction.payload.type │ HOODIE_AVRO │ HOODIE_AVRO ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.compaction.record.merger.strategy │ eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 │ eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.datasource.write.drop.partition.columns │ false │ false ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.checksum │ 2702201862 │ 2702201862 ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.metadata.partitions │ files │ files ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.metadata.partitions.inflight │ │ ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.name │ test_table │ test_table ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.type │ MERGE_ON_READ │ COPY_ON_WRITE ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.version │ 6 │ 6 ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.timeline.layout.version │ 1 │ 1 ║
+╚════════════════════════════════════════════════╧══════════════════════════════════════╧══════════════════════════════════════╝
+```
diff --git a/website/versioned_docs/version-1.0.0/cloud.md b/website/versioned_docs/version-1.0.0/cloud.md
new file mode 100644
index 00000000000..123abd5e6be
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/cloud.md
@@ -0,0 +1,38 @@
+---
+title: Cloud Storage
+keywords: [hudi, aws, gcp, oss, azure, cloud, juicefs]
+summary: "In this page, we introduce how Hudi work with different Cloud providers."
+toc: true
+last_modified_at: 2021-10-12T10:50:00+08:00
+---
+
+## Talking to Cloud Storage
+
+Immaterial of whether RDD/WriteClient APIs or Datasource is used, the following information helps configure access
+to cloud stores.
+
+* [AWS S3](s3_hoodie) <br/>
+ Configurations required for S3 and Hudi co-operability.
+* [Google Cloud Storage](gcs_hoodie) <br/>
+ Configurations required for GCS and Hudi co-operability.
+* [Alibaba Cloud OSS](oss_hoodie) <br/>
+ Configurations required for OSS and Hudi co-operability.
+* [Microsoft Azure](azure_hoodie) <br/>
+ Configurations required for Azure and Hudi co-operability.
+* [Tencent Cloud Object Storage](cos_hoodie) <br/>
+ Configurations required for COS and Hudi co-operability.
+* [IBM Cloud Object Storage](ibm_cos_hoodie) <br/>
+ Configurations required for IBM Cloud Object Storage and Hudi co-operability.
+* [Baidu Cloud Object Storage](bos_hoodie) <br/>
+ Configurations required for BOS and Hudi co-operability.
+* [JuiceFS](jfs_hoodie) <br/>
+ Configurations required for JuiceFS and Hudi co-operability.
+* [Oracle Cloud Infrastructure](oci_hoodie) <br/>
+ Configurations required for OCI and Hudi co-operability.
+
+:::note
+Many cloud object storage systems like [Amazon S3](https://docs.aws.amazon.com/s3/) allow you to set
+lifecycle policies, such as [S3 Lifecycle](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html),
+to manage objects. One of the policies is related to object expiration. If your organisation has configured such policies,
+then please ensure to exclude (or have a longer expiry period) for Hudi tables.
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-1.0.0/clustering.md b/website/versioned_docs/version-1.0.0/clustering.md
new file mode 100644
index 00000000000..0bbbad9781a
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/clustering.md
@@ -0,0 +1,346 @@
+---
+title: Clustering
+summary: "In this page, we describe async compaction in Hudi."
+toc: true
+last_modified_at:
+---
+
+## Background
+
+Apache Hudi brings stream processing to big data, providing fresh data while being an order of magnitude efficient over traditional batch processing. In a data lake/warehouse, one of the key trade-offs is between ingestion speed and query performance. Data ingestion typically prefers small files to improve parallelism and make data available to queries as soon as possible. However, query performance degrades poorly with a lot of small files. Also, during ingestion, data is typically co-l [...]
+<!--truncate-->
+
+## How is compaction different from clustering?
+
+Hudi is modeled like a log-structured storage engine with multiple versions of the data.
+Particularly, [Merge-On-Read](/docs/table_types#merge-on-read-table)
+tables in Hudi store data using a combination of base file in columnar format and row-based delta logs that contain
+updates. Compaction is a way to merge the delta logs with base files to produce the latest file slices with the most
+recent snapshot of data. Compaction helps to keep the query performance in check (larger delta log files would incur
+longer merge times on query side). On the other hand, clustering is a data layout optimization technique. One can stitch
+together small files into larger files using clustering. Additionally, data can be clustered by sort key so that queries
+can take advantage of data locality.
+
+## Clustering Architecture
+
+At a high level, Hudi provides different operations such as insert/upsert/bulk_insert through it’s write client API to be able to write data to a Hudi table. To be able to choose a trade-off between file size and ingestion speed, Hudi provides a knob `hoodie.parquet.small.file.limit` to be able to configure the smallest allowable file size. Users are able to configure the small file [soft limit](https://hudi.apache.org/docs/configurations/#hoodieparquetsmallfilelimit) to `0` to force new [...]
+
+
+
+To be able to support an architecture that allows for fast ingestion without compromising query performance, we have introduced a ‘clustering’ service to rewrite the data to optimize Hudi data lake file layout.
+
+Clustering table service can run asynchronously or synchronously adding a new action type called “REPLACE”, that will mark the clustering action in the Hudi metadata timeline.
+
+
+
+### Overall, there are 2 steps to clustering
+
+1. Scheduling clustering: Create a clustering plan using a pluggable clustering strategy.
+2. Execute clustering: Process the plan using an execution strategy to create new files and replace old files.
+
+
+### Schedule clustering
+
+Following steps are followed to schedule clustering.
+
+1. Identify files that are eligible for clustering: Depending on the clustering strategy chosen, the scheduling logic will identify the files eligible for clustering.
+2. Group files that are eligible for clustering based on specific criteria. Each group is expected to have data size in multiples of ‘targetFileSize’. Grouping is done as part of ‘strategy’ defined in the plan. Additionally, there is an option to put a cap on group size to improve parallelism and avoid shuffling large amounts of data.
+3. Finally, the clustering plan is saved to the timeline in an avro [metadata format](https://github.com/apache/hudi/blob/master/hudi-common/src/main/avro/HoodieClusteringPlan.avsc).
+
+
+### Execute clustering
+
+1. Read the clustering plan and get the ‘clusteringGroups’ that mark the file groups that need to be clustered.
+2. For each group, we instantiate appropriate strategy class with strategyParams (example: sortColumns) and apply that strategy to rewrite the data.
+3. Create a “REPLACE” commit and update the metadata in [HoodieReplaceCommitMetadata](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieReplaceCommitMetadata.java).
+
+
+Clustering Service builds on Hudi’s MVCC based design to allow for writers to continue to insert new data while clustering action runs in the background to reformat data layout, ensuring snapshot isolation between concurrent readers and writers.
+
+NOTE: Clustering can only be scheduled for tables / partitions not receiving any concurrent updates. In the future, concurrent updates use-case will be supported as well.
+
+
+_Figure: Illustrating query performance improvements by clustering_
+
+## Clustering Usecases
+
+### Batching small files
+
+As mentioned in the intro, streaming ingestion generally results in smaller files in your data lake. But having a lot of
+such small files could lead to higher query latency. From our experience supporting community users, there are quite a
+few users who are using Hudi just for small file handling capabilities. So, you could employ clustering to batch a lot
+of such small files into larger ones.
+
+
+
+### Cluster by sort key
+
+Another classic problem in data lake is the arrival time vs event time problem. Generally you write data based on
+arrival time, while query predicates do not sit well with it. With clustering, you can re-write your data by sorting
+based on query predicates and so, your data skipping will be very efficient and your query can ignore scanning a lot of
+unnecessary data.
+
+
+
+## Clustering Strategies
+
+On a high level, clustering creates a plan based on a configurable strategy, groups eligible files based on specific
+criteria and then executes the plan. As mentioned before, clustering plan as well as execution depends on configurable
+strategy. These strategies can be broadly classified into three types: clustering plan strategy, execution strategy and
+update strategy.
+
+### Plan Strategy
+
+This strategy comes into play while creating clustering plan. It helps to decide what file groups should be clustered
+and how many output file groups should the clustering produce. Note that these strategies are easily pluggable using the
+config [hoodie.clustering.plan.strategy.class](/docs/configurations#hoodieclusteringplanstrategyclass).
+
+Different plan strategies are as follows:
+
+#### Size-based clustering strategies
+
+This strategy creates clustering groups based on max size allowed per group. Also, it excludes files that are greater
+than the small file limit from the clustering plan. Available strategies depending on write client
+are: `SparkSizeBasedClusteringPlanStrategy`, `FlinkSizeBasedClusteringPlanStrategy`
+and `JavaSizeBasedClusteringPlanStrategy`. Furthermore, Hudi provides flexibility to include or exclude partitions for
+clustering, tune the file size limits, maximum number of output groups. Please refer to [hoodie.clustering.plan.strategy.small.file.limit](https://hudi.apache.org/docs/next/configurations/#hoodieclusteringplanstrategysmallfilelimit)
+, [hoodie.clustering.plan.strategy.max.num.groups](https://hudi.apache.org/docs/next/configurations/#hoodieclusteringplanstrategymaxnumgroups), [hoodie.clustering.plan.strategy.max.bytes.per.group](https://hudi.apache.org/docs/next/configurations/#hoodieclusteringplanstrategymaxbytespergroup)
+, [hoodie.clustering.plan.strategy.target.file.max.bytes](https://hudi.apache.org/docs/next/configurations/#hoodieclusteringplanstrategytargetfilemaxbytes) for more details.
+
+| Config Name | Default | Description [...]
+|---------------------------------------------------------| -------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.clustering.plan.strategy.partition.selected | N/A **(Required)** | Comma separated list of partitions to run clustering<br /><br />`Config Param: PARTITION_SELECTED`<br />`Since Version: 0.11.0` [...]
+| hoodie.clustering.plan.strategy.partition.regex.pattern | N/A **(Required)** | Filter clustering partitions that matched regex pattern<br /><br />`Config Param: PARTITION_REGEX_PATTERN`<br />`Since Version: 0.11.0` [...]
+| hoodie.clustering.plan.partition.filter.mode | NONE (Optional) | Partition filter mode used in the creation of clustering plan. Possible values:<br /><ul><li>`NONE`: Do not filter partitions. The clustering plan will include all partitions that have clustering candidates.</li><li>`RECENT_DAYS`: This filter assumes that your data is partitioned by date. The clustering plan will only include partitions from K days ago to N days ago, where K >= N. K is determined by `hood [...]
+
+
+#### SparkSingleFileSortPlanStrategy
+
+In this strategy, clustering group for each partition is built in the same way as `SparkSizeBasedClusteringPlanStrategy`
+. The difference is that the output group is 1 and file group id remains the same,
+while `SparkSizeBasedClusteringPlanStrategy` can create multiple file groups with newer fileIds.
+
+#### SparkConsistentBucketClusteringPlanStrategy
+
+This strategy is specifically used for consistent bucket index. This will be leveraged to expand your bucket index (from
+static partitioning to dynamic). Typically, users don’t need to use this strategy. Hudi internally uses this for
+dynamically expanding the buckets for bucket index datasets.
+
+:::note The latter two strategies are applicable only for the Spark engine.
+:::
+
+### Execution Strategy
+
+After building the clustering groups in the planning phase, Hudi applies execution strategy, for each group, primarily
+based on sort columns and size. The strategy can be specified using the
+config [hoodie.clustering.execution.strategy.class](/docs/configurations/#hoodieclusteringexecutionstrategyclass). By
+default, Hudi sorts the file groups in the plan by the specified columns, while meeting the configured target file
+sizes.
+
+| Config Name | Default | Description [...]
+| --------------------------------------------| ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.clustering.execution.strategy.class | org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy (Optional) | Config to provide a strategy class (subclass of RunClusteringStrategy) to define how the clustering plan is executed. By default, we sort the file groups in th plan by the specified columns, while meeting the configured target file sizes.<br /><br />`Config Param: EXECUTION_STRATEGY_CLASS_NAME`<br />`Since Version: 0.7.0` [...]
+
+The available strategies are as follows:
+
+1. `SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY`: Uses bulk_insert to re-write data from input file groups.
+ 1. Set `hoodie.clustering.execution.strategy.class`
+ to `org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy`.
+ 2. `hoodie.clustering.plan.strategy.sort.columns`: Columns to sort the data while clustering. This goes in
+ conjunction with layout optimization strategies depending on your query predicates. One can set comma separated
+ list of columns that needs to be sorted in this config.
+2. `JAVA_SORT_AND_SIZE_EXECUTION_STRATEGY`: Similar to `SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY`, for the Java and Flink
+ engines. Set `hoodie.clustering.execution.strategy.class`
+ to `org.apache.hudi.client.clustering.run.strategy.JavaSortAndSizeExecutionStrategy`.
+3. `SPARK_CONSISTENT_BUCKET_EXECUTION_STRATEGY`: As the name implies, this is applicable to dynamically expand
+ consistent bucket index and only applicable to the Spark engine. Set `hoodie.clustering.execution.strategy.class`
+ to `org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+
+### Update Strategy
+
+Currently, clustering can only be scheduled for tables/partitions not receiving any concurrent updates. By default,
+the config for update strategy - [`hoodie.clustering.updates.strategy`](/docs/configurations/#hoodieclusteringupdatesstrategy) is set to ***SparkRejectUpdateStrategy***. If some file group has updates during clustering then it will reject updates and throw an
+exception. However, in some use-cases updates are very sparse and do not touch most file groups. The default strategy to
+simply reject updates does not seem fair. In such use-cases, users can set the config to ***SparkAllowUpdateStrategy***.
+
+We discussed the critical strategy configurations. All other configurations related to clustering are
+listed [here](/docs/configurations/#Clustering-Configs). Out of this list, a few configurations that will be very useful
+for inline or async clustering are shown below with code samples.
+
+## Inline clustering
+
+Inline clustering happens synchronously with the regular ingestion writer or as part of the data ingestion pipeline. This means the next round of ingestion cannot proceed until the clustering is complete With inline clustering, Hudi will schedule, plan clustering operations after each commit is completed and execute the clustering plans after it’s created. This is the simplest deployment model to run because it’s easier to manage than running different asynchronous Spark jobs. This mode [...]
+
+For this deployment mode, please enable and set: `hoodie.clustering.inline`
+
+To choose how often clustering is triggered, also set: `hoodie.clustering.inline.max.commits`.
+
+Inline clustering can be setup easily using spark dataframe options.
+See sample below:
+
+```scala
+import org.apache.hudi.QuickstartUtils._
+import scala.collection.JavaConversions._
+import org.apache.spark.sql.SaveMode._
+import org.apache.hudi.DataSourceReadOptions._
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.config.HoodieWriteConfig._
+
+
+val df = //generate data frame
+df.write.format("org.apache.hudi").
+ options(getQuickstartWriteConfigs).
+ option("hoodie.datasource.write.precombine.field", "ts").
+ option("hoodie.datasource.write.recordkey.field", "uuid").
+ option("hoodie.datasource.write.partitionpath.field", "partitionpath").
+ option("hoodie.table.name", "tableName").
+ option("hoodie.parquet.small.file.limit", "0").
+ option("hoodie.clustering.inline", "true").
+ option("hoodie.clustering.inline.max.commits", "4").
+ option("hoodie.clustering.plan.strategy.target.file.max.bytes", "1073741824").
+ option("hoodie.clustering.plan.strategy.small.file.limit", "629145600").
+ option("hoodie.clustering.plan.strategy.sort.columns", "column1,column2"). //optional, if sorting is needed as part of rewriting data
+ mode(Append).
+ save("dfs://location");
+```
+
+## Async Clustering
+
+Async clustering runs the clustering table service in the background without blocking the regular ingestions writers. There are three different ways to deploy an asynchronous clustering process:
+
+- **Asynchronous execution within the same process**: In this deployment mode, Hudi will schedule and plan the clustering operations after each commit is completed as part of the ingestion pipeline. Separately, Hudi spins up another thread within the same job and executes the clustering table service. This is supported by Spark Streaming, Flink and DeltaStreamer in continuous mode. For this deployment mode, please enable `hoodie.clustering.async.enabled` and `hoodie.clustering.async.max. [...]
+- **Asynchronous scheduling and execution by a separate process**: In this deployment mode, the application will write data to a Hudi table as part of the ingestion pipeline. A separate clustering job will schedule, plan and execute the clustering operation. By running a different job for the clustering operation, it rebalances how Hudi uses compute resources: fewer compute resources are needed for the ingestion, which makes ingestion latency stable, and an independent set of compute res [...]
+- **Scheduling inline and executing async**: In this deployment mode, the application ingests data and schedules the clustering in one job; in another, the application executes the clustering plan. The supported writers (see below) won’t be blocked from ingesting data. If the metadata table is enabled, a lock provider is not needed. However, if the metadata table is enabled, please ensure all jobs have the lock providers configured for concurrency control. All writers support this deploy [...]
+
+Hudi supports [multi-writers](https://hudi.apache.org/docs/concurrency_control#enabling-multi-writing) which provides
+snapshot isolation between multiple table services, thus allowing writers to continue with ingestion while clustering
+runs in the background.
+
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.clustering.async.enabled | false (Optional) | Enable running of clustering service, asynchronously as inserts happen on the table.<br /><br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| hoodie.clustering.async.max.commits | 4 (Optional) | Config to control frequency of async clustering<br /><br />`Config Param: ASYNC_CLUSTERING_MAX_COMMITS`<br />`Since Version: 0.9.0` [...]
+
+## Setup Asynchronous Clustering
+Users can leverage [HoodieClusteringJob](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+19+Clustering+data+for+freshness+and+query+performance#RFC19Clusteringdataforfreshnessandqueryperformance-SetupforAsyncclusteringJob)
+to setup 2-step asynchronous clustering.
+
+### HoodieClusteringJob
+By specifying the `scheduleAndExecute` mode both schedule as well as clustering can be achieved in the same step.
+The appropriate mode can be specified using `-mode` or `-m` option. There are three modes:
+
+1. `schedule`: Make a clustering plan. This gives an instant which can be passed in execute mode.
+2. `execute`: Execute a clustering plan at a particular instant. If no instant-time is specified, HoodieClusteringJob will execute for the earliest instant on the Hudi timeline.
+3. `scheduleAndExecute`: Make a clustering plan first and execute that plan immediately.
+
+Note that to run this job while the original writer is still running, please enable multi-writing:
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
+```
+
+A sample spark-submit command to setup HoodieClusteringJob is as below:
+
+```bash
+spark-submit \
+--jars "packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle_2.12-1.0.0.jar,packaging/hudi-spark-bundle/target/hudi-spark3.5-bundle_2.12-1.0.0.jar" \
+--class org.apache.hudi.utilities.HoodieClusteringJob \
+/path/to/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle_2.12-1.0.0.jar \
+--props /path/to/config/clusteringjob.properties \
+--mode scheduleAndExecute \
+--base-path /path/to/hudi_table/basePath \
+--table-name hudi_table_schedule_clustering \
+--spark-memory 1g
+```
+A sample `clusteringjob.properties` file:
+```
+hoodie.clustering.async.enabled=true
+hoodie.clustering.async.max.commits=4
+hoodie.clustering.plan.strategy.target.file.max.bytes=1073741824
+hoodie.clustering.plan.strategy.small.file.limit=629145600
+hoodie.clustering.execution.strategy.class=org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy
+hoodie.clustering.plan.strategy.sort.columns=column1,column2
+```
+
+### HoodieStreamer
+
+This brings us to our users' favorite utility in Hudi. Now, we can trigger asynchronous clustering with Hudi Streamer.
+Just set the `hoodie.clustering.async.enabled` config to true and specify other clustering config in properties file
+whose location can be pased as `—props` when starting the Hudi Streamer (just like in the case of HoodieClusteringJob).
+
+A sample spark-submit command to setup HoodieStreamer is as below:
+
+
+```bash
+spark-submit \
+--jars "packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle_2.12-1.0.0.jar,packaging/hudi-spark-bundle/target/hudi-spark3.5-bundle_2.12-1.0.0.jar" \
+--class org.apache.hudi.utilities.streamer.HoodieStreamer \
+/path/to/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle_2.12-1.0.0.jar \
+--props /path/to/config/clustering_kafka.properties \
+--schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+--source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+--source-ordering-field impresssiontime \
+--table-type COPY_ON_WRITE \
+--target-base-path /path/to/hudi_table/basePath \
+--target-table impressions_cow_cluster \
+--op INSERT \
+--hoodie-conf hoodie.clustering.async.enabled=true \
+--continuous
+```
+
+### Spark Structured Streaming
+
+We can also enable asynchronous clustering with Spark structured streaming sink as shown below.
+```scala
+val commonOpts = Map(
+ "hoodie.insert.shuffle.parallelism" -> "4",
+ "hoodie.upsert.shuffle.parallelism" -> "4",
+ "hoodie.datasource.write.recordkey.field" -> "_row_key",
+ "hoodie.datasource.write.partitionpath.field" -> "partition",
+ "hoodie.datasource.write.precombine.field" -> "timestamp",
+ "hoodie.table.name" -> "hoodie_test"
+)
+
+def getAsyncClusteringOpts(isAsyncClustering: String,
+ clusteringNumCommit: String,
+ executionStrategy: String):Map[String, String] = {
+ commonOpts + (DataSourceWriteOptions.ASYNC_CLUSTERING_ENABLE.key -> isAsyncClustering,
+ HoodieClusteringConfig.ASYNC_CLUSTERING_MAX_COMMITS.key -> clusteringNumCommit,
+ HoodieClusteringConfig.EXECUTION_STRATEGY_CLASS_NAME.key -> executionStrategy
+ )
+}
+
+def initStreamingWriteFuture(hudiOptions: Map[String, String]): Future[Unit] = {
+ val streamingInput = // define the source of streaming
+ Future {
+ println("streaming starting")
+ streamingInput
+ .writeStream
+ .format("org.apache.hudi")
+ .options(hudiOptions)
+ .option("checkpointLocation", basePath + "/checkpoint")
+ .mode(Append)
+ .start()
+ .awaitTermination(10000)
+ println("streaming ends")
+ }
+}
+
+def structuredStreamingWithClustering(): Unit = {
+ val df = //generate data frame
+ val hudiOptions = getClusteringOpts("true", "1", "org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy")
+ val f1 = initStreamingWriteFuture(hudiOptions)
+ Await.result(f1, Duration.Inf)
+}
+```
+
+## Java Client
+
+Clustering is also supported via Java client. Plan strategy `org.apache.hudi.client.clustering.plan.strategy.JavaSizeBasedClusteringPlanStrategy`
+and execution strategy `org.apache.hudi.client.clustering.run.strategy.JavaSortAndSizeExecutionStrategy` are supported
+out-of-the-box. Note that as of now only linear sort is supported in Java execution strategy.
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Understanding Clustering in Apache Hudi and the Benefits of Asynchronous Clustering](https://www.youtube.com/watch?v=R_sm4wlGXuE)
diff --git a/website/versioned_docs/version-1.0.0/compaction.md b/website/versioned_docs/version-1.0.0/compaction.md
new file mode 100644
index 00000000000..7859030052a
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/compaction.md
@@ -0,0 +1,228 @@
+---
+title: Compaction
+summary: "In this page, we describe async compaction in Hudi."
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+last_modified_at:
+---
+## Background
+Compaction is a table service employed by Hudi specifically in Merge On Read(MOR) tables to merge updates from row-based log
+files to the corresponding columnar-based base file periodically to produce a new version of the base file. Compaction is
+not applicable to Copy On Write(COW) tables and only applies to MOR tables.
+
+### Why MOR tables need compaction?
+To understand the significance of compaction in MOR tables, it is helpful to understand the MOR table layout first. In Hudi,
+data is organized in terms of [file groups](https://hudi.apache.org/docs/file_layouts/). Each file group in a MOR table
+consists of a base file and one or more log files. Typically, during writes, inserts are stored in the base file, and updates
+are appended to log files.
+
+
+_Figure: MOR table file layout showing different file groups with base data file and log files_
+
+During the compaction process, updates from the log files are merged with the base file to form a new version of the
+base file as shown below. Since MOR is designed to be write-optimized, on new writes, after index tagging is complete,
+Hudi appends the records pertaining to each file groups as log blocks in log files. There is no synchronous merge
+happening during write, resulting in a lower write amplification and better write latency. In contrast, on new writes to a
+COW table, Hudi combines the new writes with the older base file to produce a new version of the base file resulting in
+a higher write amplification and higher write latencies.
+
+
+_Figure: Compaction on a given file group_
+
+While serving the read query(snapshot read), for each file group, records in base file and all its corresponding log
+files are merged together and served. And hence the read latency for MOR snapshot query might be higher compared to
+COW table since there is no merge involved in case of COW at read time. Compaction takes care of merging the updates from
+log files with the base file at regular intervals to bound the growth of log files and to ensure the read latencies do not
+spike up.
+
+## Compaction Architecture
+There are two steps to compaction.
+- ***Compaction Scheduling***: In this step, Hudi scans the partitions and selects file slices to be compacted. A compaction
+ plan is finally written to Hudi timeline.
+- ***Compaction Execution***: In this step the compaction plan is read and file slices are compacted.
+
+### Strategies in Compaction Scheduling
+There are two strategies involved in scheduling the compaction:
+- Trigger Strategy: Determines how often to trigger scheduling of the compaction.
+- Compaction Strategy: Determines which file groups to compact.
+
+Hudi provides various options for both these strategies as discussed below.
+
+#### Trigger Strategies
+
+| Config Name | Default | Description |
+|----------------------------------------------------|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.compact.inline.trigger.strategy | NUM_COMMITS (Optional) | org.apache.hudi.table.action.compact.CompactionTriggerStrategy: Controls when compaction is scheduled.<br />`Config Param: INLINE_COMPACT_TRIGGER_STRATEGY` <br/>
+<ul><li>`NUM_COMMITS`: triggers compaction when there are at least N delta commits after last completed compaction.</li><li>`NUM_COMMITS_AFTER_LAST_REQUEST`: triggers compaction when there are at least N delta commits after last completed or requested compaction.</li><li>`TIME_ELAPSED`: triggers compaction after N seconds since last compaction.</li><li>`NUM_AND_TIME`: triggers compaction when both there are at least N delta commits and N seconds elapsed (both must be satisfied) after las [...]
+
+#### Compaction Strategies
+| Config Name | Default | Description [...]
+|----------------------------------------------------|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.compaction.strategy | org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy (Optional) | Compaction strategy decides which file groups are picked up for compaction during each compaction run. By default. Hudi picks the log file with most accumulated unmerged data. <br /><br />`Config Param: COMPACTION_STRATEGY` |
+
+Available Strategies (Provide the full package name when using the strategy): <ul><li>`LogFileNumBasedCompactionStrategy`:
+orders the compactions based on the total log files count, filters the file group with log files count greater than the
+threshold and limits the compactions within a configured IO bound.</li><li>`LogFileSizeBasedCompactionStrategy`: orders
+the compactions based on the total log files size, filters the file group which log files size is greater than the
+threshold and limits the compactions within a configured IO bound.</li><li>`BoundedIOCompactionStrategy`: CompactionStrategy
+which looks at total IO to be done for the compaction (read + write) and limits the list of compactions to be under a
+configured limit on the IO.</li><li>`BoundedPartitionAwareCompactionStrategy`:This strategy ensures that the last N partitions
+are picked up even if there are later partitions created for the table. lastNPartitions is defined as the N partitions before
+the currentDate. currentDay = 2018/01/01 The table has partitions for 2018/02/02 and 2018/03/03 beyond the currentDay This
+strategy will pick up the following partitions for compaction : (2018/01/01, allPartitionsInRange[(2018/01/01 - lastNPartitions)
+to 2018/01/01), 2018/02/02, 2018/03/03)</li><li>`DayBasedCompactionStrategy`:This strategy orders compactions in reverse
+order of creation of Hive Partitions. It helps to compact data in latest partitions first and then older capped at the
+Total_IO allowed.</li><li>`UnBoundedCompactionStrategy`: UnBoundedCompactionStrategy will not change ordering or filter
+any compaction. It is a pass-through and will compact all the base files which has a log file. This usually means
+no-intelligence on compaction.</li><li>`UnBoundedPartitionAwareCompactionStrategy`:UnBoundedPartitionAwareCompactionStrategy is a custom UnBounded Strategy. This will filter all the partitions that
+are eligible to be compacted by a \{@link BoundedPartitionAwareCompactionStrategy} and return the result. This is done
+so that a long running UnBoundedPartitionAwareCompactionStrategy does not step over partitions in a shorter running
+BoundedPartitionAwareCompactionStrategy. Essentially, this is an inverse of the partitions chosen in
+BoundedPartitionAwareCompactionStrategy</li></ul>
+
+:::note
+Please refer to [advanced configs](https://hudi.apache.org/docs/next/configurations#Compaction-Configs) for more details.
+:::
+
+## Ways to trigger Compaction
+
+### Inline
+By default, compaction is run asynchronously.
+
+If latency of ingesting records is important for you, you are most likely using Merge-On-Read tables.
+Merge-On-Read tables store data using a combination of columnar (e.g parquet) + row based (e.g avro) file formats.
+Updates are logged to delta files & later compacted to produce new versions of columnar files.
+To improve ingestion latency, Async Compaction is the default configuration.
+
+If immediate read performance of a new commit is important for you, or you want simplicity of not managing separate compaction jobs,
+you may want synchronous inline compaction, which means that as a commit is written it is also compacted by the same job.
+
+For this deployment mode, please use `hoodie.compact.inline = true` for Spark Datasource and Spark SQL writers. For
+HoodieStreamer sync once mode inline compaction can be achieved by passing the flag `--disable-compaction` (Meaning to
+disable async compaction). Further in HoodieStreamer when both
+ingestion and compaction is running in the same spark context, you can use resource allocation configuration
+in Hudi Streamer CLI such as (`--delta-sync-scheduling-weight`,
+`--compact-scheduling-weight`, `--delta-sync-scheduling-minshare`, and `--compact-scheduling-minshare`)
+to control executor allocation between ingestion and compaction.
+
+
+### Async & Offline Compaction models
+
+There are a couple of ways here to trigger compaction .
+
+#### Async execution within the same process
+In streaming ingestion write models like HoodieStreamer
+continuous mode, Flink and Spark Streaming, async compaction is enabled by default and runs alongside without blocking
+regular ingestion.
+
+##### Spark Structured Streaming
+
+Compactions are scheduled and executed asynchronously inside the
+streaming job.Here is an example snippet in java
+
+```properties
+import org.apache.hudi.DataSourceWriteOptions;
+import org.apache.hudi.HoodieDataSourceHelpers;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import org.apache.spark.sql.streaming.OutputMode;
+import org.apache.spark.sql.streaming.ProcessingTime;
+
+
+ DataStreamWriter<Row> writer = streamingInput.writeStream().format("org.apache.hudi")
+ .option("hoodie.datasource.write.operation", operationType)
+ .option("hoodie.datasource.write.table.type", tableType)
+ .option("hoodie.datasource.write.recordkey.field", "_row_key")
+ .option("hoodie.datasource.write.partitionpath.field", "partition")
+ .option("hoodie.datasource.write.precombine.field"(), "timestamp")
+ .option("hoodie.compact.inline.max.delta.commits", "10")
+ .option("hoodie.datasource.compaction.async.enable", "true")
+ .option("hoodie.table.name", tableName).option("checkpointLocation", checkpointLocation)
+ .outputMode(OutputMode.Append());
+ writer.trigger(new ProcessingTime(30000)).start(tablePath);
+```
+
+##### Hudi Streamer Continuous Mode
+Hudi Streamer provides continuous ingestion mode where a single long running spark application
+ingests data to Hudi table continuously from upstream sources. In this mode, Hudi supports managing asynchronous
+compactions. Here is an example snippet for running in continuous mode with async compactions
+
+```properties
+spark-submit --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
+--class org.apache.hudi.utilities.streamer.HoodieStreamer \
+--table-type MERGE_ON_READ \
+--target-base-path <hudi_base_path> \
+--target-table <hudi_table> \
+--source-class org.apache.hudi.utilities.sources.JsonDFSSource \
+--source-ordering-field ts \
+--props /path/to/source.properties \
+--continous
+```
+
+#### Scheduling and Execution by a separate process
+For some use cases with long running table services, instead of having the regular writes block, users have the option to run
+both steps of the compaction ([scheduling and execution](#compaction-architecture)) offline in a separate process altogether.
+This allows for regular writers to not bother about these compaction steps and allows users to provide more resources for
+the compaction job as needed.
+
+:::note
+This model needs a lock provider configured for all jobs - the regular writer as well as the offline compaction job.
+:::
+
+#### Scheduling inline and executing async
+
+In this model, it is possible for a Spark Datasource writer or a Flink job to just schedule the compaction inline ( that
+will serialize the compaction plan in the timeline but will not execute it). And then a separate utility like
+HudiCompactor or HoodieFlinkCompactor can take care of periodically executing the compaction plan.
+
+:::note
+This model may need a lock provider **if** metadata table is enabled.
+:::
+
+#### Hudi Compactor Utility
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
+The compactor utility allows to do scheduling and execution of compaction.
+
+Example:
+```properties
+spark-submit --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
+--class org.apache.hudi.utilities.HoodieCompactor \
+--base-path <base_path> \
+--table-name <table_name> \
+--schema-file <schema_file> \
+--instant-time <compaction_instant>
+```
+
+Note, the `instant-time` parameter is now optional for the Hudi Compactor Utility. If using the utility without `--instant time`,
+the spark-submit will execute the earliest scheduled compaction on the Hudi timeline.
+
+#### Hudi CLI
+Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](/docs/cli#compactions)
+
+Example:
+```properties
+hudi:trips->compaction run --tableName <table_name> --parallelism <parallelism> --compactionInstant <InstantTime>
+...
+```
+
+#### Flink Offline Compaction
+Offline compaction needs to submit the Flink task on the command line. The program entry is as follows: `hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar` :
+`org.apache.hudi.sink.compact.HoodieFlinkCompactor`
+
+```bash
+# Command line
+./bin/flink run -c org.apache.hudi.sink.compact.HoodieFlinkCompactor lib/hudi-flink-bundle_2.11-0.9.0.jar --path hdfs://xxx:9000/table
+```
+
+#### Options
+
+| Option Name | Default | Description |
+| ----------- |-------------------------------------------------------------------------------------------------------------------------------| ------- |
+| `--path` | `n/a **(Required)**` | The path where the target table is stored on Hudi |
+| `--compaction-max-memory` | `100` (Optional) | The index map size of log data during compaction, 100 MB by default. If you have enough memory, you can turn up this parameter |
+| `--schedule` | `false` (Optional) | whether to execute the operation of scheduling compaction plan. When the write process is still writing， turning on this parameter have a risk of losing data. Therefore, it must be ensured that there are no write tasks currently writing data to this table when this parameter is turned on |
+| `--seq` | `LIFO` (Optional) | The order in which compaction tasks are executed. Executing from the latest compaction plan by default. `LIFO`: executing from the latest plan. `FIFO`: executing from the oldest plan. |
+| `--service` | `false` (Optional) | Whether to start a monitoring service that checks and schedules new compaction task in configured interval. |
+| `--min-compaction-interval-seconds` | `600(s)` (optional) | The checking interval for service mode, by default 10 minutes. |
diff --git a/website/versioned_docs/version-1.0.0/comparison.md b/website/versioned_docs/version-1.0.0/comparison.md
new file mode 100644
index 00000000000..681b359a4de
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/comparison.md
@@ -0,0 +1,56 @@
+---
+title: "Comparison"
+keywords: [ apache, hudi, kafka, kudu, hive, hbase, stream processing]
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+
+Apache Hudi fills a big void for processing data on top of DFS, and thus mostly co-exists nicely with these technologies. However,
+it would be useful to understand how Hudi fits into the current big data ecosystem, contrasting it with a few related systems
+and bring out the different tradeoffs these systems have accepted in their design.
+
+## Kudu
+
+[Apache Kudu](https://kudu.apache.org) is a storage system that has similar goals as Hudi, which is to bring real-time analytics on petabytes of data via first
+class support for `upserts`. A key differentiator is that Kudu also attempts to serve as a datastore for OLTP workloads, something that Hudi does not aspire to be.
+Consequently, Kudu does not support incremental pulling (as of early 2017), something Hudi does to enable incremental processing use cases.
+
+
+Kudu diverges from a distributed file system abstraction and HDFS altogether, with its own set of storage servers talking to each other via RAFT.
+Hudi, on the other hand, is designed to work with an underlying Hadoop compatible filesystem (HDFS,S3 or Ceph) and does not have its own fleet of storage servers,
+instead relying on Apache Spark to do the heavy-lifting. Thus, Hudi can be scaled easily, just like other Spark jobs, while Kudu would require hardware
+& operational support, typical to datastores like HBase or Vertica. We have not at this point, done any head to head benchmarks against Kudu (given RTTable is WIP).
+But, if we were to go with results shared by [CERN](https://db-blog.web.cern.ch/blog/zbigniew-baranowski/2017-01-performance-comparison-different-file-formats-and-storage-engines) ,
+we expect Hudi to positioned at something that ingests parquet with superior performance.
+
+
+## Hive Transactions
+
+[Hive Transactions/ACID](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions) is another similar effort, which tries to implement storage like
+`merge-on-read`, on top of ORC file format. Understandably, this feature is heavily tied to Hive and other efforts like [LLAP](https://cwiki.apache.org/confluence/display/Hive/LLAP).
+Hive transactions does not offer the read-optimized storage option or the incremental pulling, that Hudi does. In terms of implementation choices, Hudi leverages
+the full power of a processing framework like Spark, while Hive transactions feature is implemented underneath by Hive tasks/queries kicked off by user or the Hive metastore.
+Based on our production experience, embedding Hudi as a library into existing Spark pipelines was much easier and less operationally heavy, compared with the other approach.
+Hudi is also designed to work with non-hive engines like PrestoDB/Spark and will incorporate file formats other than parquet over time.
+
+## HBase
+
+Even though [HBase](https://hbase.apache.org) is ultimately a key-value store for OLTP workloads, users often tend to associate HBase with analytics given the proximity to Hadoop.
+Given HBase is heavily write-optimized, it supports sub-second upserts out-of-box and Hive-on-HBase lets users query that data. However, in terms of actual performance for analytical workloads,
+hybrid columnar storage formats like Parquet/ORC handily beat HBase, since these workloads are predominantly read-heavy. Hudi bridges this gap between faster data and having
+analytical storage formats. From an operational perspective, arming users with a library that provides faster data, is more scalable, than managing a big farm of HBase region servers,
+just for analytics. Finally, HBase does not support incremental processing primitives like `commit times`, `incremental pull` as first class citizens like Hudi.
+
+## Stream Processing
+
+A popular question, we get is : "How does Hudi relate to stream processing systems?", which we will try to answer here. Simply put, Hudi can integrate with
+batch (`copy-on-write table`) and streaming (`merge-on-read table`) jobs of today, to store the computed results in Hadoop. For Spark apps, this can happen via direct
+integration of Hudi library with Spark/Spark streaming DAGs. In case of Non-Spark processing systems (eg: Flink, Hive), the processing can be done in the respective systems
+and later sent into a Hudi table via a Kafka topic/DFS intermediate file. In more conceptual level, data processing
+pipelines just consist of three components : `source`, `processing`, `sink`, with users ultimately running queries against the sink to use the results of the pipeline.
+Hudi can act as either a source or sink, that stores data on DFS. Applicability of Hudi to a given stream processing pipeline ultimately boils down to suitability
+of PrestoDB/SparkSQL/Hive for your queries.
+
+More advanced use cases revolve around the concepts of [incremental processing](https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop), which effectively
+uses Hudi even inside the `processing` engine to speed up typical batch pipelines. For e.g: Hudi can be used as a state store inside a processing DAG (similar
+to how [rocksDB](https://ci.apache.org/projects/flink/flink-docs-release-1.2/ops/state_backends#the-rocksdbstatebackend) is used by Flink). This is an item on the roadmap
+and will eventually happen as a [Beam Runner](https://issues.apache.org/jira/browse/HUDI-60)
diff --git a/website/versioned_docs/version-1.0.0/concepts.md b/website/versioned_docs/version-1.0.0/concepts.md
new file mode 100644
index 00000000000..8d0adf8dd5a
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/concepts.md
@@ -0,0 +1,172 @@
+---
+version: 0.6.0
+title: "Concepts"
+keywords: [ hudi, design, table, queries, timeline]
+summary: "Here we introduce some basic concepts & give a broad technical overview of Hudi"
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+
+Apache Hudi (pronounced “Hudi”) provides the following streaming primitives over hadoop compatible storages
+
+ * Update/Delete Records (how do I change records in a table?)
+ * Change Streams (how do I fetch records that changed?)
+
+In this section, we will discuss key concepts & terminologies that are important to understand, to be able to effectively use these primitives.
+
+## Timeline
+At its core, Hudi maintains a `timeline` of all actions performed on the table at different `instants` of time that helps provide instantaneous views of the table,
+while also efficiently supporting retrieval of data in the order of arrival. A Hudi instant consists of the following components
+
+ * `Instant action` : Type of action performed on the table
+ * `Instant time` : Instant time is typically a timestamp (e.g: 20190117010349), which monotonically increases in the order of action's begin time.
+ * `state` : current state of the instant
+
+Hudi guarantees that the actions performed on the timeline are atomic & timeline consistent based on the instant time.
+
+Key actions performed include
+
+ * `COMMITS` - A commit denotes an **atomic write** of a batch of records into a table.
+ * `CLEANS` - Background activity that gets rid of older versions of files in the table, that are no longer needed.
+ * `DELTA_COMMIT` - A delta commit refers to an **atomic write** of a batch of records into a MergeOnRead type table, where some/all of the data could be just written to delta logs.
+ * `COMPACTION` - Background activity to reconcile differential data structures within Hudi e.g: moving updates from row based log files to columnar formats. Internally, compaction manifests as a special commit on the timeline
+ * `ROLLBACK` - Indicates that a commit/delta commit was unsuccessful & rolled back, removing any partial files produced during such a write
+ * `SAVEPOINT` - Marks certain file groups as "saved", such that cleaner will not delete them. It helps restore the table to a point on the timeline, in case of disaster/data recovery scenarios.
+
+Any given instant can be
+in one of the following states
+
+ * `REQUESTED` - Denotes an action has been scheduled, but has not initiated
+ * `INFLIGHT` - Denotes that the action is currently being performed
+ * `COMPLETED` - Denotes completion of an action on the timeline
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_timeline.png" alt="hudi_timeline.png" />
+</figure>
+
+Example above shows upserts happenings between 10:00 and 10:20 on a Hudi table, roughly every 5 mins, leaving commit metadata on the Hudi timeline, along
+with other background cleaning/compactions. One key observation to make is that the commit time indicates the `arrival time` of the data (10:20AM), while the actual data
+organization reflects the actual time or `event time`, the data was intended for (hourly buckets from 07:00). These are two key concepts when reasoning about tradeoffs between latency and completeness of data.
+
+When there is late arriving data (data intended for 9:00 arriving >1 hr late at 10:20), we can see the upsert producing new data into even older time buckets/folders.
+With the help of the timeline, an incremental query attempting to get all new data that was committed successfully since 10:00 hours, is able to very efficiently consume
+only the changed files without say scanning all the time buckets > 07:00.
+
+## File management
+Hudi organizes a table into a directory structure under a `basepath` on DFS. Table is broken up into partitions, which are folders containing data files for that partition,
+very similar to Hive tables. Each partition is uniquely identified by its `partitionpath`, which is relative to the basepath.
+
+Within each partition, files are organized into `file groups`, uniquely identified by a `file id`. Each file group contains several
+`file slices`, where each slice contains a base file (`*.parquet`) produced at a certain commit/compaction instant time,
+ along with set of log files (`*.log.*`) that contain inserts/updates to the base file since the base file was produced.
+Hudi adopts a MVCC design, where compaction action merges logs and base files to produce new file slices and cleaning action gets rid of
+unused/older file slices to reclaim space on DFS.
+
+## Index
+Hudi provides efficient upserts, by mapping a given hoodie key (record key + partition path) consistently to a file id, via an indexing mechanism.
+This mapping between record key and file group/file id, never changes once the first version of a record has been written to a file. In short, the
+mapped file group contains all versions of a group of records.
+
+## Table Types & Queries
+Hudi table types define how data is indexed & laid out on the DFS and how the above primitives and timeline activities are implemented on top of such organization (i.e how data is written).
+In turn, `query types` define how the underlying data is exposed to the queries (i.e how data is read).
+
+| Table Type | Supported Query types |
+|-------------- |------------------|
+| Copy On Write | Snapshot Queries + Incremental Queries |
+| Merge On Read | Snapshot Queries + Incremental Queries + Read Optimized Queries |
+
+### Table Types
+Hudi supports the following table types.
+
+ - [Copy On Write](#copy-on-write-table) : Stores data using exclusively columnar file formats (e.g parquet). Updates simply version & rewrite the files by performing a synchronous merge during write.
+ - [Merge On Read](#merge-on-read-table) : Stores data using a combination of columnar (e.g parquet) + row based (e.g avro) file formats. Updates are logged to delta files & later compacted to produce new versions of columnar files synchronously or asynchronously.
+
+Following table summarizes the trade-offs between these two table types
+
+| Trade-off | CopyOnWrite | MergeOnRead |
+|-------------- |------------------| ------------------|
+| Data Latency | Higher | Lower |
+| Update cost (I/O) | Higher (rewrite entire parquet) | Lower (append to delta log) |
+| Parquet File Size | Smaller (high update(I/0) cost) | Larger (low update cost) |
+| Write Amplification | Higher | Lower (depending on compaction strategy) |
+
+
+### Query types
+Hudi supports the following query types
+
+ - **Snapshot Queries** : Queries see the latest snapshot of the table as of a given commit or compaction action. In case of merge on read table, it exposes near-real time data(few mins) by merging
+ the base and delta files of the latest file slice on-the-fly. For copy on write table, it provides a drop-in replacement for existing parquet tables, while providing upsert/delete and other write side features.
+ - **Incremental Queries** : Queries only see new data written to the table, since a given commit/compaction. This effectively provides change streams to enable incremental data pipelines.
+ - **Read Optimized Queries** : Queries see the latest snapshot of table as of a given commit/compaction action. Exposes only the base/columnar files in latest file slices and guarantees the
+ same columnar query performance compared to a non-hudi columnar table.
+
+Following table summarizes the trade-offs between the different query types.
+
+| Trade-off | Snapshot | Read Optimized |
+|-------------- |-------------| ------------------|
+| Data Latency | Lower | Higher
+| Query Latency | Higher (merge base / columnar file + row based delta / log files) | Lower (raw base / columnar file performance)
+
+
+## Copy On Write Table
+
+File slices in Copy-On-Write table only contain the base/columnar file and each commit produces new versions of base files.
+In other words, we implicitly compact on every commit, such that only columnar data exists. As a result, the write amplification
+(number of bytes written for 1 byte of incoming data) is much higher, where read amplification is zero.
+This is a much desired property for analytical workloads, which is predominantly read-heavy.
+
+Following illustrates how this works conceptually, when data written into copy-on-write table and two queries running on top of it.
+
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_cow.png" alt="hudi_cow.png" />
+</figure>
+
+
+As data gets written, updates to existing file groups produce a new slice for that file group stamped with the commit instant time,
+while inserts allocate a new file group and write its first slice for that file group. These file slices and their commit instant times are color coded above.
+SQL queries running against such a table (eg: `select count(*)` counting the total records in that partition), first checks the timeline for the latest commit
+and filters all but latest file slices of each file group. As you can see, an old query does not see the current inflight commit's files color coded in pink,
+but a new query starting after the commit picks up the new data. Thus queries are immune to any write failures/partial writes and only run on committed data.
+
+The intention of copy on write table, is to fundamentally improve how tables are managed today through
+
+ - First class support for atomically updating data at file-level, instead of rewriting whole tables/partitions
+ - Ability to incremental consume changes, as opposed to wasteful scans or fumbling with heuristics
+ - Tight control of file sizes to keep query performance excellent (small files hurt query performance considerably).
+
+
+## Merge On Read Table
+
+Merge on read table is a superset of copy on write, in the sense it still supports read optimized queries of the table by exposing only the base/columnar files in latest file slices.
+Additionally, it stores incoming upserts for each file group, onto a row based delta log, to support snapshot queries by applying the delta log,
+onto the latest version of each file id on-the-fly during query time. Thus, this table type attempts to balance read and write amplification intelligently, to provide near real-time data.
+The most significant change here, would be to the compactor, which now carefully chooses which delta log files need to be compacted onto
+their columnar base file, to keep the query performance in check (larger delta log files would incur longer merge times with merge data on query side)
+
+Following illustrates how the table works, and shows two types of queries - snapshot query and read optimized query.
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_mor.png" alt="hudi_mor.png" />
+</figure>
+
+There are lot of interesting things happening in this example, which bring out the subtleties in the approach.
+
+ - We now have commits every 1 minute or so, something we could not do in the other table type.
+ - Within each file id group, now there is an delta log file, which holds incoming updates to the records already present in the base columnar files. In the example, the delta log files hold
+ all the data from 10:05 to 10:10. The base columnar files are still versioned with the commit, as before.
+ Thus, if one were to simply look at base files alone, then the table layout looks exactly like a copy on write table.
+ - A periodic compaction process reconciles these changes from the delta log and produces a new version of base file, just like what happened at 10:05 in the example.
+ - There are two ways of querying the same underlying table: Read Optimized query and Snapshot query, depending on whether we chose query performance or freshness of data.
+ - The semantics around when data from a commit is available to a query changes in a subtle way for a read optimized query. Note, that such a query
+ running at 10:10, wont see data after 10:05 above, while a snapshot query always sees the freshest data.
+ - When we trigger compaction & what it decides to compact hold all the key to solving these hard problems. By implementing a compacting
+ strategy, where we aggressively compact the latest partitions compared to older partitions, we could ensure the read optimized queries see data
+ published within X minutes in a consistent fashion.
+
+The intention of merge on read table is to enable near real-time processing directly on top of DFS, as opposed to copying
+data out to specialized systems, which may not be able to handle the data volume. There are also a few secondary side benefits to
+this table such as reduced write amplification by avoiding synchronous merge of data, i.e, the amount of data written per 1 bytes of data in a batch
+
+
diff --git a/website/versioned_docs/version-1.0.0/concurrency_control.md b/website/versioned_docs/version-1.0.0/concurrency_control.md
new file mode 100644
index 00000000000..549f1ddd17e
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/concurrency_control.md
@@ -0,0 +1,339 @@
+---
+title: "Concurrency Control"
+summary: In this page, we will discuss how to perform concurrent writes to Hudi Tables.
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+last_modified_at: 2021-03-19T15:59:57-04:00
+---
+Concurrency control defines how different writers/readers/table services coordinate access to a Hudi table. Hudi ensures atomic writes, by way of publishing commits atomically to the timeline,
+stamped with an instant time that denotes the time at which the action is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between
+writer processes that issue [write operations](write_operations) and table services that (re)write data/metadata to optimize/perform bookkeeping and
+readers (that execute queries and read data).
+
+Hudi provides
+* **Snapshot isolation** between all three types of processes, meaning they all operate on a consistent snapshot of the table.
+* **Optimistic concurrency control (OCC)** between writers to provide standard relational database semantics.
+* **Multiversion Concurrency Control (MVCC)** based concurrency control between writers and table-services and between different table services.
+* **Non-blocking Concurrency Control (NBCC)** between writers, to provide streaming semantics and avoiding live-locks/starvation between writers.
+
+In this section, we will discuss the different concurrency controls supported by Hudi and how they are leveraged to provide flexible deployment models for single and multiple writer scenarios.
+We’ll also describe ways to ingest data into a Hudi Table from multiple writers using different writers, like Hudi Streamer, Hudi datasource, Spark Structured Streaming and Spark SQL.
+
+:::note
+If there is only one process performing writing AND async/inline table services on the table, you can
+avoid the overhead of a distributed lock requirement by configuring the in process lock provider.
+
+```properties
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider
+```
+:::
+
+## Distributed Locking
+A pre-requisite for distributed co-ordination in Hudi, like many other distributed database systems is a distributed lock provider, that different processes can use to plan, schedule and
+execute actions on the Hudi timeline in a concurrent fashion. Locks are also used to [generate TrueTime](timeline#truetime-generation), as discussed before.
+
+External locking is typically used in conjunction with optimistic concurrency control
+because it provides a way to prevent conflicts that might occur when two or more transactions (commits in our case) attempt to modify the same resource concurrently.
+When a transaction attempts to modify a resource that is currently locked by another transaction, it must wait until the lock is released before proceeding.
+
+In case of multi-writing in Hudi, the locks are acquired on the Hudi table for a very short duration during specific phases (such as just before committing the writes or before scheduling table services) instead of locking for the entire span of time. This approach allows multiple writers to work on the same table simultaneously, increasing concurrency and avoids conflicts.
+
+There are 4 different lock providers that require different configurations to be set. Please refer to comprehensive locking configs [here](https://hudi.apache.org/docs/next/configurations#LOCK).
+
+#### Zookeeper based
+```
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
+```
+Following are the basic configs required to setup this lock provider:
+
+| Config Name| Default| Description |
+| ---------------------------------------------------------------------------- | ------------------------ |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.write.lock.zookeeper.base_path | N/A **(Required)** | The base path on Zookeeper under which to create lock related ZNodes. This should be same for all concurrent writers to the same table<br /><br />`Config Param: ZK_BASE_PATH`<br />`Since Version: 0.8.0` |
+| hoodie.write.lock.zookeeper.port | N/A **(Required)** | Zookeeper port to connect to.<br /><br />`Config Param: ZK_PORT`<br />`Since Version: 0.8.0` |
+| hoodie.write.lock.zookeeper.url | N/A **(Required)** | Zookeeper URL to connect to.<br /><br />`Config Param: ZK_CONNECT_URL`<br />`Since Version: 0.8.0` |
+
+#### HiveMetastore based
+
+```
+hoodie.write.lock.provider=org.apache.hudi.hive.transaction.lock.HiveMetastoreBasedLockProvider
+```
+Following are the basic configs required to setup this lock provider:
+
+| Config Name| Default| Description |
+| ----------------------------------------------------------------------- | ------------------------ |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.write.lock.hivemetastore.database | N/A **(Required)** | For Hive based lock provider, the Hive database to acquire lock against<br /><br />`Config Param: HIVE_DATABASE_NAME`<br />`Since Version: 0.8.0` |
+| hoodie.write.lock.hivemetastore.table | N/A **(Required)** | For Hive based lock provider, the Hive table to acquire lock against<br /><br />`Config Param: HIVE_TABLE_NAME`<br />`Since Version: 0.8.0` |
+
+`The HiveMetastore URI's are picked up from the hadoop configuration file loaded during runtime.`
+
+#### Amazon DynamoDB based
+```
+hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
+```
+Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters. You can refer to the
+[DynamoDB based Locks Configurations](https://hudi.apache.org/docs/configurations#DynamoDB-based-Locks-Configurations)
+section for the details of each related configuration knob. Following are the basic configs required to setup this lock provider:
+
+| Config Name| Default| Description |
+| ----------------------------------------------------------------------- | ------------------------ |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.write.lock.dynamodb.endpoint_url| N/A **(Required)** | For DynamoDB based lock provider, the url endpoint used for Amazon DynamoDB service. Useful for development with a local dynamodb instance.<br /><br />`Config Param: DYNAMODB_ENDPOINT_URL`<br />`Since Version: 0.10.1`|
+
+For advanced configs refer [here](https://hudi.apache.org/docs/next/configurations#DynamoDB-based-Locks-Configurations)
+
+
+When using the DynamoDB-based lock provider, the name of the DynamoDB table acting as the lock table for Hudi is
+specified by the config `hoodie.write.lock.dynamodb.table`. This DynamoDB table is automatically created by Hudi, so you
+don't have to create the table yourself. If you want to use an existing DynamoDB table, make sure that an attribute with
+the name `key` is present in the table. The `key` attribute should be the partition key of the DynamoDB table. The
+config `hoodie.write.lock.dynamodb.partition_key` specifies the value to put for the `key` attribute (not the attribute
+name), which is used for the lock on the same table. By default, `hoodie.write.lock.dynamodb.partition_key` is set to
+the table name, so that multiple writers writing to the same table share the same lock. If you customize the name, make
+sure it's the same across multiple writers.
+
+Also, to set up the credentials for accessing AWS resources, customers can pass the following props to Hudi jobs:
+```
+hoodie.aws.access.key
+hoodie.aws.secret.key
+hoodie.aws.session.token
+```
+If not configured, Hudi falls back to use [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html).
+
+IAM policy for your service instance will need to add the following permissions:
+
+```json
+{
+ "Sid":"DynamoDBLocksTable",
+ "Effect": "Allow",
+ "Action": [
+ "dynamodb:CreateTable",
+ "dynamodb:DeleteItem",
+ "dynamodb:DescribeTable",
+ "dynamodb:GetItem",
+ "dynamodb:PutItem",
+ "dynamodb:Scan",
+ "dynamodb:UpdateItem"
+ ],
+ "Resource": "arn:${Partition}:dynamodb:${Region}:${Account}:table/${TableName}"
+}
+```
+- `TableName` : same as `hoodie.write.lock.dynamodb.partition_key`
+- `Region`: same as `hoodie.write.lock.dynamodb.region`
+
+AWS SDK dependencies are not bundled with Hudi from v0.10.x and will need to be added to your classpath.
+Add the following Maven packages (check the latest versions at time of install):
+```
+com.amazonaws:dynamodb-lock-client
+com.amazonaws:aws-java-sdk-dynamodb
+com.amazonaws:aws-java-sdk-core
+```
+
+#### FileSystem based (not for production use)
+
+FileSystem based lock provider supports multiple writers cross different jobs/applications based on atomic create/delete operations of the underlying filesystem.
+
+```
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.FileSystemBasedLockProvider
+```
+
+When using the FileSystem based lock provider, by default, the lock file will store into `hoodie.base.path`+`/.hoodie/lock`. You may use a custom folder to store the lock file by specifying `hoodie.write.lock.filesystem.path`.
+
+In case the lock cannot release during job crash, you can set `hoodie.write.lock.filesystem.expire` (lock will never expire by default) to a desired expire time in minutes. You may also delete lock file manually in such situation.
+:::note
+FileSystem based lock provider is not supported with cloud storage like S3 or GCS.
+:::
+
+
+## Simple Single writer + table services
+
+Data lakehouse pipelines tend to be predominantly single writer, with the most common need for distributed co-ordination on a table coming from table management. For e.g. a Apache Flink
+job producing fast writes into a table, requiring regular file-size management or cleaning. Hudi's storage engine and platform tools provide a lot of support for such common scenarios.
+
+### Inline table services
+
+This is the simplest form of concurrency, meaning there is no concurrency at all in the write processes. In this model, Hudi eliminates the need for concurrency control and maximizes throughput by supporting these table services out-of-box and running inline after every write to the table. Execution plans are idempotent, persisted to the timeline and auto-recover from failures. For most simple use-cases, this means just writing is sufficient to get a well-managed table that needs no conc [...]
+
+There is no actual concurrent writing in this model. **MVCC** is leveraged to provide snapshot isolation guarantees between ingestion writer and multiple readers and also between multiple table service writers and readers. Writes to the table either from ingestion or from table services produce versioned data that are available to readers only after the writes are committed. Until then, readers can access only the previous version of the data.
+
+A single writer with all table services such as cleaning, clustering, compaction, etc can be configured to be inline (such as Hudi Streamer sync-once mode and Spark Datasource with default configs) without any additional configs.
+
+### Async table services
+
+Hudi provides the option of running the table services in an async fashion, where most of the heavy lifting (e.g actually rewriting the columnar data by compaction service) is done asynchronously. In this model, the async deployment eliminates any repeated wasteful retries and optimizes the table using clustering techniques while a single writer consumes the writes to the table without having to be blocked by such table services. This model avoids the need for taking an [external lock](# [...]
+
+A single writer along with async table services runs in the same process. For example, you can have a Hudi Streamer in continuous mode write to a MOR table using async compaction; you can use Spark Streaming (where [compaction](https://hudi.apache.org/docs/compaction) is async by default), and you can use Flink streaming or your own job setup and enable async table services inside the same writer.
+
+Hudi leverages **MVCC** in this model to support running any number of table service jobs concurrently, without any concurrency conflict. This is made possible by ensuring Hudi 's ingestion writer and async table services coordinate among themselves to ensure no conflicts and avoid race conditions. The same single write guarantees described in Model A above can be achieved in this model as well.
+With this model users don't need to spin up different spark jobs and manage the orchestration among it. For larger deployments, this model can ease the operational burden significantly while getting the table services running without blocking the writers.
+
+**Single Writer Guarantees**
+
+In this model, the following are the guarantees on [write operations](https://hudi.apache.org/docs/write_operations/) to expect:
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table wilL NEVER have duplicates if dedup: [`hoodie.datasource.write.insert.drop.duplicates`](https://hudi.apache.org/docs/configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](https://hudi.apache.org/docs/configurations/#hoodiecombinebeforeinsert), is enabled.
+- *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if dedup: [`hoodie.datasource.write.insert.drop.duplicates`](https://hudi.apache.org/docs/configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](https://hudi.apache.org/docs/configurations/#hoodiecombinebeforeinsert), is enabled.
+- *INCREMENTAL QUERY Guarantee*: Data consumption and checkpoints are NEVER out of order.
+
+## Full-on Multi-writer + Async table services
+
+Hudi has introduced a new concurrency mode `NON_BLOCKING_CONCURRENCY_CONTROL`, where unlike OCC, multiple writers can
+operate on the table with non-blocking conflict resolution. The writers can write into the same file group with the
+conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is currently
+available for preview in version 1.0.0-beta only. You can read more about it under section [Model C: Multi-writer](#model-c-multi-writer).
+
+It is not always possible to serialize all write operations to a table (such as UPSERT, INSERT or DELETE) into the same write process and therefore, multi-writing capability may be required.
+In multi-writing, disparate distributed processes run in parallel or overlapping time windows to write to the same table. In such cases, an external locking mechanism is a must to safely
+coordinate concurrent accesses. Here are few different scenarios that would all fall under multi-writing:
+
+- Multiple ingestion writers to the same table:For instance, two Spark Datasource writers working on different sets of partitions form a source kafka topic.
+- Multiple ingestion writers to the same table, including one writer with async table services: For example, a Hudi Streamer with async compaction for regular ingestion & a Spark Datasource writer for backfilling.
+- A single ingestion writer and a separate compaction (HoodieCompactor) or clustering (HoodieClusteringJob) job apart from the ingestion writer: This is considered as multi-writing as they are not running in the same process.
+
+Hudi's concurrency model intelligently differentiates actual writing to the table from table services that manage or optimize the table. Hudi offers similar **optimistic concurrency control across multiple writers**, but **table services can still execute completely lock-free and async** as long as they run in the same process as one of the writers.
+For multi-writing, Hudi leverages file level optimistic concurrency control(OCC). For example, when two writers write to non overlapping files, both writes are allowed to succeed. However, when the writes from different writers overlap (touch the same set of files), only one of them will succeed. Please note that this feature is currently experimental and requires external lock providers to acquire locks briefly at critical sections during the write. More on lock providers below.
+
+#### Multi Writer Guarantees
+
+With multiple writers using OCC, these are the write guarantees to expect:
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if dedup is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if dedup is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order. If there are inflight commits
+ (due to multi-writing), incremental queries will not expose the completed commits following the inflight commits.
+
+## Non-Blocking Concurrency Control
+
+`NON_BLOCKING_CONCURRENCY_CONTROL`, offers the same set of guarantees as mentioned in the case of OCC but without
+explicit locks for serializing the writes. Lock is only needed for writing the commit metadata to the Hudi timeline. The
+completion time for the commits reflects the serialization order and file slicing is done based on completion time.
+Multiple writers can operate on the table with non-blocking conflict resolution. The writers can write into the same
+file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
+currently available for preview in version 1.0.0-beta only with the caveat that conflict resolution is not supported yet
+between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
+writers [here](sql_dml#non-blocking-concurrency-control-experimental).
+
+:::note
+`NON_BLOCKING_CONCURRENCY_CONTROL` between ingestion writer and table service writer is not yet supported for clustering.
+Please use `OPTIMISTIC_CONCURRENCY_CONTROL` for clustering.
+:::
+
+## Early conflict Detection
+
+Multi writing using OCC allows multiple writers to concurrently write and atomically commit to the Hudi table if there is no overlapping data file to be written, to guarantee data consistency, integrity and correctness. Prior to 0.13.0 release, as the OCC (optimistic concurrency control) name suggests, each writer will optimistically proceed with ingestion and towards the end, just before committing will go about conflict resolution flow to deduce overlapping writes and abort one if need [...]
+
+To improve the concurrency control, the [0.13.0 release](https://hudi.apache.org/releases/release-0.13.0#early-conflict-detection-for-multi-writer) introduced a new feature, early conflict detection in OCC, to detect the conflict during the data writing phase and abort the writing early on once a conflict is detected, using Hudi's marker mechanism. Hudi can now stop a conflicting writer much earlier because of the early conflict detection and release computing resources necessary to clus [...]
+
+By default, this feature is turned off. To try this out, a user needs to set `hoodie.write.concurrency.early.conflict.detection.enable` to true, when using OCC for concurrency control (Refer [configs](https://hudi.apache.org/docs/next/configurations#Write-Configurations-advanced-configs) page for all relevant configs).
+:::note
+Early conflict Detection in OCC is an **EXPERIMENTAL** feature
+:::
+
+
+## Enabling Multi Writing
+
+The following properties are needed to be set appropriately to turn on optimistic concurrency control to achieve multi writing.
+
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.write.lock.provider=<lock-provider-classname>
+hoodie.cleaner.policy.failed.writes=LAZY
+```
+
+| Config Name | Default | Description [...]
+|-------------------------------------|-------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.write.concurrency.mode | SINGLE_WRITER (Optional) | <u>[Concurrency modes](https://github.com/apache/hudi/blob/00ece7bce0a4a8d0019721a28049723821e01842/hudi-common/src/main/java/org/apache/hudi/common/model/WriteConcurrencyMode.java)</u> for write operations.<br />Possible values:<br /><ul><li>`SINGLE_WRITER`: Only one active writer to the table. Maximizes throughput.</li><li>`OPTIMISTIC_CONCURRENCY_CONTROL`: Multiple wr [...]
+| hoodie.write.lock.provider | org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider (Optional) | Lock provider class name, user can provide their own implementation of LockProvider which should be subclass of org.apache.hudi.common.lock.LockProvider<br /><br />`Config Param: LOCK_PROVIDER_CLASS_NAME`<br />`Since Version: 0.8.0` [...]
+| hoodie.cleaner.policy.failed.writes | EAGER (Optional) | org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy: Policy that controls how to clean up failed writes. Hudi will delete any files written by failed writes to re-claim space. EAGER(default): Clean failed writes inline after every write operation. LAZY: Clean failed writes lazily after heartbeat timeout when the cleaning service runs. This policy is re [...]
+
+
+### Multi Writing via Hudi Streamer
+
+The `HoodieStreamer` utility (part of hudi-utilities-slim-bundle) provides ways to ingest from different sources such as DFS or Kafka, with the following capabilities.
+
+Using optimistic_concurrency_control via Hudi Streamer requires adding the above configs to the properties file that can be passed to the
+job. For example below, adding the configs to kafka-source.properties file and passing them to Hudi Streamer will enable optimistic concurrency.
+A Hudi Streamer job can then be triggered as follows:
+
+```java
+[hoodie]$ spark-submit \
+ --jars "packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle_2.12-1.0.0.jar,packaging/hudi-spark-bundle/target/hudi-spark3.5-bundle_2.12-1.0.0.jar" \
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer `ls packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle-*.jar` \
+ --props file://${PWD}/hudi-utilities/src/test/resources/streamer-config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --target-base-path file:\/\/\/tmp/hudi-streamer-op \
+ --target-table tableName \
+ --op BULK_INSERT
+```
+
+### Multi Writing via Spark Datasource Writer
+
+The `hudi-spark` module offers the DataSource API to write (and read) a Spark DataFrame into a Hudi table.
+
+Following is an example of how to use optimistic_concurrency_control via spark datasource
+
+```java
+inputDF.write.format("hudi")
+ .options(getQuickstartWriteConfigs)
+ .option("hoodie.datasource.write.precombine.field", "ts")
+ .option("hoodie.cleaner.policy.failed.writes", "LAZY")
+ .option("hoodie.write.concurrency.mode", "optimistic_concurrency_control")
+ .option("hoodie.write.lock.zookeeper.url", "zookeeper")
+ .option("hoodie.write.lock.zookeeper.port", "2181")
+ .option("hoodie.write.lock.zookeeper.base_path", "/test")
+ .option("hoodie.datasource.write.recordkey.field", "uuid")
+ .option("hoodie.datasource.write.partitionpath.field", "partitionpath")
+ .option("hoodie.table.name", tableName)
+ .mode(Overwrite)
+ .save(basePath)
+```
+
+## Disabling Multi Writing
+
+Remove the following settings that were used to enable multi-writer or override with default values.
+
+```
+hoodie.write.concurrency.mode=single_writer
+hoodie.cleaner.policy.failed.writes=EAGER
+```
+
+## OCC Best Practices
+
+Concurrent Writing to Hudi tables requires acquiring a lock with one of the lock providers mentioned above. Due to several reasons you might want to configure retries to allow your application to acquire the lock.
+1. Network connectivity or excessive load on servers increasing time for lock acquisition resulting in timeouts
+2. Running a large number of concurrent jobs that are writing to the same hudi table can result in contention during lock acquisition can cause timeouts
+3. In some scenarios of conflict resolution, Hudi commit operations might take upto 10's of seconds while the lock is being held. This can result in timeouts for other jobs waiting to acquire a lock.
+
+Set the correct native lock provider client retries.
+:::note
+Please note that sometimes these settings are set on the server once and all clients inherit the same configs. Please check your settings before enabling optimistic concurrency.
+:::
+
+```
+hoodie.write.lock.wait_time_ms
+hoodie.write.lock.num_retries
+```
+
+Set the correct hudi client retries for Zookeeper & HiveMetastore. This is useful in cases when native client retry settings cannot be changed. Please note that these retries will happen in addition to any native client retries that you may have set.
+
+```
+hoodie.write.lock.client.wait_time_ms
+hoodie.write.lock.client.num_retries
+```
+
+*Setting the right values for these depends on a case by case basis; some defaults have been provided for general cases.*
+
+
+## Caveats
+
+If you are using the `WriteClient` API, please note that multiple writes to the table need to be initiated from 2 different instances of the write client.
+It is **NOT** recommended to use the same instance of the write client to perform multi writing.
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Hands on Lab with using DynamoDB as lock table for Apache Hudi Data Lakes](https://youtu.be/JP0orl9_0yQ)
+* [Non Blocking Concurrency Control Flink Demo](/blog/2024/12/06/non-blocking-concurrency-control)
\ No newline at end of file
diff --git a/website/versioned_docs/version-1.0.0/configurations.md b/website/versioned_docs/version-1.0.0/configurations.md
new file mode 100644
index 00000000000..0758147e0df
--- /dev/null
+++ b/website/versioned_docs/version-1.0.0/configurations.md
@@ -0,0 +1,2287 @@
+---
+title: All Configurations
+keywords: [ configurations, default, flink options, spark, configs, parameters ]
+permalink: /docs/configurations.html
+summary: This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+last_modified_at: 2024-12-06T17:38:05.818
+---
+
+
+This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
+
+- [**Hudi Table Config**](#TABLE_CONFIG): Basic Hudi Table configuration parameters.
+- [**Environment Config**](#ENVIRONMENT_CONFIG): Hudi supports passing configurations via a configuration file `hudi-defaults.conf` in which each line consists of a key and a value separated by whitespace or = sign. For example:
+```
+hoodie.datasource.hive_sync.mode jdbc
+hoodie.datasource.hive_sync.jdbcurl jdbc:hive2://localhost:10000
+hoodie.datasource.hive_sync.support_timestamp false
+```
+It helps to have a central configuration file for your common cross job configurations/tunings, so all the jobs on your cluster can utilize it. It also works with Spark SQL DML/DDL, and helps avoid having to pass configs inside the SQL statements.
+
+Hudi always loads the configuration file under default directory `file:/etc/hudi/conf`, if exists, to set the default configs. Besides, you can specify another configuration directory location by setting the `HUDI_CONF_DIR` environment variable. The configs stored in `HUDI_CONF_DIR/hudi-defaults.conf` are loaded, overriding any configs already set by the config file in the default directory.
+- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
+- [**Flink Sql Configs**](#FLINK_SQL): These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
+- [**Write Client Configs**](#WRITE_CLIENT): Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.
+- [**Reader Configs**](#READER): Please fill in the description for Config Group Name: Reader Configs
+- [**Metastore and Catalog Sync Configs**](#META_SYNC): Configurations used by the Hudi to sync metadata to external metastores and catalogs.
+- [**Metrics Configs**](#METRICS): These set of configs are used to enable monitoring and reporting of key Hudi stats and metrics.
+- [**Record Payload Config**](#RECORD_PAYLOAD): This is the lowest level of customization offered by Hudi. Record payloads define how to produce new values to upsert based on incoming new record and stored old record. Hudi provides default implementations such as OverwriteWithLatestAvroPayload which simply update table with the latest/last-written record. This can be overridden to a custom class extending HoodieRecordPayload class, on both datasource and WriteClient levels.
+- [**Kafka Connect Configs**](#KAFKA_CONNECT): These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables
+- [**Amazon Web Services Configs**](#AWS): Configurations specific to Amazon Web Services.
+- [**Hudi Streamer Configs**](#HUDI_STREAMER): These set of configs are used for Hudi Streamer utility which provides the way to ingest from different sources such as DFS or Kafka.
+
+:::note
+In the tables below **(N/A)** means there is no default value set
+:::
+
+## Externalized Config File
+Instead of directly passing configuration settings to every Hudi job, you can also centrally set them in a configuration
+file `hudi-defaults.conf`. By default, Hudi would load the configuration file under `/etc/hudi/conf` directory. You can
+specify a different configuration directory location by setting the `HUDI_CONF_DIR` environment variable. This can be
+useful for uniformly enforcing repeated configs (like Hive sync or write/index tuning), across your entire data lake.
+
+## Hudi Table Config {#TABLE_CONFIG}
+Basic Hudi Table configuration parameters.
+
+
+### Hudi Table Basic Configs {#Hudi-Table-Basic-Configs}
+Configurations of the Hudi Table like type of ingestion, storage formats, hive table name etc. Configurations are loaded from hoodie.properties, these properties are usually set during initializing a path as hoodie base path and never changes during the lifetime of a hoodie table.
+
+
+
+[**Basic Configs**](#Hudi-Table-Basic-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BOOTSTRAP_BASE_PATH` [...]
+| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | (N/A) | Payload class to use for performing merges, compactions, i.e merge delta logs with current base file and then produce a new base file.<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name. If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME` [...]
+| [hoodie.record.merge.strategy.id](#hoodierecordmergestrategyid) | (N/A) | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in `hoodie.write.record.merge.custom.implementation.classes` which has the same merger strategy id<br />`Config Param: RECORD_MERGE_STRATEGY_ID`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.<br />`Config Param: TABLE_CHECKSUM`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.create.schema](#hoodietablecreateschema) | (N/A) | Schema used when creating the table<br />`Config Param: CREATE_SCHEMA` [...]
+| [hoodie.table.index.defs.path](#hoodietableindexdefspath) | (N/A) | Relative path to table base path where the index definitions are stored<br />`Config Param: RELATIVE_INDEX_DEFINITION_PATH`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.keygenerator.class](#hoodietablekeygeneratorclass) | (N/A) | Key Generator class property for the hoodie table<br />`Config Param: KEY_GENERATOR_CLASS_NAME` [...]
+| [hoodie.table.keygenerator.type](#hoodietablekeygeneratortype) | (N/A) | Key Generator type to determine key generator class<br />`Config Param: KEY_GENERATOR_TYPE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.metadata.partitions](#hoodietablemetadatapartitions) | (N/A) | Comma-separated list of metadata partitions that have been completely built and in-sync with data table. These partitions are ready for use by the readers<br />`Config Param: TABLE_METADATA_PARTITIONS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.metadata.partitions.inflight](#hoodietablemetadatapartitionsinflight) | (N/A) | Comma-separated list of metadata partitions whose building is in progress. These partitions are not yet ready for use by the readers.<br />`Config Param: TABLE_METADATA_PARTITIONS_INFLIGHT`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: NAME` [...]
+| [hoodie.table.partition.fields](#hoodietablepartitionfields) | (N/A) | Comma separated field names used to partition the table. These field names also include the partition type which is used by custom key generators<br />`Config Param: PARTITION_FIELDS` [...]
+| [hoodie.table.precombine.field](#hoodietableprecombinefield) | (N/A) | Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.table.recordkey.fields](#hoodietablerecordkeyfields) | (N/A) | Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br />`Config Param: RECORDKEY_FIELDS` [...]
+| [hoodie.table.secondary.indexes.metadata](#hoodietablesecondaryindexesmetadata) | (N/A) | The metadata of secondary indexes<br />`Config Param: SECONDARY_INDEXES_METADATA`<br />`Since Version: 0.13.0` [...]
+| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | (N/A) | Version of timeline used, by the table.<br />`Config Param: TIMELINE_LAYOUT_VERSION` [...]
+| [hoodie.archivelog.folder](#hoodiearchivelogfolder) | archived | path under the meta folder, to store archived timeline instants at.<br />`Config Param: ARCHIVELOG_FOLDER` [...]
+| [hoodie.bootstrap.index.class](#hoodiebootstrapindexclass) | org.apache.hudi.common.bootstrap.index.hfile.HFileBootstrapIndex | Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_CLASS_NAME` [...]
+| [hoodie.bootstrap.index.enable](#hoodiebootstrapindexenable) | true | Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined, default true.<br />`Config Param: BOOTSTRAP_INDEX_ENABLE` [...]
+| [hoodie.bootstrap.index.type](#hoodiebootstrapindextype) | HFILE | Bootstrap index type determines which implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_TYPE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` [...]
+| [hoodie.partition.metafile.use.base.format](#hoodiepartitionmetafileusebaseformat) | false | If true, partition metafiles are saved in the same format as base-files for this dataset (e.g. Parquet / ORC). If false (default) partition metafiles are saved as properties files.<br />`Config Param: PARTITION_METAFILE_USE_BASE_FORMAT` [...]
+| [hoodie.populate.meta.fields](#hoodiepopulatemetafields) | true | When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br />`Config Param: POPULATE_META_FIELDS` [...]
+| [hoodie.record.merge.mode](#hoodierecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge reco [...]
+| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | Base file format to store all the base file data.<br />`Config Param: BASE_FILE_FORMAT` [...]
+| [hoodie.table.cdc.enabled](#hoodietablecdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode.<br />`Config Param: CDC_ENABLED`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.cdc.supplemental.logging.mode](#hoodietablecdcsupplementalloggingmode) | DATA_BEFORE_AFTER | org.apache.hudi.common.table.cdc.HoodieCDCSupplementalLoggingMode: Change log capture supplemental logging mode. The supplemental log is used for accelerating the generation of change log details. OP_KEY_ONLY: Only keeping record keys in the supplemental logs, so the reader needs to figure out the update before image [...]
+| [hoodie.table.initial.version](#hoodietableinitialversion) | EIGHT | Initial Version of table when the table was created. Used for upgrade/downgrade to identify what upgrade/downgrade paths happened on the table. This is only configured when the table is initially setup.<br />`Config Param: INITIAL_VERSION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.log.file.format](#hoodietablelogfileformat) | HOODIE_LOG | Log format used for the delta logs.<br />`Config Param: LOG_FILE_FORMAT` [...]
+| [hoodie.table.multiple.base.file.formats.enable](#hoodietablemultiplebasefileformatsenable) | false | When set to true, the table can support reading and writing multiple base file formats.<br />`Config Param: MULTIPLE_BASE_FILE_FORMATS_ENABLE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.timeline.timezone](#hoodietabletimelinetimezone) | LOCAL | User can set hoodie commit timeline timezone, such as utc, local and so on. local is default<br />`Config Param: TIMELINE_TIMEZONE` [...]
+| [hoodie.table.type](#hoodietabletype) | COPY_ON_WRITE | The table type for the underlying data.<br />`Config Param: TYPE` [...]
+| [hoodie.table.version](#hoodietableversion) | EIGHT | Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br />`Config Param: VERSION` [...]
+| [hoodie.timeline.history.path](#hoodietimelinehistorypath) | history | path under the meta folder, to store timeline history at.<br />`Config Param: TIMELINE_HISTORY_PATH` [...]
+| [hoodie.timeline.path](#hoodietimelinepath) | timeline | path under the meta folder, to store timeline instants at.<br />`Config Param: TIMELINE_PATH` [...]
+
+[**Advanced Configs**](#Hudi-Table-Basic-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.write.drop.partition.columns](#hoodiedatasourcewritedroppartitioncolumns) | false | When set to true, will not write the partition columns into hudi. By default, false.<br />`Config Param: DROP_PARTITION_COLUMNS` |
+| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Should we url encode the partition path value, before creating the folder structure.<br />`Config Param: URL_ENCODE_PARTITIONING` |
+---
+
+## Spark Datasource Configs {#SPARK_DATASOURCE}
+These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
+
+
+### Read Options {#Read-Options}
+Options useful for reading tables via `read.format.option(...)`
+
+
+
+
+[**Basic Configs**](#Read-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| -------------------------------------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.read.begin.instanttime](#hoodiedatasourcereadbegininstanttime) | (N/A) | Required when `hoodie.datasource.query.type` is set to `incremental`. Represents the completion time to start incrementally pulling data from. The completion time here need not necessarily correspond to an instant on the timeline. New data written with completion_time >= START_COMMIT are fetched out. For e.g: ‘20170901080000’ will get all new data written on or after Sep 1, 201 [...]
+| [hoodie.datasource.read.end.instanttime](#hoodiedatasourcereadendinstanttime) | (N/A) | Used when `hoodie.datasource.query.type` is set to `incremental`. Represents the completion time to limit incrementally fetched data to. When not specified latest commit completion time from timeline is assumed by default. When specified, new data written with completion_time <= END_COMMIT are fetched out. Point in time type queries make more sense with begin and end completion [...]
+| [hoodie.datasource.read.incr.table.version](#hoodiedatasourcereadincrtableversion) | (N/A) | The table version assumed for incremental read<br />`Config Param: INCREMENTAL_READ_TABLE_VERSION` [...]
+| [hoodie.datasource.read.streaming.table.version](#hoodiedatasourcereadstreamingtableversion) | (N/A) | The table version assumed for streaming read<br />`Config Param: STREAMING_READ_TABLE_VERSION` [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files)<br />`Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: READ_PRE_COMBINE_FIELD` [...]
+
+[**Advanced Configs**](#Read-Options-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [as.of.instant](#asofinstant) | (N/A) | The query instant for time travel. Without specified this option, we query the latest snapshot.<br />`Config Param: TIME_TRAVEL_AS_OF_INSTANT` [...]
+| [hoodie.datasource.read.paths](#hoodiedatasourcereadpaths) | (N/A) | Comma separated list of file paths to read within a Hudi table.<br />`Config Param: READ_PATHS` [...]
+| [hoodie.datasource.merge.type](#hoodiedatasourcemergetype) | payload_combine | For Snapshot query on merge on read table. Use this key to define how the payloads are merged, in 1) skip_merge: read the base file records plus the log file records without merging; 2) payload_combine: read the base file records first, for each record in base file, checks whether the key is in the log file [...]
+| [hoodie.datasource.query.incremental.format](#hoodiedatasourcequeryincrementalformat) | latest_state | This config is used alone with the 'incremental' query type.When set to 'latest_state', it returns the latest records' values.When set to 'cdc', it returns the cdc data.<br />`Config Param: INCREMENTAL_FORMAT`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.read.create.filesystem.relation](#hoodiedatasourcereadcreatefilesystemrelation) | false | When this is set, the relation created by DefaultSource is for a view representing the result set of the table valued function hudi_filesystem_view(...)<br />`Config Param: CREATE_FILESYSTEM_RELATION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.read.extract.partition.values.from.path](#hoodiedatasourcereadextractpartitionvaluesfrompath) | false | When set to true, values for partition columns (partition values) will be extracted from physical partition path (default Spark behavior). When set to false partition values will be read from the data file (in Hudi partition columns are persisted by default). This config is a fallback allowing to preserve exis [...]
+| [hoodie.datasource.read.file.index.listing.mode](#hoodiedatasourcereadfileindexlistingmode) | lazy | Overrides Hudi's file-index implementation's file listing mode: when set to 'eager', file-index will list all partition paths and corresponding file slices w/in them eagerly, during initialization, prior to partition-pruning kicking in, meaning that all partitions will be listed including ones that might be s [...]
+| [hoodie.datasource.read.file.index.listing.partition-path-prefix.analysis.enabled](#hoodiedatasourcereadfileindexlistingpartition-path-prefixanalysisenabled) | true | Controls whether partition-path prefix analysis is enabled w/in the file-index, allowing to avoid necessity to recursively list deep folder structures of partitioned tables w/ multiple partition columns, by carefully analyzing provided partition-column predicates and deducing corresponding partition-path prefi [...]
+| [hoodie.datasource.read.incr.fallback.fulltablescan.enable](#hoodiedatasourcereadincrfallbackfulltablescanenable) | false | When doing an incremental query whether we should fall back to full table scans if file does not exist.<br />`Config Param: INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN` [...]
+| [hoodie.datasource.read.incr.filters](#hoodiedatasourcereadincrfilters) | | For use-cases like DeltaStreamer which reads from Hoodie Incremental table and applies opaque map functions, filters appearing late in the sequence of transformations cannot be automatically pushed down. This option allows setting filters directly on Hoodie Source.<br />`Config Param: PUSH_DOWN_INCR_FILTERS` [...]
+| [hoodie.datasource.read.incr.path.glob](#hoodiedatasourcereadincrpathglob) | | For the use-cases like users only want to incremental pull from certain partitions instead of the full table. This option allows using glob pattern to directly filter on path.<br />`Config Param: INCR_PATH_GLOB` [...]
+| [hoodie.datasource.read.schema.use.end.instanttime](#hoodiedatasourcereadschemauseendinstanttime) | false | Uses end instant schema when incrementally fetched data to. Default: users latest instant schema.<br />`Config Param: INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME` [...]
+| [hoodie.datasource.read.table.valued.function.filesystem.relation.subpath](#hoodiedatasourcereadtablevaluedfunctionfilesystemrelationsubpath) | | A regex under the table's base path to get file system view information<br />`Config Param: FILESYSTEM_RELATION_ARG_SUBPATH`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.read.table.valued.function.timeline.relation](#hoodiedatasourcereadtablevaluedfunctiontimelinerelation) | false | When this is set, the relation created by DefaultSource is for a view representing the result set of the table valued function hudi_query_timeline(...)<br />`Config Param: CREATE_TIMELINE_RELATION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.read.table.valued.function.timeline.relation.archived](#hoodiedatasourcereadtablevaluedfunctiontimelinerelationarchived) | false | When this is set, the result set of the table valued function hudi_query_timeline(...) will include archived timeline<br />`Config Param: TIMELINE_RELATION_ARG_ARCHIVED_TIMELINE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.streaming.startOffset](#hoodiedatasourcestreamingstartOffset) | earliest | Start offset to pull data from hoodie streaming source. allow earliest, latest, and specified start instant time<br />`Config Param: START_OFFSET`<br />`Since Version: 0.13.0` [...]
+| [hoodie.enable.data.skipping](#hoodieenabledataskipping) | true | Enables data-skipping allowing queries to leverage indexes to reduce the search space by skipping over files<br />`Config Param: ENABLE_DATA_SKIPPING`<br />`Since Version: 0.10.0` [...]
+| [hoodie.file.index.enable](#hoodiefileindexenable) | true | Enables use of the spark file index implementation for Hudi, that speeds up listing of large tables.<br />`Config Param: ENABLE_HOODIE_FILE_INDEX` [...]
+| [hoodie.read.timeline.holes.resolution.policy](#hoodiereadtimelineholesresolutionpolicy) | FAIL | When doing incremental queries, there could be hollow commits (requested or inflight commits that are not the latest) that are produced by concurrent writers and could lead to potential data loss. This config allows users to have different ways of handling this situation. The valid values are [FAIL, BLOCK, USE [...]
+| [hoodie.schema.on.read.enable](#hoodieschemaonreadenable) | false | Enables support for Schema Evolution feature<br />`Config Param: SCHEMA_EVOLUTION_ENABLED` [...]
+---
+
+
+### Write Options {#Write-Options}
+You can pass down any of the WriteClient level configs directly using `options()` or `option(k,v)` methods.
+
+```java
+inputDF.write()
+.format("org.apache.hudi")
+.options(clientOpts) // any of the Hudi client opts can be passed in as well
+.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")
+.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")
+.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")
+.option(HoodieWriteConfig.TABLE_NAME, tableName)
+.mode(SaveMode.Append)
+.save(basePath);
+```
+
+Options useful for writing tables via `write.format.option(...)`
+
+
+
+
+[**Basic Configs**](#Write-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` [...]
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD` [...]
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD` [...]
+| [hoodie.datasource.write.secondarykey.column](#hoodiedatasourcewritesecondarykeycolumn) | (N/A) | Columns that constitute the secondary key component. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: SECONDARYKEY_COLUMN_NAME` [...]
+| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` [...]
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` [...]
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` [...]
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING` [...]
+| [hoodie.datasource.write.operation](#hoodiedatasourcewriteoperation) | upsert | Whether to do upsert, insert or bulk_insert for the write operation. Use bulk_insert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br />`Config Param: OPERATION` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.datasource.write.table.type](#hoodiedatasourcewritetabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TABLE_TYPE` [...]
+| [hoodie.write.record.merge.mode](#hoodiewriterecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge records, i.e., the record with the large [...]
+
+[**Advanced Configs**](#Write-Options-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` [...]
+| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` [...]
+| [hoodie.datasource.overwrite.mode](#hoodiedatasourceoverwritemode) | (N/A) | Controls whether overwrite use dynamic or static mode, if not configured, respect spark.sql.sources.partitionOverwriteMode<br />`Config Param: OVERWRITE_MODE`<br />`Since Version: 0.14.0` [...]
+| [hoodie.datasource.write.partitions.to.delete](#hoodiedatasourcewritepartitionstodelete) | (N/A) | Comma separated list of partitions to delete. Allows use of wildcard *<br />`Config Param: PARTITIONS_TO_DELETE` [...]
+| [hoodie.datasource.write.payload.class](#hoodiedatasourcewritepayloadclass) | (N/A) | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
+| [hoodie.datasource.write.table.name](#hoodiedatasourcewritetablename) | (N/A) | Table name for the datasource write. Also used to register the table into meta stores.<br />`Config Param: TABLE_NAME` [...]
+| [hoodie.write.record.merge.custom.implementation.classes](#hoodiewriterecordmergecustomimplementationclasses) | (N/A) | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These record merge impls will filter by hoodie.write.record.merge.strategy.idHudi will pick most efficient implementation to perform merging/combining of the records (during update, re [...]
+| [hoodie.write.record.merge.strategy.id](#hoodiewriterecordmergestrategyid) | (N/A) | ID of record merge strategy. Hudi will pick HoodieRecordMerger implementations in `hoodie.write.record.merge.custom.implementation.classes` which has the same merge strategy id<br />`Config Param: RECORD_MERGE_STRATEGY_ID`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.compaction.async.enable](#hoodiedatasourcecompactionasyncenable) | true | Controls whether async compaction should be turned on for MOR table writing.<br />`Config Param: ASYNC_COMPACT_ENABLE` [...]
+| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` [...]
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: HIVE_BASE_FILE_FORMAT` [...]
+| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` [...]
+| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` [...]
+| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` [...]
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: HIVE_DATABASE` [...]
+| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` [...]
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: HIVE_PARTITION_EXTRACTOR_CLASS` [...]
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: HIVE_PARTITION_FIELDS` [...]
+| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` [...]
+| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` [...]
+| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` [...]
+| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` [...]
+| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` [...]
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: HIVE_TABLE` [...]
+| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` [...]
+| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Config Pa [...]
+| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` [...]
+| [hoodie.datasource.insert.dup.policy](#hoodiedatasourceinsertduppolicy) | none | **Note** This is only applicable to Spark SQL writing.<br />When operation type is set to "insert", users can optionally enforce a dedup policy. This policy will be employed when records being ingested already exists in storage. Default policy is none and no action will be taken. [...]
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: HIVE_CONDITIONAL_SYNC` [...]
+| [hoodie.datasource.write.commitmeta.key.prefix](#hoodiedatasourcewritecommitmetakeyprefix) | _ | Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata. This is useful to store checkpointing information, in a consistent way with the hudi timeline<br />`Config Param: COMMIT_METADATA_KEYPREFIX` [...]
+| [hoodie.datasource.write.drop.partition.columns](#hoodiedatasourcewritedroppartitioncolumns) | false | When set to true, will not write the partition columns into hudi. By default, false.<br />`Config Param: DROP_PARTITION_COLUMNS` [...]
+| [hoodie.datasource.write.insert.drop.duplicates](#hoodiedatasourcewriteinsertdropduplicates) | false | If set to true, records from the incoming dataframe will not overwrite existing records with the same key during the write operation. <br /> **Note** Just for Insert operation in Spark SQL writing since 0.14.0, users can switch to the config `hoodie.datasource.insert.dup.policy` i [...]
+| [hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass) | org.apache.hudi.keygen.SimpleKeyGenerator | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`<br />`Config Param: KEYGENERATOR_CLASS_NAME` [...]
+| [hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled](#hoodiedatasourcewritekeygeneratorconsistentlogicaltimestampenabled) | false | When set to true, consistent value will be generated for a logical timestamp type column, like timestamp-millis and timestamp-micros, irrespective of whether row-writer is enabled. Disabled by default so as not to break the pipeline that deploy either fully row-writer path or non row-wr [...]
+| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Should we url encode the partition path value, before creating the folder structure.<br />`Config Param: URL_ENCODE_PARTITIONING` [...]
+| [hoodie.datasource.write.reconcile.schema](#hoodiedatasourcewritereconcileschema) | false | This config controls how writer's schema will be selected based on the incoming batch's schema as well as existing table's one. When schema reconciliation is DISABLED, incoming batch's schema will be picked as a writer-schema (therefore updating table's schema). When schema reconciliati [...]
+| [hoodie.datasource.write.row.writer.enable](#hoodiedatasourcewriterowwriterenable) | true | When set to true, will perform write operations directly using the spark native `Row` representation, avoiding any additional conversion costs.<br />`Config Param: ENABLE_ROW_WRITER` [...]
+| [hoodie.datasource.write.streaming.checkpoint.identifier](#hoodiedatasourcewritestreamingcheckpointidentifier) | default_single_writer | A stream identifier used for HUDI to fetch the right checkpoint(`batch id` to be more specific) corresponding this writer. Please note that keep the identifier an unique value for different writer if under multi-writer scenario. If the value is not set, will only keep the checkpoint inf [...]
+| [hoodie.datasource.write.streaming.disable.compaction](#hoodiedatasourcewritestreamingdisablecompaction) | false | By default for MOR table, async compaction is enabled with spark streaming sink. By setting this config to true, we can disable it and the expectation is that, users will schedule and execute compaction in a different process/job altogether. Some users may wish to run it separately to m [...]
+| [hoodie.datasource.write.streaming.ignore.failed.batch](#hoodiedatasourcewritestreamingignorefailedbatch) | false | Config to indicate whether to ignore any non exception error (e.g. writestatus error) within a streaming microbatch. Turning this on, could hide the write status errors while the spark checkpoint moves ahead.So, would recommend users to use this with caution.<br />`Config Param: STREAMI [...]
+| [hoodie.datasource.write.streaming.retry.count](#hoodiedatasourcewritestreamingretrycount) | 3 | Config to indicate how many times streaming job should retry for a failed micro batch.<br />`Config Param: STREAMING_RETRY_CNT` [...]
+| [hoodie.datasource.write.streaming.retry.interval.ms](#hoodiedatasourcewritestreamingretryintervalms) | 2000 | Config to indicate how long (by millisecond) before a retry should issued for failed microbatch<br />`Config Param: STREAMING_RETRY_INTERVAL_MS` [...]
+| [hoodie.meta.sync.client.tool.class](#hoodiemetasyncclienttoolclass) | org.apache.hudi.hive.HiveSyncTool | Sync tool class name used to sync to metastore. Defaults to Hive.<br />`Config Param: META_SYNC_CLIENT_TOOL_CLASS_NAME` [...]
+| [hoodie.spark.sql.insert.into.operation](#hoodiesparksqlinsertintooperation) | insert | Sql write operation to use with INSERT_INTO spark sql command. This comes with 3 possible values, bulk_insert, insert and upsert. bulk_insert is generally meant for initial loads and is known to be performant compared to insert. But bulk_insert may not do small file management. If you p [...]
+| [hoodie.spark.sql.merge.into.partial.updates](#hoodiesparksqlmergeintopartialupdates) | true | Whether to write partial updates to the data blocks containing updates in MOR tables with Spark SQL MERGE INTO statement. The data blocks containing partial updates have a schema with a subset of fields compared to the full schema of the table.<br />`Config Param: ENABLE_MERGE_INTO_PART [...]
+| [hoodie.spark.sql.optimized.writes.enable](#hoodiesparksqloptimizedwritesenable) | true | Controls whether spark sql prepped update, delete, and merge are enabled.<br />`Config Param: SPARK_SQL_OPTIMIZED_WRITES`<br />`Since Version: 0.14.0` [...]
+| [hoodie.sql.bulk.insert.enable](#hoodiesqlbulkinsertenable) | false | When set to true, the sql insert statement will use bulk insert. This config is deprecated as of 0.14.0. Please use hoodie.spark.sql.insert.into.operation instead.<br />`Config Param: SQL_ENABLE_BULK_INSERT` [...]
+| [hoodie.sql.insert.mode](#hoodiesqlinsertmode) | upsert | Insert mode when insert data to pk-table. The optional modes are: upsert, strict and non-strict.For upsert mode, insert statement do the upsert operation for the pk-table which will update the duplicate record.For strict mode, insert statement will keep the primary key uniqueness constr [...]
+| [hoodie.streamer.source.kafka.value.deserializer.class](#hoodiestreamersourcekafkavaluedeserializerclass) | io.confluent.kafka.serializers.KafkaAvroDeserializer | This class is used by kafka client to deserialize the records<br />`Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.9.0` [...]
+| [hoodie.write.set.null.for.missing.columns](#hoodiewritesetnullformissingcolumns) | false | When a nullable column is missing from incoming batch during a write operation, the write operation will fail schema compatibility check. Set this option to true will make the missing column be filled with null values to successfully complete the write operation.<br />`Config Param: S [...]
+---
+
+
+### PreCommit Validator Configurations {#PreCommit-Validator-Configurations}
+The following set of configurations help validate new data before commits.
+
+
+
+[**Advanced Configs**](#PreCommit-Validator-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.precommit.validators](#hoodieprecommitvalidators) | | Comma separated list of class names that can be invoked to validate commit<br />`Config Param: VALIDATOR_CLASS_NAMES` |
+| [hoodie.precommit.validators.equality.sql.queries](#hoodieprecommitvalidatorsequalitysqlqueries) | | Spark SQL queries to run on table before committing new data to validate state before and after commit. Multiple queries separated by ';' delimiter are supported. Example: "select count(*) from \<TABLE_NAME\> Note \<TABLE_NAME\> is replaced by table state before and after commit.<br />`Config Param: EQUALITY_SQL_QUERIES` |
+| [hoodie.precommit.validators.inequality.sql.queries](#hoodieprecommitvalidatorsinequalitysqlqueries) | | Spark SQL queries to run on table before committing new data to validate state before and after commit.Multiple queries separated by ';' delimiter are supported.Example query: 'select count(*) from \<TABLE_NAME\> where col=null'Note \<TABLE_NAME\> variable is expected to be present in query.<br />`Config Param: INEQUALITY_SQL_QUERIES` |
+| [hoodie.precommit.validators.single.value.sql.queries](#hoodieprecommitvalidatorssinglevaluesqlqueries) | | Spark SQL queries to run on table before committing new data to validate state after commit.Multiple queries separated by ';' delimiter are supported.Expected result is included as part of query separated by '#'. Example query: 'query1#result1:query2#result2'Note \<TABLE_NAME\> variable is expected to be present in query.<br />`Config Param: SINGLE_VALUE_SQL_QUERIES` |
+---
+
+## Flink Sql Configs {#FLINK_SQL}
+These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
+
+
+### Flink Options {#Flink-Options}
+Flink jobs using the SQL can be configured through the options in WITH clause. The actual datasource level configs are listed below.
+
+
+
+[**Basic Configs**](#Flink-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name to register to Hive metastore<br /> `Config Param: DATABASE_NAME` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name to register to Hive metastore<br /> `Config Param: TABLE_NAME` [...]
+| [path](#path) | (N/A) | Base path for the target hoodie table. The path would be created if it does not exist, otherwise a Hoodie table expects to be initialized successfully<br /> `Config Param: PATH` [...]
+| [read.commits.limit](#readcommitslimit) | (N/A) | The maximum number of commits allowed to read in each instant check, if it is streaming read, the avg read instants number per-second would be 'read.commits.limit'/'read.streaming.check-interval', by default no limit<br /> `Config Param: READ_COMMITS_LIMIT` [...]
+| [read.end-commit](#readend-commit) | (N/A) | End commit instant for reading, the commit time format should be 'yyyyMMddHHmmss'<br /> `Config Param: READ_END_COMMIT` [...]
+| [read.start-commit](#readstart-commit) | (N/A) | Start commit instant for reading, the commit time format should be 'yyyyMMddHHmmss', by default reading from the latest instant for streaming read<br /> `Config Param: READ_START_COMMIT` [...]
+| [archive.max_commits](#archivemax_commits) | 50 | Max number of commits to keep before archiving older commits into a sequential log, default 50<br /> `Config Param: ARCHIVE_MAX_COMMITS` [...]
+| [archive.min_commits](#archivemin_commits) | 40 | Min number of commits to keep before archiving older commits into a sequential log, default 40<br /> `Config Param: ARCHIVE_MIN_COMMITS` [...]
+| [cdc.enabled](#cdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode<br /> `Config Param: CDC_ENABLED` [...]
+| [cdc.supplemental.logging.mode](#cdcsupplementalloggingmode) | DATA_BEFORE_AFTER | Setting 'op_key_only' persists the 'op' and the record key only, setting 'data_before' persists the additional 'before' image, and setting 'data_before_after' persists the additional 'before' and 'after' images.<br /> `Config Param: SUPPLEMENTAL_LOGGING_MODE` [...]
+| [changelog.enabled](#changelogenabled) | false | Whether to keep all the intermediate changes, we try to keep all the changes of a record when enabled: 1). The sink accept the UPDATE_BEFORE message; 2). The source try to emit every changes of a record. The semantics is best effort because the compaction job would finally merge all changes of a record into one. default false to have UPSERT semantics<br /> [...]
+| [clean.async.enabled](#cleanasyncenabled) | true | Whether to cleanup the old commits immediately on new commits, enabled by default<br /> `Config Param: CLEAN_ASYNC_ENABLED` [...]
+| [clean.retain_commits](#cleanretain_commits) | 30 | Number of commits to retain. So data will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much you can incrementally pull on this table, default 30<br /> `Config Param: CLEAN_RETAIN_COMMITS` [...]
+| [clustering.async.enabled](#clusteringasyncenabled) | false | Async Clustering, default false<br /> `Config Param: CLUSTERING_ASYNC_ENABLED` [...]
+| [clustering.plan.strategy.small.file.limit](#clusteringplanstrategysmallfilelimit) | 600 | Files smaller than the size specified here are candidates for clustering, default 600 MB<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_SMALL_FILE_LIMIT` [...]
+| [clustering.plan.strategy.target.file.max.bytes](#clusteringplanstrategytargetfilemaxbytes) | 1073741824 | Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups, default 1 GB<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_TARGET_FILE_MAX_BYTES` [...]
+| [compaction.async.enabled](#compactionasyncenabled) | true | Async Compaction, enabled by default for MOR<br /> `Config Param: COMPACTION_ASYNC_ENABLED` [...]
+| [compaction.delta_commits](#compactiondelta_commits) | 5 | Max delta commits needed to trigger compaction, default 5 commits<br /> `Config Param: COMPACTION_DELTA_COMMITS` [...]
+| [hive_sync.enabled](#hive_syncenabled) | false | Asynchronously sync Hive meta to HMS, default false<br /> `Config Param: HIVE_SYNC_ENABLED` [...]
+| [hive_sync.jdbc_url](#hive_syncjdbc_url) | jdbc:hive2://localhost:10000 | Jdbc URL for hive sync, default 'jdbc:hive2://localhost:10000'<br /> `Config Param: HIVE_SYNC_JDBC_URL` [...]
+| [hive_sync.metastore.uris](#hive_syncmetastoreuris) | | Metastore uris for hive sync, default ''<br /> `Config Param: HIVE_SYNC_METASTORE_URIS` [...]
+| [hive_sync.mode](#hive_syncmode) | HMS | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql, default 'hms'<br /> `Config Param: HIVE_SYNC_MODE` [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Decides how data files need to be read, in 1) Snapshot mode (obtain latest view, based on row & columnar data); 2) incremental mode (new data since an instantTime); 3) Read Optimized mode (obtain latest view, based on columnar data) .Default: snapshot<br /> `Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br /> `Config Param: HIVE_STYLE_PARTITIONING` [...]
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | | Partition path field. Value to be used at the `partitionPath` component of `HoodieKey`. Actual value obtained by invoking .toString(), default ''<br /> `Config Param: PARTITION_PATH_FIELD` [...]
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | uuid | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br /> `Config Param: RECORD_KEY_FIELD` [...]
+| [index.type](#indextype) | FLINK_STATE | Index type of Flink write job, default is using state backed index.<br /> `Config Param: INDEX_TYPE` [...]
+| [lookup.join.cache.ttl](#lookupjoincachettl) | PT1H | The cache TTL (e.g. 10min) for the build table in lookup join.<br /> `Config Param: LOOKUP_JOIN_CACHE_TTL` [...]
+| [metadata.compaction.delta_commits](#metadatacompactiondelta_commits) | 10 | Max delta commits for metadata table to trigger compaction, default 10<br /> `Config Param: METADATA_COMPACTION_DELTA_COMMITS` [...]
+| [metadata.enabled](#metadataenabled) | true | Enable the internal metadata table which serves table metadata like level file listings, default enabled<br /> `Config Param: METADATA_ENABLED` [...]
+| [precombine.field](#precombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br /> `Config Param: PRECOMBINE_FIELD` [...]
+| [read.streaming.enabled](#readstreamingenabled) | false | Whether to read as streaming source, default false<br /> `Config Param: READ_AS_STREAMING` [...]
+| [read.streaming.skip_insertoverwrite](#readstreamingskip_insertoverwrite) | false | Whether to skip insert overwrite instants to avoid reading base files of insert overwrite operations for streaming read. In streaming scenarios, insert overwrite is usually used to repair data, here you can control the visibility of downstream streaming read.<br /> `Config Param: READ_STREAMING_SKIP_INSERT_OVERWRITE` [...]
+| [table.type](#tabletype) | COPY_ON_WRITE | Type of table to write. COPY_ON_WRITE (or) MERGE_ON_READ<br /> `Config Param: TABLE_TYPE` [...]
+| [write.operation](#writeoperation) | upsert | The write operation, that this write should do<br /> `Config Param: OPERATION` [...]
+| [write.parquet.max.file.size](#writeparquetmaxfilesize) | 120 | Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /> `Config Param: WRITE_PARQUET_MAX_FILE_SIZE` [...]
+
+[**Advanced Configs**](#Flink-Options-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [clustering.tasks](#clusteringtasks) | (N/A) | Parallelism of tasks that do actual clustering, default same as the write task parallelism<br /> `Config Param: CLUSTERING_TASKS` [...]
+| [compaction.tasks](#compactiontasks) | (N/A) | Parallelism of tasks that do actual compaction, default same as the write task parallelism<br /> `Config Param: COMPACTION_TASKS` [...]
+| [hive_sync.conf.dir](#hive_syncconfdir) | (N/A) | The hive configuration directory, where the hive-site.xml lies in, the file should be put on the client machine<br /> `Config Param: HIVE_SYNC_CONF_DIR` [...]
+| [hive_sync.serde_properties](#hive_syncserde_properties) | (N/A) | Serde properties to hive table, the data format is k1=v1 k2=v2<br /> `Config Param: HIVE_SYNC_TABLE_SERDE_PROPERTIES` [...]
+| [hive_sync.table_properties](#hive_synctable_properties) | (N/A) | Additional properties to store with table, the data format is k1=v1 k2=v2<br /> `Config Param: HIVE_SYNC_TABLE_PROPERTIES` [...]
+| [hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass) | (N/A) | Key generator class, that implements will extract the key out of incoming record<br /> `Config Param: KEYGEN_CLASS_NAME` [...]
+| [read.tasks](#readtasks) | (N/A) | Parallelism of tasks that do actual read, default is the parallelism of the execution environment<br /> `Config Param: READ_TASKS` [...]
+| [source.avro-schema](#sourceavro-schema) | (N/A) | Source avro schema string, the parsed schema is used for deserialization<br /> `Config Param: SOURCE_AVRO_SCHEMA` [...]
+| [source.avro-schema.path](#sourceavro-schemapath) | (N/A) | Source avro schema file path, the parsed schema is used for deserialization<br /> `Config Param: SOURCE_AVRO_SCHEMA_PATH` [...]
+| [write.bucket_assign.tasks](#writebucket_assigntasks) | (N/A) | Parallelism of tasks that do bucket assign, default same as the write task parallelism<br /> `Config Param: BUCKET_ASSIGN_TASKS` [...]
+| [write.index_bootstrap.tasks](#writeindex_bootstraptasks) | (N/A) | Parallelism of tasks that do index bootstrap, default same as the write task parallelism<br /> `Config Param: INDEX_BOOTSTRAP_TASKS` [...]
+| [write.partition.format](#writepartitionformat) | (N/A) | Partition path format, only valid when 'write.datetime.partitioning' is true, default is: 1) 'yyyyMMddHH' for timestamp(3) WITHOUT TIME ZONE, LONG, FLOAT, DOUBLE, DECIMAL; 2) 'yyyyMMdd' for DATE and INT.<br /> `Config Param: PARTITION_FORMAT` [...]
+| [write.tasks](#writetasks) | (N/A) | Parallelism of tasks that do actual write, default is the parallelism of the execution environment<br /> `Config Param: WRITE_TASKS` [...]
+| [clean.policy](#cleanpolicy) | KEEP_LATEST_COMMITS | Clean policy to manage the Hudi table. Available option: KEEP_LATEST_COMMITS, KEEP_LATEST_FILE_VERSIONS, KEEP_LATEST_BY_HOURS.Default is KEEP_LATEST_COMMITS.<br /> `Config Param: CLEAN_POLICY` [...]
+| [clean.retain_file_versions](#cleanretain_file_versions) | 5 | Number of file versions to retain. default 5<br /> `Config Param: CLEAN_RETAIN_FILE_VERSIONS` [...]
+| [clean.retain_hours](#cleanretain_hours) | 24 | Number of hours for which commits need to be retained. This config provides a more flexible option ascompared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times [...]
+| [clustering.delta_commits](#clusteringdelta_commits) | 4 | Max delta commits needed to trigger clustering, default 4 commits<br /> `Config Param: CLUSTERING_DELTA_COMMITS` [...]
+| [clustering.plan.partition.filter.mode](#clusteringplanpartitionfiltermode) | NONE | Partition filter mode used in the creation of clustering plan. Available values are - NONE: do not filter table partition and thus the clustering plan will include all partitions that have clustering candidate.RECENT_DAYS: keep a continuous range of partitions, worked together wit [...]
+| [clustering.plan.strategy.class](#clusteringplanstrategyclass) | org.apache.hudi.client.clustering.plan.strategy.FlinkSizeBasedClusteringPlanStrategy | Config to provide a strategy class (subclass of ClusteringPlanStrategy) to create clustering plan i.e select what file groups are being clustered. Default strategy, looks at the last N (determined by clustering.plan.strategy.daybased.lookback.partitions) day based partitions picks [...]
+| [clustering.plan.strategy.cluster.begin.partition](#clusteringplanstrategyclusterbeginpartition) | | Begin partition used to filter partition (inclusive)<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_CLUSTER_BEGIN_PARTITION` [...]
+| [clustering.plan.strategy.cluster.end.partition](#clusteringplanstrategyclusterendpartition) | | End partition used to filter partition (inclusive)<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_CLUSTER_END_PARTITION` [...]
+| [clustering.plan.strategy.daybased.lookback.partitions](#clusteringplanstrategydaybasedlookbackpartitions) | 2 | Number of partitions to list to create ClusteringPlan, default is 2<br /> `Config Param: CLUSTERING_TARGET_PARTITIONS` [...]
+| [clustering.plan.strategy.daybased.skipfromlatest.partitions](#clusteringplanstrategydaybasedskipfromlatestpartitions) | 0 | Number of partitions to skip from latest when choosing partitions to create ClusteringPlan<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST` [...]
+| [clustering.plan.strategy.max.num.groups](#clusteringplanstrategymaxnumgroups) | 30 | Maximum number of groups to create as part of ClusteringPlan. Increasing groups will increase parallelism, default is 30<br /> `Config Param: CLUSTERING_MAX_NUM_GROUPS` [...]
+| [clustering.plan.strategy.partition.regex.pattern](#clusteringplanstrategypartitionregexpattern) | | Filter clustering partitions that matched regex pattern<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_PARTITION_REGEX_PATTERN` [...]
+| [clustering.plan.strategy.partition.selected](#clusteringplanstrategypartitionselected) | | Partitions to run clustering<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_PARTITION_SELECTED` [...]
+| [clustering.plan.strategy.sort.columns](#clusteringplanstrategysortcolumns) | | Columns to sort the data by when clustering<br /> `Config Param: CLUSTERING_SORT_COLUMNS` [...]
+| [clustering.schedule.enabled](#clusteringscheduleenabled) | false | Schedule the cluster plan, default false<br /> `Config Param: CLUSTERING_SCHEDULE_ENABLED` [...]
+| [compaction.delta_seconds](#compactiondelta_seconds) | 3600 | Max delta seconds time needed to trigger compaction, default 1 hour<br /> `Config Param: COMPACTION_DELTA_SECONDS` [...]
+| [compaction.max_memory](#compactionmax_memory) | 100 | Max memory in MB for compaction spillable map, default 100MB<br /> `Config Param: COMPACTION_MAX_MEMORY` [...]
+| [compaction.schedule.enabled](#compactionscheduleenabled) | true | Schedule the compaction plan, enabled by default for MOR<br /> `Config Param: COMPACTION_SCHEDULE_ENABLED` [...]
+| [compaction.target_io](#compactiontarget_io) | 512000 | Target IO in MB for per compaction (both read and write), default 500 GB<br /> `Config Param: COMPACTION_TARGET_IO` [...]
+| [compaction.timeout.seconds](#compactiontimeoutseconds) | 1200 | Max timeout time in seconds for online compaction to rollback, default 20 minutes<br /> `Config Param: COMPACTION_TIMEOUT_SECONDS` [...]
+| [compaction.trigger.strategy](#compactiontriggerstrategy) | num_commits | Strategy to trigger compaction, options are 'num_commits': trigger compaction when there are at least N delta commits after last completed compaction; 'num_commits_after_last_request': trigger compaction when there are at least N delta commits after last completed/requested compac [...]
+| [hive_sync.assume_date_partitioning](#hive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/mm/dd, default false<br /> `Config Param: HIVE_SYNC_ASSUME_DATE_PARTITION` [...]
+| [hive_sync.auto_create_db](#hive_syncauto_create_db) | true | Auto create hive database if it does not exists, default true<br /> `Config Param: HIVE_SYNC_AUTO_CREATE_DB` [...]
+| [hive_sync.db](#hive_syncdb) | default | Database name for hive sync, default 'default'<br /> `Config Param: HIVE_SYNC_DB` [...]
+| [hive_sync.file_format](#hive_syncfile_format) | PARQUET | File format for hive sync, default 'PARQUET'<br /> `Config Param: HIVE_SYNC_FILE_FORMAT` [...]
+| [hive_sync.ignore_exceptions](#hive_syncignore_exceptions) | false | Ignore exceptions during hive synchronization, default false<br /> `Config Param: HIVE_SYNC_IGNORE_EXCEPTIONS` [...]
+| [hive_sync.partition_extractor_class](#hive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Tool to extract the partition value from HDFS path, default 'MultiPartKeysValueExtractor'<br /> `Config Param: HIVE_SYNC_PARTITION_EXTRACTOR_CLASS_NAME` [...]
+| [hive_sync.partition_fields](#hive_syncpartition_fields) | | Partition fields for hive sync, default ''<br /> `Config Param: HIVE_SYNC_PARTITION_FIELDS` [...]
+| [hive_sync.password](#hive_syncpassword) | hive | Password for hive sync, default 'hive'<br /> `Config Param: HIVE_SYNC_PASSWORD` [...]
+| [hive_sync.skip_ro_suffix](#hive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table when registering, default false<br /> `Config Param: HIVE_SYNC_SKIP_RO_SUFFIX` [...]
+| [hive_sync.support_timestamp](#hive_syncsupport_timestamp) | true | INT64 with original type TIMESTAMP_MICROS is converted to hive timestamp type. Disabled by default for backward compatibility.<br /> `Config Param: HIVE_SYNC_SUPPORT_TIMESTAMP` [...]
+| [hive_sync.table](#hive_synctable) | unknown | Table name for hive sync, default 'unknown'<br /> `Config Param: HIVE_SYNC_TABLE` [...]
+| [hive_sync.table.strategy](#hive_synctablestrategy) | ALL | Hive table synchronization strategy. Available option: RO, RT, ALL.<br /> `Config Param: HIVE_SYNC_TABLE_STRATEGY` [...]
+| [hive_sync.use_jdbc](#hive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled, default true<br /> `Config Param: HIVE_SYNC_USE_JDBC` [...]
+| [hive_sync.username](#hive_syncusername) | hive | Username for hive sync, default 'hive'<br /> `Config Param: HIVE_SYNC_USERNAME` [...]
+| [hoodie.bucket.index.hash.field](#hoodiebucketindexhashfield) | | Index key field. Value to be used as hashing to find the bucket ID. Should be a subset of or equal to the recordKey fields. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br /> `Config Par [...]
+| [hoodie.bucket.index.num.buckets](#hoodiebucketindexnumbuckets) | 4 | Hudi bucket number per partition. Only affected if using Hudi bucket index.<br /> `Config Param: BUCKET_INDEX_NUM_BUCKETS` [...]
+| [hoodie.datasource.merge.type](#hoodiedatasourcemergetype) | payload_combine | For Snapshot query on merge on read table. Use this key to define how the payloads are merged, in 1) skip_merge: read the base file records plus the log file records without merging; 2) payload_combine: read the base file records first, for each record in base file, checks whether [...]
+| [hoodie.datasource.write.keygenerator.type](#hoodiedatasourcewritekeygeneratortype) | SIMPLE | Key generator type, that implements will extract the key out of incoming record. **Note** This is being actively worked on. Please use `hoodie.datasource.write.keygenerator.class` instead.<br /> `Config Param: KEYGEN_TYPE` [...]
+| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Whether to encode the partition path url, default false<br /> `Config Param: URL_ENCODE_PARTITIONING` [...]
+| [hoodie.index.bucket.engine](#hoodieindexbucketengine) | SIMPLE | Type of bucket index engine. Available options: [SIMPLE | CONSISTENT_HASHING]<br /> `Config Param: BUCKET_INDEX_ENGINE_TYPE` [...]
+| [hoodie.write.table.version](#hoodiewritetableversion) | 8 | Table version produced by this writer.<br /> `Config Param: WRITE_TABLE_VERSION` [...]
+| [index.bootstrap.enabled](#indexbootstrapenabled) | false | Whether to bootstrap the index state from existing hoodie table, default false<br /> `Config Param: INDEX_BOOTSTRAP_ENABLED` [...]
+| [index.global.enabled](#indexglobalenabled) | true | Whether to update index for the old partition path if same key record with different partition path came in, default true<br /> `Config Param: INDEX_GLOBAL_ENABLED` [...]
+| [index.partition.regex](#indexpartitionregex) | .* | Whether to load partitions in state if partition path matching， default `*`<br /> `Config Param: INDEX_PARTITION_REGEX` [...]
+| [index.state.ttl](#indexstatettl) | 0.0 | Index state ttl in days, default stores the index permanently<br /> `Config Param: INDEX_STATE_TTL` [...]
+| [partition.default_name](#partitiondefault_name) | __HIVE_DEFAULT_PARTITION__ | The default partition name in case the dynamic partition column value is null/empty string<br /> `Config Param: PARTITION_DEFAULT_NAME` [...]
+| [payload.class](#payloadclass) | org.apache.hudi.common.model.EventTimeAvroPayload | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for the option in-effective<br /> `Config Param: PAYLOAD_CLASS_NAME` [...]
+| [read.cdc.from.changelog](#readcdcfromchangelog) | true | Whether to consume the delta changes only from the cdc changelog files. When CDC is enabled, i). for COW table, the changelog is generated on each file update; ii). for MOR table, the changelog is generated on compaction. By default, always read from the changelog file, once it is [...]
+| [read.data.skipping.enabled](#readdataskippingenabled) | false | Enables data-skipping allowing queries to leverage indexes to reduce the search space by skipping over files<br /> `Config Param: READ_DATA_SKIPPING_ENABLED` [...]
+| [read.streaming.check-interval](#readstreamingcheck-interval) | 60 | Check interval for streaming read of SECOND, default 1 minute<br /> `Config Param: READ_STREAMING_CHECK_INTERVAL` [...]
+| [read.streaming.skip_clustering](#readstreamingskip_clustering) | true | Whether to skip clustering instants to avoid reading base files of clustering operations for streaming read to improve read performance.<br /> `Config Param: READ_STREAMING_SKIP_CLUSTERING` [...]
+| [read.streaming.skip_compaction](#readstreamingskip_compaction) | true | Whether to skip compaction instants and avoid reading compacted base files for streaming read to improve read performance. This option can be used to avoid reading duplicates when changelog mode is enabled, it is a solution to keep data integrity <br /> `Config Param: READ_STREAMI [...]
+| [read.utc-timezone](#readutc-timezone) | true | Use UTC timezone or local timezone to the conversion between epoch time and LocalDateTime. Hive 0.x/1.x/2.x use local timezone. But Hive 3.x use UTC timezone, by default true<br /> `Config Param: READ_UTC_TIMEZONE` [...]
+| [record.merger.impls](#recordmergerimpls) | org.apache.hudi.common.model.HoodieAvroRecordMerger | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These merger impls will filter by record.merger.strategy. Hudi will pick most efficient implementation to perform merging/combining of the records (during update, reading MOR tab [...]
+| [record.merger.strategy](#recordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in record.merger.impls which has the same merger strategy id<br /> `Config Param: RECORD_MERGER_STRATEGY_ID` [...]
+| [write.batch.size](#writebatchsize) | 256.0 | Batch buffer size in MB to flush data into the underneath filesystem, default 256MB<br /> `Config Param: WRITE_BATCH_SIZE` [...]
+| [write.bulk_insert.shuffle_input](#writebulk_insertshuffle_input) | true | Whether to shuffle the inputs by specific fields for bulk insert tasks, default true<br /> `Config Param: WRITE_BULK_INSERT_SHUFFLE_INPUT` [...]
+| [write.bulk_insert.sort_input](#writebulk_insertsort_input) | true | Whether to sort the inputs by specific fields for bulk insert tasks, default true<br /> `Config Param: WRITE_BULK_INSERT_SORT_INPUT` [...]
+| [write.bulk_insert.sort_input.by_record_key](#writebulk_insertsort_inputby_record_key) | false | Whether to sort the inputs by record keys for bulk insert tasks, default false<br /> `Config Param: WRITE_BULK_INSERT_SORT_INPUT_BY_RECORD_KEY` [...]
+| [write.client.id](#writeclientid) | | Unique identifier used to distinguish different writer pipelines for concurrent mode<br /> `Config Param: WRITE_CLIENT_ID` [...]
+| [write.commit.ack.timeout](#writecommitacktimeout) | -1 | Timeout limit for a writer task after it finishes a checkpoint and waits for the instant commit success, only for internal use<br /> `Config Param: WRITE_COMMIT_ACK_TIMEOUT` [...]
+| [write.ignore.failed](#writeignorefailed) | false | Flag to indicate whether to ignore any non exception error (e.g. writestatus error). within a checkpoint batch. By default false. Turning this on, could hide the write status errors while the flink checkpoint moves ahead. So, would recommend users to use this with caution.<br /> [...]
+| [write.insert.cluster](#writeinsertcluster) | false | Whether to merge small files for insert mode, if true, the write throughput will decrease because the read/write of existing small file, only valid for COW table, default false<br /> `Config Param: INSERT_CLUSTER` [...]
+| [write.log.max.size](#writelogmaxsize) | 1024 | Maximum size allowed in MB for a log file before it is rolled over to the next version, default 1GB<br /> `Config Param: WRITE_LOG_MAX_SIZE` [...]
+| [write.log_block.size](#writelog_blocksize) | 128 | Max log block size in MB for log file, default 128MB<br /> `Config Param: WRITE_LOG_BLOCK_SIZE` [...]
+| [write.merge.max_memory](#writemergemax_memory) | 100 | Max memory in MB for merge, default 100MB<br /> `Config Param: WRITE_MERGE_MAX_MEMORY` [...]
+| [write.parquet.block.size](#writeparquetblocksize) | 120 | Parquet RowGroup size. It's recommended to make this large enough that scan costs can be amortized by packing enough column values into a single row group.<br /> `Config Param: WRITE_PARQUET_BLOCK_SIZE` [...]
+| [write.parquet.page.size](#writeparquetpagesize) | 1 | Parquet page size. Page is the unit of read within a parquet file. Within a block, pages are compressed separately.<br /> `Config Param: WRITE_PARQUET_PAGE_SIZE` [...]
+| [write.partition.overwrite.mode](#writepartitionoverwritemode) | STATIC | When INSERT OVERWRITE a partitioned data source table, we currently support 2 modes: static and dynamic. Static mode deletes all the partitions that match the partition specification(e.g. PARTITION(a=1,b)) in the INSERT statement, before overwriting. Dynamic mode doesn't delete pa [...]
+| [write.precombine](#writeprecombine) | false | Flag to indicate whether to drop duplicates before insert/upsert. By default these cases will accept duplicates, to gain extra performance: 1) insert operation; 2) upsert for MOR table, the MOR table deduplicate on reading<br /> `Config Param: PRE_COMBINE` [...]
+| [write.rate.limit](#writeratelimit) | 0 | Write record rate limit per second to prevent traffic jitter and improve stability, default 0 (no limit)<br /> `Config Param: WRITE_RATE_LIMIT` [...]
+| [write.retry.interval.ms](#writeretryintervalms) | 2000 | Flag to indicate how long (by millisecond) before a retry should issued for failed checkpoint batch. By default 2000 and it will be doubled by every retry<br /> `Config Param: RETRY_INTERVAL_MS` [...]
+| [write.retry.times](#writeretrytimes) | 3 | Flag to indicate how many times streaming job should retry for a failed checkpoint batch. By default 3<br /> `Config Param: RETRY_TIMES` [...]
+| [write.sort.memory](#writesortmemory) | 128 | Sort memory in MB, default 128MB<br /> `Config Param: WRITE_SORT_MEMORY` [...]
+| [write.task.max.size](#writetaskmaxsize) | 1024.0 | Maximum memory in MB for a write task, when the threshold hits, it flushes the max size data bucket to avoid OOM, default 1GB<br /> `Config Param: WRITE_TASK_MAX_SIZE` [...]
+| [write.utc-timezone](#writeutc-timezone) | true | Use UTC timezone or local timezone to the conversion between epoch time and LocalDateTime. Default value is utc timezone for forward compatibility.<br /> `Config Param: WRITE_UTC_TIMEZONE` [...]
+---
+
+## Write Client Configs {#WRITE_CLIENT}
+Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.
+
+
+### Common Configurations {#Common-Configurations}
+The following set of configurations are common across Hudi.
+
+
+
+[**Basic Configs**](#Common-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` |
+
+[**Advanced Configs**](#Common-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description [...]
... 18070 lines suppressed ...