You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by yi...@apache.org on 2026/05/28 06:34:29 UTC
(hudi) branch asf-site updated: docs: Adding release docs for 0.15.1 (#18804)
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4aa979174a0b docs: Adding release docs for 0.15.1 (#18804)
4aa979174a0b is described below
commit 4aa979174a0b3166ccad184b7bd4dafccac1d298
Author: voonhous <vo...@gmail.com>
AuthorDate: Thu May 28 14:34:23 2026 +0800
docs: Adding release docs for 0.15.1 (#18804)
Co-authored-by: Y Ethan Guo <et...@gmail.com>
---
website/docusaurus.config.js | 1 +
website/releases/download.md | 5 +
website/releases/release-0.15.md | 28 +
.../versioned_docs/version-0.15.1/azure_hoodie.md | 50 +
.../version-0.15.1/basic_configurations.md | 754 +++++++
.../versioned_docs/version-0.15.1/bos_hoodie.md | 57 +
website/versioned_docs/version-0.15.1/cli.md | 757 +++++++
website/versioned_docs/version-0.15.1/cloud.md | 38 +
.../versioned_docs/version-0.15.1/clustering.md | 343 +++
.../versioned_docs/version-0.15.1/compaction.md | 228 ++
.../versioned_docs/version-0.15.1/comparison.md | 56 +
website/versioned_docs/version-0.15.1/concepts.md | 172 ++
.../version-0.15.1/concurrency_control.md | 304 +++
.../version-0.15.1/configurations.md | 2187 ++++++++++++++++++++
.../versioned_docs/version-0.15.1/cos_hoodie.md | 71 +
.../versioned_docs/version-0.15.1/deployment.md | 261 +++
.../version-0.15.1/disaster_recovery.md | 309 +++
.../versioned_docs/version-0.15.1/docker_demo.md | 1526 ++++++++++++++
.../versioned_docs/version-0.15.1/encryption.md | 73 +
website/versioned_docs/version-0.15.1/faq.md | 15 +
.../version-0.15.1/faq_design_and_concepts.md | 61 +
.../versioned_docs/version-0.15.1/faq_general.md | 98 +
.../version-0.15.1/faq_integrations.md | 68 +
.../version-0.15.1/faq_reading_tables.md | 31 +
.../versioned_docs/version-0.15.1/faq_storage.md | 207 ++
.../version-0.15.1/faq_table_services.md | 55 +
.../version-0.15.1/faq_writing_tables.md | 194 ++
.../versioned_docs/version-0.15.1/file_layouts.md | 20 +
.../versioned_docs/version-0.15.1/file_sizing.md | 177 ++
.../version-0.15.1/flink-quick-start-guide.md | 473 +++++
.../versioned_docs/version-0.15.1/flink_tuning.md | 117 ++
.../versioned_docs/version-0.15.1/gcp_bigquery.md | 103 +
.../versioned_docs/version-0.15.1/gcs_hoodie.md | 60 +
.../version-0.15.1/hoodie_cleaner.md | 148 ++
.../version-0.15.1/hoodie_streaming_ingestion.md | 635 ++++++
.../versioned_docs/version-0.15.1/hudi_stack.md | 99 +
.../version-0.15.1/ibm_cos_hoodie.md | 77 +
website/versioned_docs/version-0.15.1/indexing.md | 158 ++
.../version-0.15.1/ingestion_flink.md | 179 ++
.../version-0.15.1/ingestion_kafka_connect.md | 7 +
.../versioned_docs/version-0.15.1/jfs_hoodie.md | 96 +
.../version-0.15.1/key_generation.md | 212 ++
website/versioned_docs/version-0.15.1/markers.md | 91 +
website/versioned_docs/version-0.15.1/metadata.md | 200 ++
.../version-0.15.1/metadata_indexing.md | 216 ++
website/versioned_docs/version-0.15.1/metrics.md | 232 +++
.../version-0.15.1/migration_guide.md | 117 ++
.../versioned_docs/version-0.15.1/oci_hoodie.md | 80 +
.../versioned_docs/version-0.15.1/oss_hoodie.md | 70 +
website/versioned_docs/version-0.15.1/overview.mdx | 79 +
.../versioned_docs/version-0.15.1/performance.md | 133 ++
.../platform_services_post_commit_callback.md | 57 +
.../version-0.15.1/precommit_validator.md | 115 +
website/versioned_docs/version-0.15.1/privacy.md | 22 +
.../versioned_docs/version-0.15.1/procedures.md | 1999 ++++++++++++++++++
.../python-rust-quick-start-guide.md | 247 +++
.../versioned_docs/version-0.15.1/querying_data.md | 99 +
.../version-0.15.1/quick-start-guide.md | 1187 +++++++++++
.../version-0.15.1/reading_tables_batch_reads.md | 35 +
.../reading_tables_streaming_reads.md | 99 +
.../version-0.15.1/record_payload.md | 177 ++
website/versioned_docs/version-0.15.1/rollbacks.md | 72 +
website/versioned_docs/version-0.15.1/s3_hoodie.md | 94 +
.../version-0.15.1/schema_evolution.md | 319 +++
.../version-0.15.1/snapshot_exporter.md | 116 ++
website/versioned_docs/version-0.15.1/sql_ddl.md | 674 ++++++
website/versioned_docs/version-0.15.1/sql_dml.md | 464 +++++
.../versioned_docs/version-0.15.1/sql_queries.md | 399 ++++
website/versioned_docs/version-0.15.1/structure.md | 20 +
.../syncing_aws_glue_data_catalog.md | 72 +
.../version-0.15.1/syncing_datahub.md | 49 +
.../version-0.15.1/syncing_metastore.md | 299 +++
.../version-0.15.1/syncing_xtable.md | 54 +
.../versioned_docs/version-0.15.1/table_types.md | 152 ++
website/versioned_docs/version-0.15.1/timeline.md | 115 +
.../version-0.15.1/troubleshooting.md | 228 ++
.../versioned_docs/version-0.15.1/tuning-guide.md | 106 +
website/versioned_docs/version-0.15.1/use_cases.md | 138 ++
.../version-0.15.1/write_operations.md | 121 ++
.../versioned_docs/version-0.15.1/writing_data.md | 445 ++++
.../writing_tables_streaming_writes.md | 95 +
.../version-0.15.1-sidebars.json | 163 ++
website/versions.json | 1 +
83 files changed, 19961 insertions(+)
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index 07d506d0a1cb..f0e054a7c1e4 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -162,6 +162,7 @@ module.exports = {
},
{
from: [
+ "/releases/release-0.15.1",
"/releases/release-0.15.0",
],
to: "/releases/release-0.15",
diff --git a/website/releases/download.md b/website/releases/download.md
index 814cafdafebe..f1053edd8410 100644
--- a/website/releases/download.md
+++ b/website/releases/download.md
@@ -205,6 +205,11 @@ toc: true
* Source Release : [Apache Hudi 1.0.0 Source Release](https://downloads.apache.org/hudi/1.0.0/hudi-1.0.0.src.tgz) ([asc](https://downloads.apache.org/hudi/1.0.0/hudi-1.0.0.src.tgz.asc), [sha512](https://downloads.apache.org/hudi/1.0.0/hudi-1.0.0.src.tgz.sha512))
* Release Note : ([Release Note for Apache Hudi 1.0.0](/releases/release-1.0#release-100))
+## Release 0.15.1
+
+* Source Release : [Apache Hudi 0.15.1 Source Release](https://downloads.apache.org/hudi/0.15.1/hudi-0.15.1.src.tgz) ([asc](https://downloads.apache.org/hudi/0.15.1/hudi-0.15.1.src.tgz.asc), [sha512](https://downloads.apache.org/hudi/0.15.1/hudi-0.15.1.src.tgz.sha512))
+* Release Note : ([Release Note for Apache Hudi 0.15.1](/releases/release-0.15#release-0151))
+
## Release 0.15.0
* Source Release : [Apache Hudi 0.15.0 Source Release](https://downloads.apache.org/hudi/0.15.0/hudi-0.15.0.src.tgz) ([asc](https://downloads.apache.org/hudi/0.15.0/hudi-0.15.0.src.tgz.asc), [sha512](https://downloads.apache.org/hudi/0.15.0/hudi-0.15.0.src.tgz.sha512))
diff --git a/website/releases/release-0.15.md b/website/releases/release-0.15.md
index 2701f7ae82bf..8a2fb7ad1d20 100644
--- a/website/releases/release-0.15.md
+++ b/website/releases/release-0.15.md
@@ -9,6 +9,7 @@ import TabItem from '@theme/TabItem';
This page contains release notes for all Apache Hudi 0.15.x releases, including:
- [Release 0.15.0](#release-0150)
+- [Release 0.15.1](#release-0151)
---
@@ -335,3 +336,30 @@ Avoid upgrading any existing table to 0.14.1 and 0.15.0 from any prior version i
The raw release notes are
available [here](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12322822&version=12353381).
+
+## [Release 0.15.1](https://github.com/apache/hudi/releases/tag/release-0.15.1)
+
+## Migration Guide
+* This release (0.15.1) does not introduce any new table version, thus no migration is needed if you are on 0.15.0.
+* If migrating from an older release, please check the migration guide from the previous release notes, specifically
+ the upgrade instructions in [0.6.0](/releases/release-0.6.0),
+ [0.9.0](/releases/release-0.9.0), [0.10.0](/releases/release-0.10.0),
+ [0.11.0](/releases/release-0.11.0), [0.12.0](/releases/release-0.12.0), [0.13.0](/releases/release-0.13.0), [0.14.0](/releases/release-0.14#release-0140), and
+ [0.15.0](/releases/release-0.15#release-0150)
+
+### Bug fixes
+
+0.15.1 release is primarily intended for bug fixes and stability, including these areas:
+
+* Fix for the Complex Key Generator regression flagged in 0.15.0.
+* Databricks Spark runtime compatibility for reading Hudi tables.
+* Column stats index correctness with metadata table and timestamp column logical type fixes.
+* Hive/Glue and DataHub meta sync improvements.
+* Flink bug fixes.
+* Security fixes.
+
+## Known Regressions
+
+No known regressions at the time of release. The Complex Key Generator regression reported in 0.15.0 is fixed in 0.15.1
+via [HUDI-9666](https://issues.apache.org/jira/browse/HUDI-9666); users on 0.14.1 / 0.15.0 with `ComplexKeyGenerator`
+and a single record key field are encouraged to upgrade.
diff --git a/website/versioned_docs/version-0.15.1/azure_hoodie.md b/website/versioned_docs/version-0.15.1/azure_hoodie.md
new file mode 100644
index 000000000000..77aa6e55583a
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/azure_hoodie.md
@@ -0,0 +1,50 @@
+---
+title: Microsoft Azure
+keywords: [ hudi, hive, azure, spark, presto]
+summary: In this page, we go over how to configure Hudi with Azure filesystem.
+last_modified_at: 2020-05-25T19:00:57-04:00
+---
+In this page, we explain how to use Hudi on Microsoft Azure.
+
+## Disclaimer
+
+This page is maintained by the Hudi community.
+If the information is inaccurate or you have additional information to add.
+Please feel free to create a GitHub issue. Contribution is highly appreciated.
+
+## Supported Storage System
+
+There are two storage systems support Hudi .
+
+- Azure Blob Storage
+- Azure Data Lake Gen 2
+
+## Verified Combination of Spark and storage system
+
+#### HDInsight Spark2.4 on Azure Data Lake Storage Gen 2
+This combination works out of the box. No extra config needed.
+
+#### Databricks Spark2.4 on Azure Data Lake Storage Gen 2
+- Import Hudi jar to databricks workspace
+
+- Mount the file system to dbutils.
+ ```scala
+ dbutils.fs.mount(
+ source = "abfss://xxx@xxx.dfs.core.windows.net",
+ mountPoint = "/mountpoint",
+ extraConfigs = configs)
+ ```
+- When writing Hudi dataset, use abfss URL
+ ```scala
+ inputDF.write
+ .format("org.apache.hudi")
+ .options(opts)
+ .mode(SaveMode.Append)
+ .save("abfss://<<storage-account>>.dfs.core.windows.net/hudi-tables/customer")
+ ```
+- When reading Hudi dataset, use the mounting point
+ ```scala
+ spark.read
+ .format("org.apache.hudi")
+ .load("/mountpoint/hudi-tables/customer")
+ ```
diff --git a/website/versioned_docs/version-0.15.1/basic_configurations.md b/website/versioned_docs/version-0.15.1/basic_configurations.md
new file mode 100644
index 000000000000..ae9099b2facb
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/basic_configurations.md
@@ -0,0 +1,754 @@
+---
+title: Basic Configurations
+summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
+last_modified_at: 2024-06-06T12:05:06.444
+---
+
+
+This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
+
+- [**Hudi Table Config**](#TABLE_CONFIG): Basic Hudi Table configuration parameters.
+- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
+- [**Flink Sql Configs**](#FLINK_SQL): These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
+- [**Write Client Configs**](#WRITE_CLIENT): Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.
+- [**Metastore and Catalog Sync Configs**](#META_SYNC): Configurations used by the Hudi to sync metadata to external metastores and catalogs.
+- [**Metrics Configs**](#METRICS): These set of configs are used to enable monitoring and reporting of key Hudi stats and metrics.
+- [**Kafka Connect Configs**](#KAFKA_CONNECT): These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables
+- [**Hudi Streamer Configs**](#HUDI_STREAMER): These set of configs are used for Hudi Streamer utility which provides the way to ingest from different sources such as DFS or Kafka.
+
+:::note
+In the tables below **(N/A)** means there is no default value set
+:::
+
+## Hudi Table Config {#TABLE_CONFIG}
+Basic Hudi Table configuration parameters.
+
+
+### Hudi Table Basic Configs {#Hudi-Table-Basic-Configs}
+Configurations of the Hudi Table like type of ingestion, storage formats, hive table name etc. Configurations are loaded from hoodie.properties, these properties are usually set during initializing a path as hoodie base path and never changes during the lifetime of a hoodie table.
+
+
+
+
+[**Basic Configs**](#Hudi-Table-Basic-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BOOTSTRAP_BASE_PATH` [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name that will be used for incremental query.If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME` [...]
+| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.<br />`Config Param: TABLE_CHECKSUM`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.create.schema](#hoodietablecreateschema) | (N/A) | Schema used when creating the table, for the first time.<br />`Config Param: CREATE_SCHEMA` [...]
+| [hoodie.table.keygenerator.class](#hoodietablekeygeneratorclass) | (N/A) | Key Generator class property for the hoodie table<br />`Config Param: KEY_GENERATOR_CLASS_NAME` [...]
+| [hoodie.table.metadata.partitions](#hoodietablemetadatapartitions) | (N/A) | Comma-separated list of metadata partitions that have been completely built and in-sync with data table. These partitions are ready for use by the readers<br />`Config Param: TABLE_METADATA_PARTITIONS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.metadata.partitions.inflight](#hoodietablemetadatapartitionsinflight) | (N/A) | Comma-separated list of metadata partitions whose building is in progress. These partitions are not yet ready for use by the readers.<br />`Config Param: TABLE_METADATA_PARTITIONS_INFLIGHT`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: NAME` [...]
+| [hoodie.table.partition.fields](#hoodietablepartitionfields) | (N/A) | Fields used to partition the table. Concatenated values of these fields are used as the partition path, by invoking toString()<br />`Config Param: PARTITION_FIELDS` [...]
+| [hoodie.table.precombine.field](#hoodietableprecombinefield) | (N/A) | Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.table.recordkey.fields](#hoodietablerecordkeyfields) | (N/A) | Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br />`Config Param: RECORDKEY_FIELDS` [...]
+| [hoodie.table.secondary.indexes.metadata](#hoodietablesecondaryindexesmetadata) | (N/A) | The metadata of secondary indexes<br />`Config Param: SECONDARY_INDEXES_METADATA`<br />`Since Version: 0.13.0` [...]
+| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | (N/A) | Version of timeline used, by the table.<br />`Config Param: TIMELINE_LAYOUT_VERSION` [...]
+| [hoodie.archivelog.folder](#hoodiearchivelogfolder) | archived | path under the meta folder, to store archived timeline instants at.<br />`Config Param: ARCHIVELOG_FOLDER` [...]
+| [hoodie.bootstrap.index.class](#hoodiebootstrapindexclass) | org.apache.hudi.common.bootstrap.index.hfile.HFileBootstrapIndex | Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_CLASS_NAME` [...]
+| [hoodie.bootstrap.index.enable](#hoodiebootstrapindexenable) | true | Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined, default true.<br />`Config Param: BOOTSTRAP_INDEX_ENABLE` [...]
+| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | org.apache.hudi.common.model.OverwriteWithLatestAvroPayload | Payload class to use for performing compactions, i.e merge delta logs with current base file and then produce a new base file.<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
+| [hoodie.compaction.record.merger.strategy](#hoodiecompactionrecordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in hoodie.datasource.write.record.merger.impls which has the same merger strategy id<br />`Config Param: RECORD_MERGER_STRATEGY`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` [...]
+| [hoodie.partition.metafile.use.base.format](#hoodiepartitionmetafileusebaseformat) | false | If true, partition metafiles are saved in the same format as base-files for this dataset (e.g. Parquet / ORC). If false (default) partition metafiles are saved as properties files.<br />`Config Param: PARTITION_METAFILE_USE_BASE_FORMAT` [...]
+| [hoodie.populate.meta.fields](#hoodiepopulatemetafields) | true | When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br />`Config Param: POPULATE_META_FIELDS` [...]
+| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | Base file format to store all the base file data.<br />`Config Param: BASE_FILE_FORMAT` [...]
+| [hoodie.table.cdc.enabled](#hoodietablecdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode.<br />`Config Param: CDC_ENABLED`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.cdc.supplemental.logging.mode](#hoodietablecdcsupplementalloggingmode) | DATA_BEFORE_AFTER | org.apache.hudi.common.table.cdc.HoodieCDCSupplementalLoggingMode: Change log capture supplemental logging mode. The supplemental log is used for accelerating the generation of change log details. OP_KEY_ONLY: Only keeping record keys in the supplemental logs, so the reader needs to figure out the update before image [...]
+| [hoodie.table.log.file.format](#hoodietablelogfileformat) | HOODIE_LOG | Log format used for the delta logs.<br />`Config Param: LOG_FILE_FORMAT` [...]
+| [hoodie.table.timeline.timezone](#hoodietabletimelinetimezone) | LOCAL | User can set hoodie commit timeline timezone, such as utc, local and so on. local is default<br />`Config Param: TIMELINE_TIMEZONE` [...]
+| [hoodie.table.type](#hoodietabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TYPE` [...]
+| [hoodie.table.version](#hoodietableversion) | ZERO | Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br />`Config Param: VERSION` [...]
+---
+
+## Spark Datasource Configs {#SPARK_DATASOURCE}
+These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
+
+
+### Read Options {#Read-Options}
+Options useful for reading tables via `read.format.option(...)`
+
+
+
+
+
+[**Basic Configs**](#Read-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.read.begin.instanttime](#hoodiedatasourcereadbegininstanttime) | (N/A) | Required when `hoodie.datasource.query.type` is set to `incremental`. Represents the instant time to start incrementally pulling data from. The instanttime here need not necessarily correspond to an instant on the timeline. New data written with an instant_time > BEGIN_INSTANTTIME are fetched out. For e.g: ‘20170901080000’ will get all new data written after Sep 1, 2017 08:00AM. Note that [...]
+| [hoodie.datasource.read.end.instanttime](#hoodiedatasourcereadendinstanttime) | (N/A) | Used when `hoodie.datasource.query.type` is set to `incremental`. Represents the instant time to limit incrementally fetched data to. When not specified latest commit time from timeline is assumed by default. When specified, new data written with an instant_time <= END_INSTANTTIME are fetched out. Point in time type queries make more sense with begin and end instant times specified. Note [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files)<br />`Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: READ_PRE_COMBINE_FIELD` [...]
+---
+
+
+### Write Options {#Write-Options}
+You can pass down any of the WriteClient level configs directly using `options()` or `option(k,v)` methods.
+
+```java
+inputDF.write()
+.format("org.apache.hudi")
+.options(clientOpts) // any of the Hudi client opts can be passed in as well
+.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")
+.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")
+.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")
+.option(HoodieWriteConfig.TABLE_NAME, tableName)
+.mode(SaveMode.Append)
+.save(basePath);
+```
+
+Options useful for writing tables via `write.format.option(...)`
+
+
+
+
+
+[**Basic Configs**](#Write-Options-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD` |
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD` |
+| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING` |
+| [hoodie.datasource.write.operation](#hoodiedatasourcewriteoperation) | upsert | Whether to do upsert, insert or bulk_insert for the write operation. Use bulk_insert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br />`Config Param: OPERATION` |
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD` |
+| [hoodie.datasource.write.table.type](#hoodiedatasourcewritetabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TABLE_TYPE` |
+---
+
+## Flink Sql Configs {#FLINK_SQL}
+These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
+
+
+### Flink Options {#Flink-Options}
+Flink jobs using the SQL can be configured through the options in WITH clause. The actual datasource level configs are listed below.
+
+
+
+
+[**Basic Configs**](#Flink-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name to register to Hive metastore<br /> `Config Param: DATABASE_NAME` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name to register to Hive metastore<br /> `Config Param: TABLE_NAME` [...]
+| [path](#path) | (N/A) | Base path for the target hoodie table. The path would be created if it does not exist, otherwise a Hoodie table expects to be initialized successfully<br /> `Config Param: PATH` [...]
+| [read.end-commit](#readend-commit) | (N/A) | End commit instant for reading, the commit time format should be 'yyyyMMddHHmmss'<br /> `Config Param: READ_END_COMMIT` [...]
+| [read.start-commit](#readstart-commit) | (N/A) | Start commit instant for reading, the commit time format should be 'yyyyMMddHHmmss', by default reading from the latest instant for streaming read<br /> `Config Param: READ_START_COMMIT` [...]
+| [archive.max_commits](#archivemax_commits) | 50 | Max number of commits to keep before archiving older commits into a sequential log, default 50<br /> `Config Param: ARCHIVE_MAX_COMMITS` [...]
+| [archive.min_commits](#archivemin_commits) | 40 | Min number of commits to keep before archiving older commits into a sequential log, default 40<br /> `Config Param: ARCHIVE_MIN_COMMITS` [...]
+| [cdc.enabled](#cdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode<br /> `Config Param: CDC_ENABLED` [...]
+| [cdc.supplemental.logging.mode](#cdcsupplementalloggingmode) | DATA_BEFORE_AFTER | Setting 'op_key_only' persists the 'op' and the record key only, setting 'data_before' persists the additional 'before' image, and setting 'data_before_after' persists the additional 'before' and 'after' images.<br /> `Config Param: SUPPLEMENTAL_LOGGING_MODE` [...]
+| [changelog.enabled](#changelogenabled) | false | Whether to keep all the intermediate changes, we try to keep all the changes of a record when enabled: 1). The sink accept the UPDATE_BEFORE message; 2). The source try to emit every changes of a record. The semantics is best effort because the compaction job would finally merge all changes of a record into one. default false to have UPSERT semantics<br /> [...]
+| [clean.async.enabled](#cleanasyncenabled) | true | Whether to cleanup the old commits immediately on new commits, enabled by default<br /> `Config Param: CLEAN_ASYNC_ENABLED` [...]
+| [clean.retain_commits](#cleanretain_commits) | 30 | Number of commits to retain. So data will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much you can incrementally pull on this table, default 30<br /> `Config Param: CLEAN_RETAIN_COMMITS` [...]
+| [clustering.async.enabled](#clusteringasyncenabled) | false | Async Clustering, default false<br /> `Config Param: CLUSTERING_ASYNC_ENABLED` [...]
+| [clustering.plan.strategy.small.file.limit](#clusteringplanstrategysmallfilelimit) | 600 | Files smaller than the size specified here are candidates for clustering, default 600 MB<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_SMALL_FILE_LIMIT` [...]
+| [clustering.plan.strategy.target.file.max.bytes](#clusteringplanstrategytargetfilemaxbytes) | 1073741824 | Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups, default 1 GB<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_TARGET_FILE_MAX_BYTES` [...]
+| [compaction.async.enabled](#compactionasyncenabled) | true | Async Compaction, enabled by default for MOR<br /> `Config Param: COMPACTION_ASYNC_ENABLED` [...]
+| [compaction.delta_commits](#compactiondelta_commits) | 5 | Max delta commits needed to trigger compaction, default 5 commits<br /> `Config Param: COMPACTION_DELTA_COMMITS` [...]
+| [hive_sync.enabled](#hive_syncenabled) | false | Asynchronously sync Hive meta to HMS, default false<br /> `Config Param: HIVE_SYNC_ENABLED` [...]
+| [hive_sync.jdbc_url](#hive_syncjdbc_url) | jdbc:hive2://localhost:10000 | Jdbc URL for hive sync, default 'jdbc:hive2://localhost:10000'<br /> `Config Param: HIVE_SYNC_JDBC_URL` [...]
+| [hive_sync.metastore.uris](#hive_syncmetastoreuris) | | Metastore uris for hive sync, default ''<br /> `Config Param: HIVE_SYNC_METASTORE_URIS` [...]
+| [hive_sync.mode](#hive_syncmode) | HMS | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql, default 'hms'<br /> `Config Param: HIVE_SYNC_MODE` [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Decides how data files need to be read, in 1) Snapshot mode (obtain latest view, based on row & columnar data); 2) incremental mode (new data since an instantTime); 3) Read Optimized mode (obtain latest view, based on columnar data) .Default: snapshot<br /> `Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br /> `Config Param: HIVE_STYLE_PARTITIONING` [...]
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | | Partition path field. Value to be used at the `partitionPath` component of `HoodieKey`. Actual value obtained by invoking .toString(), default ''<br /> `Config Param: PARTITION_PATH_FIELD` [...]
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | uuid | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br /> `Config Param: RECORD_KEY_FIELD` [...]
+| [index.type](#indextype) | FLINK_STATE | Index type of Flink write job, default is using state backed index.<br /> `Config Param: INDEX_TYPE` [...]
+| [metadata.compaction.delta_commits](#metadatacompactiondelta_commits) | 10 | Max delta commits for metadata table to trigger compaction, default 10<br /> `Config Param: METADATA_COMPACTION_DELTA_COMMITS` [...]
+| [metadata.enabled](#metadataenabled) | false | Enable the internal metadata table which serves table metadata like level file listings, default disabled<br /> `Config Param: METADATA_ENABLED` [...]
+| [precombine.field](#precombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br /> `Config Param: PRECOMBINE_FIELD` [...]
+| [read.streaming.enabled](#readstreamingenabled) | false | Whether to read as streaming source, default false<br /> `Config Param: READ_AS_STREAMING` [...]
+| [table.type](#tabletype) | COPY_ON_WRITE | Type of table to write. COPY_ON_WRITE (or) MERGE_ON_READ<br /> `Config Param: TABLE_TYPE` [...]
+| [write.operation](#writeoperation) | upsert | The write operation, that this write should do<br /> `Config Param: OPERATION` [...]
+| [write.parquet.max.file.size](#writeparquetmaxfilesize) | 120 | Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /> `Config Param: WRITE_PARQUET_MAX_FILE_SIZE` [...]
+---
+
+## Write Client Configs {#WRITE_CLIENT}
+Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.
+
+
+### Common Configurations {#Common-Configurations}
+The following set of configurations are common across Hudi.
+
+
+
+
+[**Basic Configs**](#Common-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` |
+---
+
+
+### Metadata Configs {#Metadata-Configs}
+Configurations used by the Hudi Metadata Table. This table maintains the metadata about a given Hudi table (e.g file listings) to avoid overhead of accessing cloud storage, during queries.
+
+
+
+
+[**Basic Configs**](#Metadata-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.metadata.enable](#hoodiemetadataenable) | true | Enable the internal metadata table which serves table metadata like level file listings<br />`Config Param: ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.metadata.index.bloom.filter.enable](#hoodiemetadataindexbloomfilterenable) | false | Enable indexing bloom filters of user data files under metadata table. When enabled, metadata table will have a partition to store the bloom filter index and will be used during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_BLOOM_FILTER`<br />`Since Version: 0.11.0` |
+| [hoodie.metadata.index.column.stats.enable](#hoodiemetadataindexcolumnstatsenable) | false | Enable indexing column ranges of user data files under metadata table key lookups. When enabled, metadata table will have a partition to store the column ranges and will be used for pruning files during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_COLUMN_STATS`<br />`Since Version: 0.11.0` |
+---
+
+
+### Storage Configs {#Storage-Configs}
+Configurations that control aspects around writing, sizing, reading base and log files.
+
+
+
+
+[**Basic Configs**](#Storage-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.parquet.compression.codec](#hoodieparquetcompressioncodec) | gzip | Compression Codec for parquet files<br />`Config Param: PARQUET_COMPRESSION_CODEC_NAME` |
+| [hoodie.parquet.max.file.size](#hoodieparquetmaxfilesize) | 125829120 | Target size in bytes for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br />`Config Param: PARQUET_MAX_FILE_SIZE` |
+---
+
+
+### Archival Configs {#Archival-Configs}
+Configurations that control archival.
+
+
+
+
+[**Basic Configs**](#Archival-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.keep.max.commits](#hoodiekeepmaxcommits) | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. <br />`Config Param: MAX_COMMITS_TO_KEEP` |
+| [hoodie.keep.min.commits](#hoodiekeepmincommits) | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline.<br />`Config Param: MIN_COMMITS_TO_KEEP` |
+---
+
+
+### Bootstrap Configs {#Bootstrap-Configs}
+Configurations that control how you want to bootstrap your existing tables for the first time into hudi. The bootstrap operation can flexibly avoid copying data over before you can use Hudi and support running the existing writers and new hudi writers in parallel, to validate the migration.
+
+
+
+
+[**Basic Configs**](#Bootstrap-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BASE_PATH`<br />`Since Version: 0.6.0` |
+---
+
+
+### Clean Configs {#Clean-Configs}
+Cleaning (reclamation of older/unused file groups/slices).
+
+
+
+
+[**Basic Configs**](#Clean-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.clean.async](#hoodiecleanasync) | false | Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_CLEAN` |
+| [hoodie.cleaner.commits.retained](#hoodiecleanercommitsretained) | 10 | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br />`Config Param: CLEANER_COMMITS_RETAINED` |
+---
+
+
+### Clustering Configs {#Clustering-Configs}
+Configurations that control the clustering table service in hudi, which optimizes the storage layout for better query performance by sorting and sizing data files.
+
+
+
+
+[**Basic Configs**](#Clustering-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.plan.strategy.small.file.limit](#hoodieclusteringplanstrategysmallfilelimit) | 314572800 | Files smaller than the size in bytes specified here are candidates for clustering<br />`Config Param: PLAN_STRATEGY_SMALL_FILE_LIMIT`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.plan.strategy.target.file.max.bytes](#hoodieclusteringplanstrategytargetfilemaxbytes) | 1073741824 | Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups<br />`Config Param: PLAN_STRATEGY_TARGET_FILE_MAX_BYTES`<br />`Since Version: 0.7.0` |
+---
+
+
+### Compaction Configs {#Compaction-Configs}
+Configurations that control compaction (merging of log files onto a new base files).
+
+
+
+
+[**Basic Configs**](#Compaction-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.compact.inline](#hoodiecompactinline) | false | When set to true, compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br />`Config Param: INLINE_COMPACT` |
+| [hoodie.compact.inline.max.delta.commits](#hoodiecompactinlinemaxdeltacommits) | 5 | Number of delta commits after the last compaction, before scheduling of a new compaction is attempted. This config takes effect only for the compaction triggering strategy based on the number of commits, i.e., NUM_COMMITS, NUM_COMMITS_AFTER_LAST_REQUEST, NUM_AND_TIME, and NUM_OR_TIME.<br />`Config Param: INLINE_COMPACT_NUM_DELTA_COMMITS` |
+---
+
+
+### Error table Configs {#Error-table-Configs}
+Configurations that are required for Error table configs
+
+
+
+
+[**Basic Configs**](#Error-table-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.errortable.base.path](#hoodieerrortablebasepath) | (N/A) | Base path for error table under which all error records would be stored.<br />`Config Param: ERROR_TABLE_BASE_PATH` |
+| [hoodie.errortable.target.table.name](#hoodieerrortabletargettablename) | (N/A) | Table name to be used for the error table<br />`Config Param: ERROR_TARGET_TABLE` |
+| [hoodie.errortable.write.class](#hoodieerrortablewriteclass) | (N/A) | Class which handles the error table writes. This config is used to configure a custom implementation for Error Table Writer. Specify the full class name of the custom error table writer as a value for this config<br />`Config Param: ERROR_TABLE_WRITE_CLASS` |
+| [hoodie.errortable.enable](#hoodieerrortableenable) | false | Config to enable error table. If the config is enabled, all the records with processing error in DeltaStreamer are transferred to error table.<br />`Config Param: ERROR_TABLE_ENABLED` |
+| [hoodie.errortable.insert.shuffle.parallelism](#hoodieerrortableinsertshuffleparallelism) | 200 | Config to set insert shuffle parallelism. The config is similar to hoodie.insert.shuffle.parallelism config but applies to the error table.<br />`Config Param: ERROR_TABLE_INSERT_PARALLELISM_VALUE` |
+| [hoodie.errortable.upsert.shuffle.parallelism](#hoodieerrortableupsertshuffleparallelism) | 200 | Config to set upsert shuffle parallelism. The config is similar to hoodie.upsert.shuffle.parallelism config but applies to the error table.<br />`Config Param: ERROR_TABLE_UPSERT_PARALLELISM_VALUE` |
+| [hoodie.errortable.validate.recordcreation.enable](#hoodieerrortablevalidaterecordcreationenable) | true | Records that fail to be created due to keygeneration failure or other issues will be sent to the Error Table<br />`Config Param: ERROR_ENABLE_VALIDATE_RECORD_CREATION`<br />`Since Version: 0.15.0` |
+| [hoodie.errortable.validate.targetschema.enable](#hoodieerrortablevalidatetargetschemaenable) | false | Records with schema mismatch with Target Schema are sent to Error Table.<br />`Config Param: ERROR_ENABLE_VALIDATE_TARGET_SCHEMA` |
+| [hoodie.errortable.write.failure.strategy](#hoodieerrortablewritefailurestrategy) | ROLLBACK_COMMIT | The config specifies the failure strategy if error table write fails. Use one of - [ROLLBACK_COMMIT (Rollback the corresponding base table write commit for which the error events were triggered) , LOG_ERROR (Error is logged but the base table write succeeds) ]<br />`Config Param: ERROR_TABLE_WRITE_FAILURE_STRATEGY` |
+---
+
+
+### Write Configurations {#Write-Configurations}
+Configurations that control write behavior on Hudi tables. These can be directly passed down from even higher level frameworks (e.g Spark datasources, Flink sink) and utilities (e.g Hudi Streamer).
+
+
+
+
+[**Basic Configs**](#Write-Configurations-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br />`Config Param: TBL_NAME` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD_NAME` [...]
+| [hoodie.write.concurrency.mode](#hoodiewriteconcurrencymode) | SINGLE_WRITER | org.apache.hudi.common.model.WriteConcurrencyMode: Concurrency modes for write operations. SINGLE_WRITER(default): Only one active writer to the table. Maximizes throughput. OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table with lazy conflict resolution using locks. This means that only one writer succeeds if multiple writers write to the same file group. [...]
+---
+
+
+### Lock Configs {#LOCK}
+Configurations that control locking mechanisms required for concurrency control between writers to a Hudi table. Concurrency between Hudi's own table services are auto managed internally.
+
+
+#### Common Lock Configurations {#Common-Lock-Configurations}
+
+
+
+
+
+[**Basic Configs**](#Common-Lock-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.write.lock.heartbeat_interval_ms](#hoodiewritelockheartbeat_interval_ms) | 60000 | Heartbeat interval in ms, to send a heartbeat to indicate that hive client holding locks.<br />`Config Param: LOCK_HEARTBEAT_INTERVAL_MS`<br />`Since Version: 0.15.0` |
+---
+
+
+### Key Generator Configs {#KEY_GENERATOR}
+Hudi maintains keys (record key + partition path) for uniquely identifying a particular record. These configs allow developers to setup the Key generator class that extracts these out of incoming records.
+
+
+#### Key Generator Options {#Key-Generator-Options}
+
+
+
+
+
+[**Basic Configs**](#Key-Generator-Options-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD_NAME` |
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` |
+---
+
+
+### Index Configs {#INDEX}
+Configurations that control indexing behavior, which tags incoming records as either inserts or updates to older records.
+
+
+#### Common Index Configs {#Common-Index-Configs}
+
+
+
+
+
+[**Basic Configs**](#Common-Index-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.index.type](#hoodieindextype) | (N/A) | org.apache.hudi.index.HoodieIndex$IndexType: Determines how input records are indexed, i.e., looked up based on the key for the location in the existing table. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java engines. HBASE: uses an external managed Apache HBase table to store record key to location mapping. HBase index is a global index, enforcing key uniqueness across all partitions in the table. INMEMORY: Use [...]
+---
+
+## Metastore and Catalog Sync Configs {#META_SYNC}
+Configurations used by the Hudi to sync metadata to external metastores and catalogs.
+
+
+### Common Metadata Sync Configs {#Common-Metadata-Sync-Configs}
+
+
+
+
+
+[**Basic Configs**](#Common-Metadata-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+
+### Glue catalog sync based client Configurations {#Glue-catalog-sync-based-client-Configurations}
+Configs that control Glue catalog sync based client.
+
+
+
+
+[**Basic Configs**](#Glue-catalog-sync-based-client-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.glue.partition_index_fields](#hoodiedatasourcemetasyncgluepartition_index_fields) | | Specify the partitions fields to index on aws glue. Separate the fields by semicolon. By default, when the feature is enabled, all the partition will be indexed. You can create up to three indexes, separate them by comma. Eg: col1;col2;col3,col2,col3<br />`Config Param: META_SYNC_PARTITION_INDEX_FIELDS`<br />`Since Version: 0.15.0` |
+| [hoodie.datasource.meta.sync.glue.partition_index_fields.enable](#hoodiedatasourcemetasyncgluepartition_index_fieldsenable) | false | Enable aws glue partition index feature, to speedup partition based query pattern<br />`Config Param: META_SYNC_PARTITION_INDEX_FIELDS_ENABLE`<br />`Since Version: 0.15.0` |
+---
+
+
+### BigQuery Sync Configs {#BigQuery-Sync-Configs}
+Configurations used by the Hudi to sync metadata to Google BigQuery.
+
+
+
+
+[**Basic Configs**](#BigQuery-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+
+### Hive Sync Configs {#Hive-Sync-Configs}
+Configurations used by the Hudi to sync metadata to Hive Metastore.
+
+
+
+
+[**Basic Configs**](#Hive-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+
+### Global Hive Sync Configs {#Global-Hive-Sync-Configs}
+Global replication configurations used by the Hudi to sync metadata to Hive Metastore.
+
+
+
+
+[**Basic Configs**](#Global-Hive-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+
+### DataHub Sync Configs {#DataHub-Sync-Configs}
+Configurations used by the Hudi to sync metadata to DataHub.
+
+
+
+
+[**Basic Configs**](#DataHub-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+---
+
+## Metrics Configs {#METRICS}
+These set of configs are used to enable monitoring and reporting of key Hudi stats and metrics.
+
+
+### Metrics Configurations {#Metrics-Configurations}
+Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean, rollback etc. The following sections list the supported reporters.
+
+
+
+
+[**Basic Configs**](#Metrics-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.on](#hoodiemetricson) | false | Turn on/off metrics reporting. off by default.<br />`Config Param: TURN_METRICS_ON`<br />`Since Version: 0.5.0` |
+| [hoodie.metrics.reporter.type](#hoodiemetricsreportertype) | GRAPHITE | Type of metrics reporter.<br />`Config Param: METRICS_REPORTER_TYPE_VALUE`<br />`Since Version: 0.5.0` |
+| [hoodie.metricscompaction.log.blocks.on](#hoodiemetricscompactionlogblockson) | false | Turn on/off metrics reporting for log blocks with compaction commit. off by default.<br />`Config Param: TURN_METRICS_COMPACTION_LOG_BLOCKS_ON`<br />`Since Version: 0.14.0` |
+---
+
+
+### Metrics Configurations for M3 {#Metrics-Configurations-for-M3}
+Enables reporting on Hudi metrics using M3. Hudi publishes metrics on every commit, clean, rollback etc.
+
+
+
+
+[**Basic Configs**](#Metrics-Configurations-for-M3-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.m3.env](#hoodiemetricsm3env) | production | M3 tag to label the environment (defaults to 'production'), applied to all metrics.<br />`Config Param: M3_ENV`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.host](#hoodiemetricsm3host) | localhost | M3 host to connect to.<br />`Config Param: M3_SERVER_HOST_NAME`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.port](#hoodiemetricsm3port) | 9052 | M3 port to connect to.<br />`Config Param: M3_SERVER_PORT_NUM`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.service](#hoodiemetricsm3service) | hoodie | M3 tag to label the service name (defaults to 'hoodie'), applied to all metrics.<br />`Config Param: M3_SERVICE`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.tags](#hoodiemetricsm3tags) | | Optional M3 tags applied to all metrics.<br />`Config Param: M3_TAGS`<br />`Since Version: 0.15.0` |
+---
+
+## Kafka Connect Configs {#KAFKA_CONNECT}
+These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables
+
+
+### Kafka Sink Connect Configurations {#Kafka-Sink-Connect-Configurations}
+Configurations for Kafka Connect Sink Connector for Hudi.
+
+
+
+
+[**Basic Configs**](#Kafka-Sink-Connect-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------- | --------------- | ----------------------------------------------------------------------------------------- |
+| [bootstrap.servers](#bootstrapservers) | localhost:9092 | The bootstrap servers for the Kafka Cluster.<br />`Config Param: KAFKA_BOOTSTRAP_SERVERS` |
+---
+
+## Hudi Streamer Configs {#HUDI_STREAMER}
+These set of configs are used for Hudi Streamer utility which provides the way to ingest from different sources such as DFS or Kafka.
+
+
+### Hudi Streamer Configs {#Hudi-Streamer-Configs}
+
+
+
+
+
+[**Basic Configs**](#Hudi-Streamer-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.kafka.topic](#hoodiestreamersourcekafkatopic) | (N/A) | Kafka topic name. The config is specific to HoodieMultiTableStreamer<br />`Config Param: KAFKA_TOPIC` |
+---
+
+
+### Hudi Streamer SQL Transformer Configs {#Hudi-Streamer-SQL-Transformer-Configs}
+Configurations controlling the behavior of SQL transformer in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#Hudi-Streamer-SQL-Transformer-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.transformer.sql](#hoodiestreamertransformersql) | (N/A) | SQL Query to be executed during write<br />`Config Param: TRANSFORMER_SQL` |
+| [hoodie.streamer.transformer.sql.file](#hoodiestreamertransformersqlfile) | (N/A) | File with a SQL script to be executed during write<br />`Config Param: TRANSFORMER_SQL_FILE` |
+---
+
+
+### Hudi Streamer Source Configs {#DELTA_STREAMER_SOURCE}
+Configurations controlling the behavior of reading source data.
+
+
+#### DFS Path Selector Configs {#DFS-Path-Selector-Configs}
+Configurations controlling the behavior of path selector for DFS source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#DFS-Path-Selector-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------- | ------- | ------------------------------------------------------------------- |
+| [hoodie.streamer.source.dfs.root](#hoodiestreamersourcedfsroot) | (N/A) | Root path of the source on DFS<br />`Config Param: ROOT_INPUT_PATH` |
+---
+
+
+#### Hudi Incremental Source Configs {#Hudi-Incremental-Source-Configs}
+Configurations controlling the behavior of incremental pulling from a Hudi table as a source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#Hudi-Incremental-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------- |
+| [hoodie.streamer.source.hoodieincr.path](#hoodiestreamersourcehoodieincrpath) | (N/A) | Base-path for the source Hudi table<br />`Config Param: HOODIE_SRC_BASE_PATH` |
+---
+
+
+#### Kafka Source Configs {#Kafka-Source-Configs}
+Configurations controlling the behavior of Kafka source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#Kafka-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.kafka.topic](#hoodiestreamersourcekafkatopic) | (N/A) | Kafka topic name.<br />`Config Param: KAFKA_TOPIC_NAME` |
+| [hoodie.streamer.source.kafka.proto.value.deserializer.class](#hoodiestreamersourcekafkaprotovaluedeserializerclass) | org.apache.kafka.common.serialization.ByteArrayDeserializer | Kafka Proto Payload Deserializer Class<br />`Config Param: KAFKA_PROTO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.15.0` |
+---
+
+
+#### Pulsar Source Configs {#Pulsar-Source-Configs}
+Configurations controlling the behavior of Pulsar source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#Pulsar-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.pulsar.topic](#hoodiestreamersourcepulsartopic) | (N/A) | Name of the target Pulsar topic to source data from<br />`Config Param: PULSAR_SOURCE_TOPIC_NAME` |
+| [hoodie.streamer.source.pulsar.endpoint.admin.url](#hoodiestreamersourcepulsarendpointadminurl) | http://localhost:8080 | URL of the target Pulsar endpoint (of the form 'pulsar://host:port'<br />`Config Param: PULSAR_SOURCE_ADMIN_ENDPOINT_URL` |
+| [hoodie.streamer.source.pulsar.endpoint.service.url](#hoodiestreamersourcepulsarendpointserviceurl) | pulsar://localhost:6650 | URL of the target Pulsar endpoint (of the form 'pulsar://host:port'<br />`Config Param: PULSAR_SOURCE_SERVICE_ENDPOINT_URL` |
+---
+
+
+#### S3 Source Configs {#S3-Source-Configs}
+Configurations controlling the behavior of S3 source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#S3-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------- |
+| [hoodie.streamer.s3.source.queue.url](#hoodiestreamers3sourcequeueurl) | (N/A) | Queue url for cloud object events<br />`Config Param: S3_SOURCE_QUEUE_URL` |
+---
+
+
+#### File-based SQL Source Configs {#File-based-SQL-Source-Configs}
+Configurations controlling the behavior of File-based SQL Source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#File-based-SQL-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.sql.file](#hoodiestreamersourcesqlfile) | (N/A) | SQL file path containing the SQL query to read source data.<br />`Config Param: SOURCE_SQL_FILE`<br />`Since Version: 0.14.0` |
+---
+
+
+#### SQL Source Configs {#SQL-Source-Configs}
+Configurations controlling the behavior of SQL source in Hudi Streamer.
+
+
+
+
+[**Basic Configs**](#SQL-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------ | ------- | ------------------------------------------------------------------- |
+| [hoodie.streamer.source.sql.sql.query](#hoodiestreamersourcesqlsqlquery) | (N/A) | SQL query for fetching source data.<br />`Config Param: SOURCE_SQL` |
+---
+
+
+### Hudi Streamer Schema Provider Configs {#SCHEMA_PROVIDER}
+Configurations that control the schema provider for Hudi Streamer.
+
+
+#### Hudi Streamer Schema Provider Configs {#Hudi-Streamer-Schema-Provider-Configs}
+
+
+
+
+
+[**Basic Configs**](#Hudi-Streamer-Schema-Provider-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.registry.targetUrl](#hoodiestreamerschemaproviderregistrytargetUrl) | (N/A) | The schema of the target you are writing to e.g. https://foo:bar@schemaregistry.org<br />`Config Param: TARGET_SCHEMA_REGISTRY_URL` |
+| [hoodie.streamer.schemaprovider.registry.url](#hoodiestreamerschemaproviderregistryurl) | (N/A) | The schema of the source you are reading from e.g. https://foo:bar@schemaregistry.org<br />`Config Param: SRC_SCHEMA_REGISTRY_URL` |
+---
+
+
+#### File-based Schema Provider Configs {#File-based-Schema-Provider-Configs}
+Configurations for file-based schema provider.
+
+
+
+
+[**Basic Configs**](#File-based-Schema-Provider-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.source.schema.file](#hoodiestreamerschemaprovidersourceschemafile) | (N/A) | The schema of the source you are reading from<br />`Config Param: SOURCE_SCHEMA_FILE` |
+| [hoodie.streamer.schemaprovider.target.schema.file](#hoodiestreamerschemaprovidertargetschemafile) | (N/A) | The schema of the target you are writing to<br />`Config Param: TARGET_SCHEMA_FILE` |
+---
+
diff --git a/website/versioned_docs/version-0.15.1/bos_hoodie.md b/website/versioned_docs/version-0.15.1/bos_hoodie.md
new file mode 100644
index 000000000000..2a6cde81c868
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/bos_hoodie.md
@@ -0,0 +1,57 @@
+---
+title: Baidu Cloud
+keywords: [ hudi, hive, baidu, bos, spark, presto]
+summary: In this page, we go over how to configure Hudi with bos filesystem.
+last_modified_at: 2021-06-09T11:38:24-10:00
+---
+In this page, we explain how to get your Hudi job to store into Baidu BOS.
+
+## Baidu BOS configs
+
+There are two configurations required for Hudi-BOS compatibility:
+
+- Adding Baidu BOS Credentials for Hudi
+- Adding required Jars to classpath
+
+### Baidu BOS Credentials
+
+Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the `fs.defaultFS` with your BOS bucket name, replace `fs.bos.endpoint` with your bos endpoint, replace `fs.bos.access.key` with your bos key, replace `fs.bos.secret.access.key` with your bos secret key. Hudi should be able to read/write from the bucket.
+
+```xml
+<property>
+ <name>fs.defaultFS</name>
+ <value>bos://bucketname/</value>
+</property>
+
+<property>
+ <name>fs.bos.endpoint</name>
+ <value>bos-endpoint-address</value>
+ <description>Baidu bos endpoint to connect to,for example : http://bj.bcebos.com</description>
+</property>
+
+<property>
+ <name>fs.bos.access.key</name>
+ <value>bos-key</value>
+ <description>Baidu access key</description>
+</property>
+
+<property>
+ <name>fs.bos.secret.access.key</name>
+ <value>bos-secret-key</value>
+ <description>Baidu secret key.</description>
+</property>
+
+<property>
+ <name>fs.bos.impl</name>
+ <value>org.apache.hadoop.fs.bos.BaiduBosFileSystem</value>
+</property>
+```
+
+### Baidu bos Libs
+
+Baidu hadoop libraries jars to add to our classpath
+
+- com.baidubce:bce-java-sdk:0.10.165
+- bos-hdfs-sdk-1.0.2-community.jar
+
+You can download the bos-hdfs-sdk jar from [here](https://sdk.bce.baidu.com/console-sdk/bos-hdfs-sdk-1.0.2-community.jar.zip) , and then unzip it.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.15.1/cli.md b/website/versioned_docs/version-0.15.1/cli.md
new file mode 100644
index 000000000000..77bc50102bba
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/cli.md
@@ -0,0 +1,757 @@
+---
+title: CLI
+keywords: [hudi, cli]
+last_modified_at: 2021-08-18T15:59:57-04:00
+---
+
+### Local set up
+Once hudi has been built, the shell can be fired by via `cd hudi-cli && ./hudi-cli.sh`.
+
+### Hudi CLI Bundle setup
+In release `0.13.0` we have now added another way of launching the `hudi cli`, which is using the `hudi-cli-bundle`. (Note this is only supported for Spark3,
+for Spark2 please see the above Local setup section)
+
+There are a couple of requirements when using this approach such as having `spark` installed locally on your machine.
+It is required to use a spark distribution with hadoop dependencies packaged such as `spark-3.3.1-bin-hadoop2.tgz` from https://archive.apache.org/dist/spark/.
+We also recommend you set an env variable `$SPARK_HOME` to the path of where spark is installed on your machine.
+One important thing to note is that the `hudi-spark-bundle` should also be present when using the `hudi-cli-bundle`.
+To provide the locations of these bundle jars you can set them in your shell like so:
+`export CLI_BUNDLE_JAR=<path-to-cli-bundle-jar-to-use>` , `export SPARK_BUNDLE_JAR=<path-to-spark-bundle-jar-to-use>`.
+
+For steps see below if you are not compiling the project and downloading the jars:
+
+1. Create an empty folder as a new directory
+2. Copy the hudi-cli-bundle jars and hudi-spark*-bundle jars to this directory
+3. Copy the following script and folder to this directory
+```
+packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+packaging/hudi-cli-bundle/conf . the `conf` folder should be in this directory.
+```
+
+4. Start Hudi CLI shell with environment variables set
+```
+export SPARK_HOME=<spark-home-folder>
+export CLI_BUNDLE_JAR=<cli-bundle-jar-to-use>
+export SPARK_BUNDLE_JAR=<spark-bundle-jar-to-use>
+
+./hudi-cli-with-bundle.sh
+
+```
+
+### Base path
+A hudi table resides on DFS, in a location referred to as the `basePath` and
+we would need this location in order to connect to a Hudi table. Hudi library effectively manages this table internally, using `.hoodie` subfolder to track all metadata.
+
+
+
+
+### Using Hudi-cli in S3
+If you are using hudi that comes packaged with AWS EMR, you can find instructions to use hudi-cli [here](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-cli.html).
+If you are not using EMR, or would like to use latest hudi-cli from master, you can follow the below steps to access S3 dataset in your local environment (laptop).
+
+Build Hudi with corresponding Spark version, for eg, -Dspark3.1.x
+
+Set the following environment variables.
+```
+export AWS_REGION=us-east-2
+export AWS_ACCESS_KEY_ID=<key_id>
+export AWS_SECRET_ACCESS_KEY=<secret_key>
+export SPARK_HOME=<spark_home>
+```
+Ensure you set the SPARK_HOME to your local spark home compatible to compiled hudi spark version above.
+
+Apart from these, we might need to add aws jars to class path so that accessing S3 is feasible from local.
+We need two jars, namely, aws-java-sdk-bundle jar and hadoop-aws jar which you can find online.
+For eg:
+```
+wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.0/hadoop-aws-3.2.0.jar -o /lib/spark-3.2.0-bin-hadoop3.2/jars/hadoop-aws-3.2.0.jar
+wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.375/aws-java-sdk-bundle-1.11.375.jar -o /lib/spark-3.2.0-bin-hadoop3.2/jars/aws-java-sdk-bundle-1.11.375.jar
+```
+
+#### Note: These AWS jar versions below are specific to Spark 3.2.0
+```
+export CLIENT_JAR=/lib/spark-3.2.0-bin-hadoop3.2/jars/aws-java-sdk-bundle-1.12.48.jar:/lib/spark-3.2.0-bin-hadoop3.2/jars/hadoop-aws-3.3.1.jar
+```
+Once these are set, you are good to launch hudi-cli and access S3 dataset.
+```
+./hudi-cli/hudi-cli.sh
+```
+### Using hudi-cli on Google Dataproc
+[Dataproc](https://cloud.google.com/dataproc) is Google's managed service for running Apache Hadoop, Apache Spark,
+Apache Flink, Presto and many other frameworks, including Hudi. If you want to run the Hudi CLI on a Dataproc node
+which has not been launched with Hudi support enabled, you can use the steps below:
+
+These steps use Hudi version 0.13.0. If you want to use a different version you will have to edit the below commands
+appropriately:
+1. Once you've started the Dataproc cluster, you can ssh into it as follows:
+```
+$ gcloud compute ssh --zone "YOUR_ZONE" "HOSTNAME_OF_MASTER_NODE" --project "YOUR_PROJECT"
+```
+
+2. Download the Hudi CLI bundle
+```
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-cli-bundle_2.12/0.13.0/hudi-cli-bundle_2.12-0.13.0.jar
+```
+
+3. Download the Hudi Spark bundle
+```
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark-bundle_2.12/0.13.0/hudi-spark-bundle_2.12-0.13.0.jar
+```
+
+4. Download the shell script that launches Hudi CLI bundle
+```
+wget https://raw.githubusercontent.com/apache/hudi/release-0.13.0/packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+```
+
+5. Launch Hudi CLI bundle with appropriate environment variables as follows:
+```
+CLIENT_JAR=$DATAPROC_DIR/lib/gcs-connector.jar CLI_BUNDLE_JAR=hudi-cli-bundle_2.12-0.13.0.jar SPARK_BUNDLE_JAR=hudi-spark-bundle_2.12-0.13.0.jar ./hudi-cli-with-bundle.sh
+```
+
+6. hudi->connect --path gs://path_to_some_table
+Metadata for table some_table loaded
+
+7. hudi:some_table->commits show --limit 5
+This command should show the recent commits, if the above steps work correctly.
+
+## Connect to a Kerberized cluster
+
+Before connecting to a Kerberized cluster, you can use **kerberos kinit** command. Following is the usage of this command.
+
+```shell
+hudi->help kerberos kinit
+NAME
+ kerberos kinit - Perform Kerberos authentication
+
+SYNOPSIS
+ kerberos kinit --krb5conf String [--principal String] [--keytab String]
+
+OPTIONS
+ --krb5conf String
+ Path to krb5.conf
+ [Optional, default = /etc/krb5.conf]
+
+ --principal String
+ Kerberos principal
+ [Mandatory]
+
+ --keytab String
+ Path to keytab
+ [Mandatory]
+```
+
+For example:
+
+```shell
+hudi->kerberos kinit --principal user/host@DOMAIN --keytab /etc/security/keytabs/user.keytab
+Perform Kerberos authentication
+Parameters:
+--krb5conf: /etc/krb5.conf
+--principal: user/host@DOMAIN
+--keytab: /etc/security/keytabs/user.keytab
+Kerberos current user: user/host@DOMAIN (auth:KERBEROS)
+Kerberos login user: user/host@DOMAIN (auth:KERBEROS)
+Kerberos authentication success
+```
+
+If you see "Kerberos authentication success" in the command output, it means Kerberos authentication has been successful.
+
+
+## Using hudi-cli
+
+To initialize a hudi table, use the following command.
+
+```java
+===================================================================
+* ___ ___ *
+* /\__\ ___ /\ \ ___ *
+* / / / /\__\ / \ \ /\ \ *
+* / /__/ / / / / /\ \ \ \ \ \ *
+* / \ \ ___ / / / / / \ \__\ / \__\ *
+* / /\ \ /\__\ / /__/ ___ / /__/ \ |__| / /\/__/ *
+* \/ \ \/ / / \ \ \ /\__\ \ \ \ / / / /\/ / / *
+* \ / / \ \ / / / \ \ / / / \ /__/ *
+* / / / \ \/ / / \ \/ / / \ \__\ *
+* / / / \ / / \ / / \/__/ *
+* \/__/ \/__/ \/__/ Apache Hudi CLI *
+* *
+===================================================================
+
+hudi->create --path /user/hive/warehouse/table1 --tableName hoodie_table_1 --tableType COPY_ON_WRITE
+.....
+```
+
+To see the description of hudi table, use the command:
+
+```java
+hudi:hoodie_table_1->desc
+18/09/06 15:57:19 INFO timeline.HoodieActiveTimeline: Loaded instants []
+ _________________________________________________________
+ | Property | Value |
+ |========================================================|
+ | basePath | ... |
+ | metaPath | ... |
+ | fileSystem | hdfs |
+ | hoodie.table.name | hoodie_table_1 |
+ | hoodie.table.type | COPY_ON_WRITE |
+ | hoodie.archivelog.folder| |
+```
+
+Following is a sample command to connect to a Hudi table contains uber trips.
+
+```java
+hudi:trips->connect --path /app/uber/trips
+
+16/10/05 23:20:37 INFO model.HoodieTableMetadata: All commits :HoodieCommits{commitList=[20161002045850, 20161002052915, 20161002055918, 20161002065317, 20161002075932, 20161002082904, 20161002085949, 20161002092936, 20161002105903, 20161002112938, 20161002123005, 20161002133002, 20161002155940, 20161002165924, 20161002172907, 20161002175905, 20161002190016, 20161002192954, 20161002195925, 20161002205935, 20161002215928, 20161002222938, 20161002225915, 20161002232906, 20161003003028, 201 [...]
+Metadata for table trips loaded
+```
+
+Once connected to the table, a lot of other commands become available. The shell has contextual autocomplete help (press TAB) and below is a list of all commands, few of which are reviewed in this section
+
+```shell
+hudi:trips->help
+* ! - Allows execution of operating system (OS) commands
+* // - Inline comment markers (start of line only)
+* ; - Inline comment markers (start of line only)
+* bootstrap index showmapping - Show bootstrap index mapping
+* bootstrap index showpartitions - Show bootstrap indexed partitions
+* bootstrap run - Run a bootstrap action for current Hudi table
+* clean showpartitions - Show partition level details of a clean
+* cleans refresh - Refresh table metadata
+* cleans run - run clean
+* cleans show - Show the cleans
+* clear - Clears the console
+* cls - Clears the console
+* clustering run - Run Clustering
+* clustering schedule - Schedule Clustering
+* clustering scheduleAndExecute - Run Clustering. Make a cluster plan first and execute that plan immediately
+* commit rollback - Rollback a commit
+* commits compare - Compare commits with another Hoodie table
+* commit show_write_stats - Show write stats of a commit
+* commit showfiles - Show file level details of a commit
+* commit showpartitions - Show partition level details of a commit
+* commits refresh - Refresh table metadata
+* commits show - Show the commits
+* commits showarchived - Show the archived commits
+* commits sync - Sync commits with another Hoodie table
+* compaction repair - Renames the files to make them consistent with the timeline as dictated by Hoodie metadata. Use when compaction unschedule fails partially.
+* compaction run - Run Compaction for given instant time
+* compaction schedule - Schedule Compaction
+* compaction scheduleAndExecute - Schedule compaction plan and execute this plan
+* compaction show - Shows compaction details for a specific compaction instant
+* compaction showarchived - Shows compaction details for a specific compaction instant
+* compactions show all - Shows all compactions that are in active timeline
+* compactions showarchived - Shows compaction details for specified time window
+* compaction unschedule - Unschedule Compaction
+* compaction unscheduleFileId - UnSchedule Compaction for a fileId
+* compaction validate - Validate Compaction
+* connect - Connect to a hoodie table
+* create - Create a hoodie table if not present
+* date - Displays the local date and time
+* desc - Describe Hoodie Table properties
+* downgrade table - Downgrades a table
+* exit - Exits the shell
+* export instants - Export Instants and their metadata from the Timeline
+* fetch table schema - Fetches latest table schema
+* hdfsparquetimport - Imports Parquet table to a hoodie table
+* help - List all commands usage
+* marker delete - Delete the marker
+* metadata create - Create the Metadata Table if it does not exist
+* metadata delete - Remove the Metadata Table
+* metadata init - Update the metadata table from commits since the creation
+* metadata list-files - Print a list of all files in a partition from the metadata
+* metadata list-partitions - List all partitions from metadata
+* metadata refresh - Refresh table metadata
+* metadata set - Set options for Metadata Table
+* metadata stats - Print stats about the metadata
+* metadata validate-files - Validate all files in all partitions from the metadata
+* quit - Exits the shell
+* refresh - Refresh table metadata
+* repair addpartitionmeta - Add partition metadata to a table, if not present
+* repair corrupted clean files - repair corrupted clean files
+* repair deduplicate - De-duplicate a partition path contains duplicates & produce repaired files to replace with
+* repair migrate-partition-meta - Migrate all partition meta file currently stored in text format to be stored in base file format. See HoodieTableConfig#PARTITION_METAFILE_USE_DATA_FORMAT.
+* repair overwrite-hoodie-props - Overwrite hoodie.properties with provided file. Risky operation. Proceed with caution!
+* savepoint create - Savepoint a commit
+* savepoint delete - Delete the savepoint
+* savepoint rollback - Savepoint a commit
+* savepoints refresh - Refresh table metadata
+* savepoints show - Show the savepoints
+* script - Parses the specified resource file and executes its commands
+* set - Set spark launcher env to cli
+* show archived commits - Read commits from archived files and show details
+* show archived commit stats - Read commits from archived files and show details
+* show env - Show spark launcher env by key
+* show envs all - Show spark launcher envs
+* show fsview all - Show entire file-system view
+* show fsview latest - Show latest file-system view
+* show logfile metadata - Read commit metadata from log files
+* show logfile records - Read records from log files
+* show rollback - Show details of a rollback instant
+* show rollbacks - List all rollback instants
+* stats filesizes - File Sizes. Display summary stats on sizes of files
+* stats wa - Write Amplification. Ratio of how many records were upserted to how many records were actually written
+* sync validate - Validate the sync by counting the number of records
+* system properties - Shows the shell's properties
+* table delete-configs - Delete the supplied table configs from the table.
+* table recover-configs - Recover table configs, from update/delete that failed midway.
+* table update-configs - Update the table configs with configs with provided file.
+* temp_delete - Delete view name
+* temp_query - query against created temp view
+* temp delete - Delete view name
+* temp query - query against created temp view
+* temps_show - Show all views name
+* temps show - Show all views name
+* upgrade table - Upgrades a table
+* utils loadClass - Load a class
+* version - Displays shell version
+
+hudi:trips->
+```
+
+
+### Inspecting Commits
+
+The task of upserting or inserting a batch of incoming records is known as a **commit** in Hudi. A commit provides basic atomicity guarantees such that only committed data is available for querying.
+Each commit has a monotonically increasing string/number called the **commit number**. Typically, this is the time at which we started the commit.
+
+To view some basic information about the last 10 commits,
+
+
+```java
+hudi:trips->commits show --sortBy "Total Bytes Written" --desc true --limit 10
+ ________________________________________________________________________________________________________________________________________________________________________
+ | CommitTime | Total Bytes Written| Total Files Added| Total Files Updated| Total Partitions Written| Total Records Written| Total Update Records Written| Total Errors|
+ |=======================================================================================================================================================================|
+ ....
+ ....
+ ....
+```
+
+At the start of each write, Hudi also writes a .inflight commit to the .hoodie folder. You can use the timestamp there to estimate how long the commit has been inflight
+
+
+```java
+$ hdfs dfs -ls /app/uber/trips/.hoodie/*.inflight
+-rw-r--r-- 3 vinoth supergroup 321984 2016-10-05 23:18 /app/uber/trips/.hoodie/20161005225920.inflight
+```
+
+
+### Drilling Down to a specific Commit
+
+To understand how the writes spread across specific partiions,
+
+
+```java
+hudi:trips->commit showpartitions --commit 20161005165855 --sortBy "Total Bytes Written" --desc true --limit 10
+ __________________________________________________________________________________________________________________________________________
+ | Partition Path| Total Files Added| Total Files Updated| Total Records Inserted| Total Records Updated| Total Bytes Written| Total Errors|
+ |=========================================================================================================================================|
+ ....
+ ....
+```
+
+If you need file level granularity , we can do the following
+
+
+```java
+hudi:trips->commit showfiles --commit 20161005165855 --sortBy "Partition Path"
+ ________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File ID | Previous Commit| Total Records Updated| Total Records Written| Total Bytes Written| Total Errors|
+ |=======================================================================================================================================================|
+ ....
+ ....
+```
+
+
+### FileSystem View
+
+Hudi views each partition as a collection of file-groups with each file-group containing a list of file-slices in commit order (See concepts).
+The below commands allow users to view the file-slices for a data-set.
+
+```java
+hudi:stock_ticks_mor->show fsview all
+ ....
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta File Size| Delta Files |
+ |==============================================================================================================================================================================================================================================================================================================================================================================================================|
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]|
+
+
+
+hudi:stock_ticks_mor->show fsview latest --partitionPath "2018/08/31"
+ ......
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ [...]
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta Size| Delta Size - compaction scheduled| Delta Size - compaction unscheduled| Delta To Base Ratio - compaction scheduled| Delta To Base Ratio - compaction unscheduled| Delta Files - compaction scheduled | Delta Files - compaction unscheduled|
+ |========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== [...]
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | 20.8 KB | 0.0 B | 0.0 B | 0.0 B | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]| [] |
+
+```
+
+
+### Statistics
+
+Since Hudi directly manages file sizes for DFS table, it might be good to get an overall picture
+
+
+```java
+hudi:trips->stats filesizes --partitionPath 2016/09/01 --sortBy "95th" --desc true --limit 10
+ ________________________________________________________________________________________________
+ | CommitTime | Min | 10th | 50th | avg | 95th | Max | NumFiles| StdDev |
+ |===============================================================================================|
+ | <COMMIT_ID> | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 2 | 2.3 KB |
+ ....
+ ....
+```
+
+In case of Hudi write taking much longer, it might be good to see the write amplification for any sudden increases
+
+
+```java
+hudi:trips->stats wa
+ __________________________________________________________________________
+ | CommitTime | Total Upserted| Total Written| Write Amplifiation Factor|
+ |=========================================================================|
+ ....
+ ....
+```
+
+
+### Archived Commits
+
+In order to limit the amount of growth of .commit files on DFS, Hudi archives older .commit files (with due respect to the cleaner policy) into a commits.archived file.
+This is a sequence file that contains a mapping from commitNumber => json with raw information about the commit (same that is nicely rolled up above).
+
+
+### Compactions
+
+To get an idea of the lag between compaction and writer applications, use the below command to list down all
+pending compactions.
+
+```java
+hudi:trips->compactions show all
+ ___________________________________________________________________
+ | Compaction Instant Time| State | Total FileIds to be Compacted|
+ |==================================================================|
+ | <INSTANT_1> | REQUESTED| 35 |
+ | <INSTANT_2> | INFLIGHT | 27 |
+```
+
+To inspect a specific compaction plan, use
+
+```java
+hudi:trips->compaction show --instant <INSTANT_1>
+ _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File Id | Base Instant | Data File Path | Total Delta Files| getMetrics |
+ |================================================================================================================================================================================================================================================
+ | 2018/07/17 | <UUID> | <INSTANT_1> | viewfs://ns-default/.../../UUID_<INSTANT>.parquet | 1 | {TOTAL_LOG_FILES=1.0, TOTAL_IO_READ_MB=1230.0, TOTAL_LOG_FILES_SIZE=2.51255751E8, TOTAL_IO_WRITE_MB=991.0, TOTAL_IO_MB=2221.0}|
+
+```
+
+To manually schedule or run a compaction, use the below command. This command uses spark launcher to perform compaction
+operations.
+
+**NOTE:** Make sure no other application is scheduling compaction for this table concurrently
+\{: .notice--info}
+
+```java
+hudi:trips->help compaction schedule
+Keyword: compaction schedule
+Description: Schedule Compaction
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: false
+ Default if specified: '__NULL__'
+ Default if unspecified: '1G'
+
+* compaction schedule - Schedule Compaction
+```
+
+```java
+hudi:trips->help compaction run
+Keyword: compaction run
+Description: Run Compaction for given instant time
+ Keyword: tableName
+ Help: Table name
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: parallelism
+ Help: Parallelism for hoodie compaction
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: schemaFilePath
+ Help: Path for Avro schema file
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: retry
+ Help: Number of retries
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: compactionInstant
+ Help: Base path for the target hoodie table
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+* compaction run - Run Compaction for given instant time
+```
+
+### Validate Compaction
+
+Validating a compaction plan : Check if all the files necessary for compactions are present and are valid
+
+```java
+hudi:stock_ticks_mor->compaction validate --instant 20181005222611
+...
+
+ COMPACTION PLAN VALID
+
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error|
+ |==========================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | true | |
+
+
+
+hudi:stock_ticks_mor->compaction validate --instant 20181005222601
+
+ COMPACTION PLAN INVALID
+
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error |
+ |=====================================================================================================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | false| All log files specified in compaction operation is not present. Missing .... |
+```
+
+**NOTE:** The following commands must be executed without any other writer/ingestion application running.
+\{: .notice--warning}
+
+Sometimes, it becomes necessary to remove a fileId from a compaction-plan inorder to speed-up or unblock compaction
+operation. Any new log-files that happened on this file after the compaction got scheduled will be safely renamed
+so that are preserved. Hudi provides the following CLI to support it
+
+
+### Unscheduling Compaction
+
+```java
+hudi:trips->compaction unscheduleFileId --fileId <FileUUID>
+....
+No File renames needed to unschedule file from pending compaction. Operation successful.
+```
+
+In other cases, an entire compaction plan needs to be reverted. This is supported by the following CLI
+
+```java
+hudi:trips->compaction unschedule --instant <compactionInstant>
+.....
+No File renames needed to unschedule pending compaction. Operation successful.
+```
+
+### Repair Compaction
+
+The above compaction unscheduling operations could sometimes fail partially (e:g -> DFS temporarily unavailable). With
+partial failures, the compaction operation could become inconsistent with the state of file-slices. When you run
+`compaction validate`, you can notice invalid compaction operations if there is one. In these cases, the repair
+command comes to the rescue, it will rearrange the file-slices so that there is no loss and the file-slices are
+consistent with the compaction plan
+
+```java
+hudi:stock_ticks_mor->compaction repair --instant 20181005222611
+......
+Compaction successfully repaired
+.....
+```
+
+### Savepoint and Restore
+As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery.md)
+
+To trigger savepoint for a hudi table
+```java
+connect --path /tmp/hudi_trips_cow/
+commits show
+set --conf SPARK_HOME=<SPARK_HOME>
+savepoint create --commit 20220128160245447 --sparkMaster local[2]
+```
+
+To restore the table to one of the savepointed commit:
+
+```java
+connect --path /tmp/hudi_trips_cow/
+commits show
+set --conf SPARK_HOME=<SPARK_HOME>
+savepoints show
+╔═══════════════════╗
+║ SavepointTime ║
+╠═══════════════════╣
+║ 20220128160245447 ║
+╚═══════════════════╝
+savepoint rollback --savepoint 20220128160245447 --sparkMaster local[2]
+```
+
+### Upgrade and Downgrade Table
+In case the user needs to downgrade the version of Hudi library used, the Hudi table needs to be manually downgraded
+on the newer version of Hudi CLI before library downgrade. To downgrade a Hudi table through CLI, user needs to specify
+the target Hudi table version as follows:
+
+```shell
+connect --path <table_path>
+downgrade table --toVersion <target_version>
+```
+
+The following table shows the Hudi table versions corresponding to the Hudi release versions:
+
+| Hudi Table Version | Hudi Release Version(s) |
+|:-------------------|:------------------------|
+| `SIX` or `6` | 0.14.x - 0.15.x |
+| `FIVE` or `5` | 0.12.x - 0.13.x |
+| `FOUR` or `4` | 0.11.x |
+| `THREE` or `3` | 0.10.x |
+| `TWO` or `2` | 0.9.x |
+| `ONE` or `1` | 0.6.x - 0.8.x |
+| `ZERO` or `0` | 0.5.x and below |
+
+For example, to downgrade a table from version `SIX`(`6`) (current version) to `FIVE`(`5`), you should run (use proper Spark master based
+on your environment)
+
+```shell
+downgrade table --toVersion FIVE --sparkMaster local[2]
+```
+
+or
+
+```shell
+downgrade table --toVersion 5 --sparkMaster local[2]
+```
+
+You can verify the table version by looking at the `hoodie.table.version` property in `.hoodie/hoodie.properties` under
+the table path:
+
+```properties
+hoodie.table.version=5
+```
+
+Hudi CLI also provides the ability to manually upgrade a Hudi table. To upgrade a Hudi table through CLI:
+
+```shell
+upgrade table --toVersion <target_version>
+```
+
+:::note
+Table upgrade is automatically handled by the Hudi write client in different deployment modes such as DeltaStreamer
+after upgrading the Hudi library so that the user does not have to do manual upgrade. Such automatic table upgrade
+is the **recommended** way in general, instead of using `upgrade` CLI command.
+
+Table upgrade from table version ONE to TWO requires key generator related configs such as
+"hoodie.datasource.write.recordkey.field", which is only available when user configures the write job. So the table
+upgrade from version ONE to TWO through CLI is not supported, and user should rely on the automatic upgrade in the write
+client instead.
+:::
+
+You may also run the upgrade command without specifying the target version. In such a case, the latest table version
+corresponding to the library release version is used:
+
+```shell
+upgrade table
+```
+
+### Change Hudi Table Type
+There are cases we want to change the hudi table type. For example, change COW table to MOR for more efficient and
+lower latency ingestion; change MOR to COW for better read performance and compatibility with downstream engines.
+So we offer the table command to perform this modification conveniently.
+
+Changing **COW to MOR**, we can simply modify the `hoodie.table.type` in `hoodie.properties` to MERGE_ON_READ.
+
+While changing **MOR to COW**, we must make sure all the log files are compacted before modifying the table type,
+or it will cause data loss.
+
+```shell
+connect --path <table_path>
+table change-table-type <target_table_type>
+```
+
+The parameter `target_table_type` candidates are below:
+
+| target table type | comment [...]
+|:------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| MOR | Change COW table to MERGE_ON_READ. [...]
+| COW | Change MOR table to COPY_ON_WRITE. <br/>By default, changing to COW will **execute all pending compactions** and **perform a full compaction** if any log file left. Set `--enable-compaction=false` will disable the default compaction. <br/>There are params can be set for the compaction operation:<br/>`--parallelism`: Default `3`. Parallelism for hoodie compaction<br/>`--sparkMaster`: Default `local`. Spark Master<br/>`--sparkMemory`: Default `4G`. Spark executor memo [...]
+
+
+Example below is changing MOR table to COW:
+```shell
+connect --path /var/dataset/test_table_mor2cow
+desc
+╔════════════════════════════════════════════════╤═════════════════════════════════════════╗
+║ Property │ Value ║
+╠════════════════════════════════════════════════╪═════════════════════════════════════════╣
+║ basePath │ /var/dataset/test_table_mor2cow ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ metaPath │ /var/dataset/test_table_mor2cow/.hoodie ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ fileSystem │ file ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.name │ test_table ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.compaction.record.merger.strategy │ eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.metadata.partitions │ files ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.type │ MERGE_ON_READ ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.metadata.partitions.inflight │ ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.archivelog.folder │ archived ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.timeline.layout.version │ 1 ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.checksum │ 2702201862 ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.compaction.payload.type │ HOODIE_AVRO ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.table.version │ 6 ║
+╟────────────────────────────────────────────────┼─────────────────────────────────────────╢
+║ hoodie.datasource.write.drop.partition.columns │ false ║
+╚════════════════════════════════════════════════╧═════════════════════════════════════════╝
+
+table change-table-type COW
+╔════════════════════════════════════════════════╤══════════════════════════════════════╤══════════════════════════════════════╗
+║ Property │ Old Value │ New Value ║
+╠════════════════════════════════════════════════╪══════════════════════════════════════╪══════════════════════════════════════╣
+║ hoodie.archivelog.folder │ archived │ archived ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.compaction.payload.type │ HOODIE_AVRO │ HOODIE_AVRO ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.compaction.record.merger.strategy │ eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 │ eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.datasource.write.drop.partition.columns │ false │ false ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.checksum │ 2702201862 │ 2702201862 ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.metadata.partitions │ files │ files ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.metadata.partitions.inflight │ │ ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.name │ test_table │ test_table ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.type │ MERGE_ON_READ │ COPY_ON_WRITE ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.table.version │ 6 │ 6 ║
+╟────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────╢
+║ hoodie.timeline.layout.version │ 1 │ 1 ║
+╚════════════════════════════════════════════════╧══════════════════════════════════════╧══════════════════════════════════════╝
+```
diff --git a/website/versioned_docs/version-0.15.1/cloud.md b/website/versioned_docs/version-0.15.1/cloud.md
new file mode 100644
index 000000000000..a3477b551bee
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/cloud.md
@@ -0,0 +1,38 @@
+---
+title: Cloud Storage
+keywords: [hudi, aws, gcp, oss, azure, cloud, juicefs]
+summary: "In this page, we introduce how Hudi work with different Cloud providers."
+toc: true
+last_modified_at: 2021-10-12T10:50:00+08:00
+---
+
+## Talking to Cloud Storage
+
+Immaterial of whether RDD/WriteClient APIs or Datasource is used, the following information helps configure access
+to cloud stores.
+
+* [AWS S3](s3_hoodie.md) <br/>
+ Configurations required for S3 and Hudi co-operability.
+* [Google Cloud Storage](gcs_hoodie.md) <br/>
+ Configurations required for GCS and Hudi co-operability.
+* [Alibaba Cloud OSS](oss_hoodie.md) <br/>
+ Configurations required for OSS and Hudi co-operability.
+* [Microsoft Azure](azure_hoodie.md) <br/>
+ Configurations required for Azure and Hudi co-operability.
+* [Tencent Cloud Object Storage](cos_hoodie.md) <br/>
+ Configurations required for COS and Hudi co-operability.
+* [IBM Cloud Object Storage](ibm_cos_hoodie.md) <br/>
+ Configurations required for IBM Cloud Object Storage and Hudi co-operability.
+* [Baidu Cloud Object Storage](bos_hoodie.md) <br/>
+ Configurations required for BOS and Hudi co-operability.
+* [JuiceFS](jfs_hoodie.md) <br/>
+ Configurations required for JuiceFS and Hudi co-operability.
+* [Oracle Cloud Infrastructure](oci_hoodie.md) <br/>
+ Configurations required for OCI and Hudi co-operability.
+
+:::note
+Many cloud object storage systems like [Amazon S3](https://docs.aws.amazon.com/s3/) allow you to set
+lifecycle policies, such as [S3 Lifecycle](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html),
+to manage objects. One of the policies is related to object expiration. If your organisation has configured such policies,
+then please ensure to exclude (or have a longer expiry period) for Hudi tables.
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.15.1/clustering.md b/website/versioned_docs/version-0.15.1/clustering.md
new file mode 100644
index 000000000000..3ed54f256c29
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/clustering.md
@@ -0,0 +1,343 @@
+---
+title: Clustering
+summary: "In this page, we describe async compaction in Hudi."
+toc: true
+last_modified_at:
+---
+
+## Background
+
+Apache Hudi brings stream processing to big data, providing fresh data while being an order of magnitude efficient over traditional batch processing. In a data lake/warehouse, one of the key trade-offs is between ingestion speed and query performance. Data ingestion typically prefers small files to improve parallelism and make data available to queries as soon as possible. However, query performance degrades poorly with a lot of small files. Also, during ingestion, data is typically co-l [...]
+<!--truncate-->
+
+## How is compaction different from clustering?
+
+Hudi is modeled like a log-structured storage engine with multiple versions of the data.
+Particularly, [Merge-On-Read](table_types.md#merge-on-read-table)
+tables in Hudi store data using a combination of base file in columnar format and row-based delta logs that contain
+updates. Compaction is a way to merge the delta logs with base files to produce the latest file slices with the most
+recent snapshot of data. Compaction helps to keep the query performance in check (larger delta log files would incur
+longer merge times on query side). On the other hand, clustering is a data layout optimization technique. One can stitch
+together small files into larger files using clustering. Additionally, data can be clustered by sort key so that queries
+can take advantage of data locality.
+
+## Clustering Architecture
+
+At a high level, Hudi provides different operations such as insert/upsert/bulk_insert through it’s write client API to be able to write data to a Hudi table. To be able to choose a trade-off between file size and ingestion speed, Hudi provides a knob `hoodie.parquet.small.file.limit` to be able to configure the smallest allowable file size. Users are able to configure the small file [soft limit](https://hudi.apache.org/docs/configurations/#hoodieparquetsmallfilelimit) to `0` to force new [...]
+
+
+
+To be able to support an architecture that allows for fast ingestion without compromising query performance, we have introduced a ‘clustering’ service to rewrite the data to optimize Hudi data lake file layout.
+
+Clustering table service can run asynchronously or synchronously adding a new action type called “REPLACE”, that will mark the clustering action in the Hudi metadata timeline.
+
+
+
+### Overall, there are 2 steps to clustering
+
+1. Scheduling clustering: Create a clustering plan using a pluggable clustering strategy.
+2. Execute clustering: Process the plan using an execution strategy to create new files and replace old files.
+
+
+### Schedule clustering
+
+Following steps are followed to schedule clustering.
+
+1. Identify files that are eligible for clustering: Depending on the clustering strategy chosen, the scheduling logic will identify the files eligible for clustering.
+2. Group files that are eligible for clustering based on specific criteria. Each group is expected to have data size in multiples of ‘targetFileSize’. Grouping is done as part of ‘strategy’ defined in the plan. Additionally, there is an option to put a cap on group size to improve parallelism and avoid shuffling large amounts of data.
+3. Finally, the clustering plan is saved to the timeline in an avro [metadata format](https://github.com/apache/hudi/blob/master/hudi-common/src/main/avro/HoodieClusteringPlan.avsc).
+
+
+### Execute clustering
+
+1. Read the clustering plan and get the ‘clusteringGroups’ that mark the file groups that need to be clustered.
+2. For each group, we instantiate appropriate strategy class with strategyParams (example: sortColumns) and apply that strategy to rewrite the data.
+3. Create a “REPLACE” commit and update the metadata in [HoodieReplaceCommitMetadata](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieReplaceCommitMetadata.java).
+
+
+Clustering Service builds on Hudi’s MVCC based design to allow for writers to continue to insert new data while clustering action runs in the background to reformat data layout, ensuring snapshot isolation between concurrent readers and writers.
+
+NOTE: Clustering can only be scheduled for tables / partitions not receiving any concurrent updates. In the future, concurrent updates use-case will be supported as well.
+
+
+_Figure: Illustrating query performance improvements by clustering_
+
+## Clustering Usecases
+
+### Batching small files
+
+As mentioned in the intro, streaming ingestion generally results in smaller files in your data lake. But having a lot of
+such small files could lead to higher query latency. From our experience supporting community users, there are quite a
+few users who are using Hudi just for small file handling capabilities. So, you could employ clustering to batch a lot
+of such small files into larger ones.
+
+
+
+### Cluster by sort key
+
+Another classic problem in data lake is the arrival time vs event time problem. Generally you write data based on
+arrival time, while query predicates do not sit well with it. With clustering, you can re-write your data by sorting
+based on query predicates and so, your data skipping will be very efficient and your query can ignore scanning a lot of
+unnecessary data.
+
+
+
+## Clustering Strategies
+
+On a high level, clustering creates a plan based on a configurable strategy, groups eligible files based on specific
+criteria and then executes the plan. As mentioned before, clustering plan as well as execution depends on configurable
+strategy. These strategies can be broadly classified into three types: clustering plan strategy, execution strategy and
+update strategy.
+
+### Plan Strategy
+
+This strategy comes into play while creating clustering plan. It helps to decide what file groups should be clustered
+and how many output file groups should the clustering produce. Note that these strategies are easily pluggable using the
+config [hoodie.clustering.plan.strategy.class](configurations.md#hoodieclusteringplanstrategyclass).
+
+Different plan strategies are as follows:
+
+#### Size-based clustering strategies
+
+This strategy creates clustering groups based on max size allowed per group. Also, it excludes files that are greater
+than the small file limit from the clustering plan. Available strategies depending on write client
+are: `SparkSizeBasedClusteringPlanStrategy`, `FlinkSizeBasedClusteringPlanStrategy`
+and `JavaSizeBasedClusteringPlanStrategy`. Furthermore, Hudi provides flexibility to include or exclude partitions for
+clustering, tune the file size limits, maximum number of output groups. Please refer to [hoodie.clustering.plan.strategy.small.file.limit](https://hudi.apache.org/docs/next/configurations/#hoodieclusteringplanstrategysmallfilelimit)
+, [hoodie.clustering.plan.strategy.max.num.groups](https://hudi.apache.org/docs/next/configurations/#hoodieclusteringplanstrategymaxnumgroups), [hoodie.clustering.plan.strategy.max.bytes.per.group](https://hudi.apache.org/docs/next/configurations/#hoodieclusteringplanstrategymaxbytespergroup)
+, [hoodie.clustering.plan.strategy.target.file.max.bytes](https://hudi.apache.org/docs/next/configurations/#hoodieclusteringplanstrategytargetfilemaxbytes) for more details.
+
+| Config Name | Default | Description [...]
+|---------------------------------------------------------| -------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.clustering.plan.strategy.partition.selected | N/A **(Required)** | Comma separated list of partitions to run clustering<br /><br />`Config Param: PARTITION_SELECTED`<br />`Since Version: 0.11.0` [...]
+| hoodie.clustering.plan.strategy.partition.regex.pattern | N/A **(Required)** | Filter clustering partitions that matched regex pattern<br /><br />`Config Param: PARTITION_REGEX_PATTERN`<br />`Since Version: 0.11.0` [...]
+| hoodie.clustering.plan.partition.filter.mode | NONE (Optional) | Partition filter mode used in the creation of clustering plan. Possible values:<br /><ul><li>`NONE`: Do not filter partitions. The clustering plan will include all partitions that have clustering candidates.</li><li>`RECENT_DAYS`: This filter assumes that your data is partitioned by date. The clustering plan will only include partitions from K days ago to N days ago, where K >= N. K is determined by `hood [...]
+
+
+#### SparkSingleFileSortPlanStrategy
+
+In this strategy, clustering group for each partition is built in the same way as `SparkSizeBasedClusteringPlanStrategy`
+. The difference is that the output group is 1 and file group id remains the same,
+while `SparkSizeBasedClusteringPlanStrategy` can create multiple file groups with newer fileIds.
+
+#### SparkConsistentBucketClusteringPlanStrategy
+
+This strategy is specifically used for consistent bucket index. This will be leveraged to expand your bucket index (from
+static partitioning to dynamic). Typically, users don’t need to use this strategy. Hudi internally uses this for
+dynamically expanding the buckets for bucket index datasets.
+
+:::note The latter two strategies are applicable only for the Spark engine.
+:::
+
+### Execution Strategy
+
+After building the clustering groups in the planning phase, Hudi applies execution strategy, for each group, primarily
+based on sort columns and size. The strategy can be specified using the
+config [hoodie.clustering.execution.strategy.class](configurations.md#hoodieclusteringexecutionstrategyclass). By
+default, Hudi sorts the file groups in the plan by the specified columns, while meeting the configured target file
+sizes.
+
+| Config Name | Default | Description [...]
+| --------------------------------------------| ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.clustering.execution.strategy.class | org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy (Optional) | Config to provide a strategy class (subclass of RunClusteringStrategy) to define how the clustering plan is executed. By default, we sort the file groups in th plan by the specified columns, while meeting the configured target file sizes.<br /><br />`Config Param: EXECUTION_STRATEGY_CLASS_NAME`<br />`Since Version: 0.7.0` [...]
+
+The available strategies are as follows:
+
+1. `SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY`: Uses bulk_insert to re-write data from input file groups.
+ 1. Set `hoodie.clustering.execution.strategy.class`
+ to `org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy`.
+ 2. `hoodie.clustering.plan.strategy.sort.columns`: Columns to sort the data while clustering. This goes in
+ conjunction with layout optimization strategies depending on your query predicates. One can set comma separated
+ list of columns that needs to be sorted in this config.
+2. `JAVA_SORT_AND_SIZE_EXECUTION_STRATEGY`: Similar to `SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY`, for the Java and Flink
+ engines. Set `hoodie.clustering.execution.strategy.class`
+ to `org.apache.hudi.client.clustering.run.strategy.JavaSortAndSizeExecutionStrategy`.
+3. `SPARK_CONSISTENT_BUCKET_EXECUTION_STRATEGY`: As the name implies, this is applicable to dynamically expand
+ consistent bucket index and only applicable to the Spark engine. Set `hoodie.clustering.execution.strategy.class`
+ to `org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+
+### Update Strategy
+
+Currently, clustering can only be scheduled for tables/partitions not receiving any concurrent updates. By default,
+the config for update strategy - [`hoodie.clustering.updates.strategy`](configurations.md#hoodieclusteringupdatesstrategy) is set to ***SparkRejectUpdateStrategy***. If some file group has updates during clustering then it will reject updates and throw an
+exception. However, in some use-cases updates are very sparse and do not touch most file groups. The default strategy to
+simply reject updates does not seem fair. In such use-cases, users can set the config to ***SparkAllowUpdateStrategy***.
+
+We discussed the critical strategy configurations. All other configurations related to clustering are
+listed [here](configurations.md#Clustering-Configs). Out of this list, a few configurations that will be very useful
+for inline or async clustering are shown below with code samples.
+
+## Inline clustering
+
+Inline clustering happens synchronously with the regular ingestion writer or as part of the data ingestion pipeline. This means the next round of ingestion cannot proceed until the clustering is complete With inline clustering, Hudi will schedule, plan clustering operations after each commit is completed and execute the clustering plans after it’s created. This is the simplest deployment model to run because it’s easier to manage than running different asynchronous Spark jobs. This mode [...]
+
+For this deployment mode, please enable and set: `hoodie.clustering.inline`
+
+To choose how often clustering is triggered, also set: `hoodie.clustering.inline.max.commits`.
+
+Inline clustering can be setup easily using spark dataframe options.
+See sample below:
+
+```scala
+import org.apache.hudi.QuickstartUtils._
+import scala.collection.JavaConversions._
+import org.apache.spark.sql.SaveMode._
+import org.apache.hudi.DataSourceReadOptions._
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.config.HoodieWriteConfig._
+
+
+val df = //generate data frame
+df.write.format("org.apache.hudi").
+ options(getQuickstartWriteConfigs).
+ option("hoodie.datasource.write.precombine.field", "ts").
+ option("hoodie.datasource.write.recordkey.field", "uuid").
+ option("hoodie.datasource.write.partitionpath.field", "partitionpath").
+ option("hoodie.table.name", "tableName").
+ option("hoodie.parquet.small.file.limit", "0").
+ option("hoodie.clustering.inline", "true").
+ option("hoodie.clustering.inline.max.commits", "4").
+ option("hoodie.clustering.plan.strategy.target.file.max.bytes", "1073741824").
+ option("hoodie.clustering.plan.strategy.small.file.limit", "629145600").
+ option("hoodie.clustering.plan.strategy.sort.columns", "column1,column2"). //optional, if sorting is needed as part of rewriting data
+ mode(Append).
+ save("dfs://location");
+```
+
+## Async Clustering
+
+Async clustering runs the clustering table service in the background without blocking the regular ingestions writers. There are three different ways to deploy an asynchronous clustering process:
+
+- **Asynchronous execution within the same process**: In this deployment mode, Hudi will schedule and plan the clustering operations after each commit is completed as part of the ingestion pipeline. Separately, Hudi spins up another thread within the same job and executes the clustering table service. This is supported by Spark Streaming, Flink and DeltaStreamer in continuous mode. For this deployment mode, please enable `hoodie.clustering.async.enabled` and `hoodie.clustering.async.max. [...]
+- **Asynchronous scheduling and execution by a separate process**: In this deployment mode, the application will write data to a Hudi table as part of the ingestion pipeline. A separate clustering job will schedule, plan and execute the clustering operation. By running a different job for the clustering operation, it rebalances how Hudi uses compute resources: fewer compute resources are needed for the ingestion, which makes ingestion latency stable, and an independent set of compute res [...]
+- **Scheduling inline and executing async**: In this deployment mode, the application ingests data and schedules the clustering in one job; in another, the application executes the clustering plan. The supported writers (see below) won’t be blocked from ingesting data. If the metadata table is enabled, a lock provider is not needed. However, if the metadata table is enabled, please ensure all jobs have the lock providers configured for concurrency control. All writers support this deploy [...]
+
+Hudi supports [multi-writers](https://hudi.apache.org/docs/concurrency_control#enabling-multi-writing) which provides
+snapshot isolation between multiple table services, thus allowing writers to continue with ingestion while clustering
+runs in the background.
+
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.clustering.async.enabled | false (Optional) | Enable running of clustering service, asynchronously as inserts happen on the table.<br /><br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| hoodie.clustering.async.max.commits | 4 (Optional) | Config to control frequency of async clustering<br /><br />`Config Param: ASYNC_CLUSTERING_MAX_COMMITS`<br />`Since Version: 0.9.0` [...]
+
+## Setup Asynchronous Clustering
+Users can leverage [HoodieClusteringJob](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+19+Clustering+data+for+freshness+and+query+performance#RFC19Clusteringdataforfreshnessandqueryperformance-SetupforAsyncclusteringJob)
+to setup 2-step asynchronous clustering.
+
+### HoodieClusteringJob
+By specifying the `scheduleAndExecute` mode both schedule as well as clustering can be achieved in the same step.
+The appropriate mode can be specified using `-mode` or `-m` option. There are three modes:
+
+1. `schedule`: Make a clustering plan. This gives an instant which can be passed in execute mode.
+2. `execute`: Execute a clustering plan at a particular instant. If no instant-time is specified, HoodieClusteringJob will execute for the earliest instant on the Hudi timeline.
+3. `scheduleAndExecute`: Make a clustering plan first and execute that plan immediately.
+
+Note that to run this job while the original writer is still running, please enable multi-writing:
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
+```
+
+A sample spark-submit command to setup HoodieClusteringJob is as below:
+
+```bash
+spark-submit \
+--class org.apache.hudi.utilities.HoodieClusteringJob \
+/path/to/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-0.9.0-SNAPSHOT.jar \
+--props /path/to/config/clusteringjob.properties \
+--mode scheduleAndExecute \
+--base-path /path/to/hudi_table/basePath \
+--table-name hudi_table_schedule_clustering \
+--spark-memory 1g
+```
+A sample `clusteringjob.properties` file:
+```
+hoodie.clustering.async.enabled=true
+hoodie.clustering.async.max.commits=4
+hoodie.clustering.plan.strategy.target.file.max.bytes=1073741824
+hoodie.clustering.plan.strategy.small.file.limit=629145600
+hoodie.clustering.execution.strategy.class=org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy
+hoodie.clustering.plan.strategy.sort.columns=column1,column2
+```
+
+### HoodieStreamer
+
+This brings us to our users' favorite utility in Hudi. Now, we can trigger asynchronous clustering with Hudi Streamer.
+Just set the `hoodie.clustering.async.enabled` config to true and specify other clustering config in properties file
+whose location can be pased as `—props` when starting the Hudi Streamer (just like in the case of HoodieClusteringJob).
+
+A sample spark-submit command to setup HoodieStreamer is as below:
+
+```bash
+spark-submit \
+--class org.apache.hudi.utilities.streamer.HoodieStreamer \
+/path/to/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-0.9.0-SNAPSHOT.jar \
+--props /path/to/config/clustering_kafka.properties \
+--schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+--source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+--source-ordering-field impresssiontime \
+--table-type COPY_ON_WRITE \
+--target-base-path /path/to/hudi_table/basePath \
+--target-table impressions_cow_cluster \
+--op INSERT \
+--hoodie-conf hoodie.clustering.async.enabled=true \
+--continuous
+```
+
+### Spark Structured Streaming
+
+We can also enable asynchronous clustering with Spark structured streaming sink as shown below.
+```scala
+val commonOpts = Map(
+ "hoodie.insert.shuffle.parallelism" -> "4",
+ "hoodie.upsert.shuffle.parallelism" -> "4",
+ "hoodie.datasource.write.recordkey.field" -> "_row_key",
+ "hoodie.datasource.write.partitionpath.field" -> "partition",
+ "hoodie.datasource.write.precombine.field" -> "timestamp",
+ "hoodie.table.name" -> "hoodie_test"
+)
+
+def getAsyncClusteringOpts(isAsyncClustering: String,
+ clusteringNumCommit: String,
+ executionStrategy: String):Map[String, String] = {
+ commonOpts + (DataSourceWriteOptions.ASYNC_CLUSTERING_ENABLE.key -> isAsyncClustering,
+ HoodieClusteringConfig.ASYNC_CLUSTERING_MAX_COMMITS.key -> clusteringNumCommit,
+ HoodieClusteringConfig.EXECUTION_STRATEGY_CLASS_NAME.key -> executionStrategy
+ )
+}
+
+def initStreamingWriteFuture(hudiOptions: Map[String, String]): Future[Unit] = {
+ val streamingInput = // define the source of streaming
+ Future {
+ println("streaming starting")
+ streamingInput
+ .writeStream
+ .format("org.apache.hudi")
+ .options(hudiOptions)
+ .option("checkpointLocation", basePath + "/checkpoint")
+ .mode(Append)
+ .start()
+ .awaitTermination(10000)
+ println("streaming ends")
+ }
+}
+
+def structuredStreamingWithClustering(): Unit = {
+ val df = //generate data frame
+ val hudiOptions = getClusteringOpts("true", "1", "org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy")
+ val f1 = initStreamingWriteFuture(hudiOptions)
+ Await.result(f1, Duration.Inf)
+}
+```
+
+## Java Client
+
+Clustering is also supported via Java client. Plan strategy `org.apache.hudi.client.clustering.plan.strategy.JavaSizeBasedClusteringPlanStrategy`
+and execution strategy `org.apache.hudi.client.clustering.run.strategy.JavaSortAndSizeExecutionStrategy` are supported
+out-of-the-box. Note that as of now only linear sort is supported in Java execution strategy.
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Understanding Clustering in Apache Hudi and the Benefits of Asynchronous Clustering](https://www.youtube.com/watch?v=R_sm4wlGXuE)
diff --git a/website/versioned_docs/version-0.15.1/compaction.md b/website/versioned_docs/version-0.15.1/compaction.md
new file mode 100644
index 000000000000..341b885f1e70
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/compaction.md
@@ -0,0 +1,228 @@
+---
+title: Compaction
+summary: "In this page, we describe async compaction in Hudi."
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+last_modified_at:
+---
+## Background
+Compaction is a table service employed by Hudi specifically in Merge On Read(MOR) tables to merge updates from row-based log
+files to the corresponding columnar-based base file periodically to produce a new version of the base file. Compaction is
+not applicable to Copy On Write(COW) tables and only applies to MOR tables.
+
+### Why MOR tables need compaction?
+To understand the significance of compaction in MOR tables, it is helpful to understand the MOR table layout first. In Hudi,
+data is organized in terms of [file groups](file_layouts.md). Each file group in a MOR table
+consists of a base file and one or more log files. Typically, during writes, inserts are stored in the base file, and updates
+are appended to log files.
+
+
+_Figure: MOR table file layout showing different file groups with base data file and log files_
+
+During the compaction process, updates from the log files are merged with the base file to form a new version of the
+base file as shown below. Since MOR is designed to be write-optimized, on new writes, after index tagging is complete,
+Hudi appends the records pertaining to each file groups as log blocks in log files. There is no synchronous merge
+happening during write, resulting in a lower write amplification and better write latency. In contrast, on new writes to a
+COW table, Hudi combines the new writes with the older base file to produce a new version of the base file resulting in
+a higher write amplification and higher write latencies.
+
+
+_Figure: Compaction on a given file group_
+
+While serving the read query(snapshot read), for each file group, records in base file and all its corresponding log
+files are merged together and served. And hence the read latency for MOR snapshot query might be higher compared to
+COW table since there is no merge involved in case of COW at read time. Compaction takes care of merging the updates from
+log files with the base file at regular intervals to bound the growth of log files and to ensure the read latencies do not
+spike up.
+
+## Compaction Architecture
+There are two steps to compaction.
+- ***Compaction Scheduling***: In this step, Hudi scans the partitions and selects file slices to be compacted. A compaction
+ plan is finally written to Hudi timeline.
+- ***Compaction Execution***: In this step the compaction plan is read and file slices are compacted.
+
+### Strategies in Compaction Scheduling
+There are two strategies involved in scheduling the compaction:
+- Trigger Strategy: Determines how often to trigger scheduling of the compaction.
+- Compaction Strategy: Determines which file groups to compact.
+
+Hudi provides various options for both these strategies as discussed below.
+
+#### Trigger Strategies
+
+| Config Name | Default | Description |
+|----------------------------------------------------|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.compact.inline.trigger.strategy | NUM_COMMITS (Optional) | org.apache.hudi.table.action.compact.CompactionTriggerStrategy: Controls when compaction is scheduled.<br />`Config Param: INLINE_COMPACT_TRIGGER_STRATEGY` |
+
+Possible values:<br /><ul><li>`NUM_COMMITS`: triggers compaction when there are at least N delta commits after last completed compaction.</li><li>`NUM_COMMITS_AFTER_LAST_REQUEST`: triggers compaction when there are at least N delta commits after last completed or requested compaction.</li><li>`TIME_ELAPSED`: triggers compaction after N seconds since last compaction.</li><li>`NUM_AND_TIME`: triggers compaction when both there are at least N delta commits and N seconds elapsed (both must b [...]
+#### Compaction Strategies
+| Config Name | Default | Description [...]
+|----------------------------------------------------|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.compaction.strategy | org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy (Optional) | Compaction strategy decides which file groups are picked up for compaction during each compaction run. By default. Hudi picks the log file with most accumulated unmerged data. <br /><br />`Config Param: COMPACTION_STRATEGY` |
+
+Available Strategies (Provide the full package name when using the strategy): <ul><li>`LogFileNumBasedCompactionStrategy`:
+orders the compactions based on the total log files count, filters the file group with log files count greater than the
+threshold and limits the compactions within a configured IO bound.</li><li>`LogFileSizeBasedCompactionStrategy`: orders
+the compactions based on the total log files size, filters the file group which log files size is greater than the
+threshold and limits the compactions within a configured IO bound.</li><li>`BoundedIOCompactionStrategy`: CompactionStrategy
+which looks at total IO to be done for the compaction (read + write) and limits the list of compactions to be under a
+configured limit on the IO.</li><li>`BoundedPartitionAwareCompactionStrategy`:This strategy ensures that the last N partitions
+are picked up even if there are later partitions created for the table. lastNPartitions is defined as the N partitions before
+the currentDate. currentDay = 2018/01/01 The table has partitions for 2018/02/02 and 2018/03/03 beyond the currentDay This
+strategy will pick up the following partitions for compaction : (2018/01/01, allPartitionsInRange[(2018/01/01 - lastNPartitions)
+to 2018/01/01), 2018/02/02, 2018/03/03)</li><li>`DayBasedCompactionStrategy`:This strategy orders compactions in reverse
+order of creation of Hive Partitions. It helps to compact data in latest partitions first and then older capped at the
+Total_IO allowed.</li><li>`UnBoundedCompactionStrategy`: UnBoundedCompactionStrategy will not change ordering or filter
+any compaction. It is a pass-through and will compact all the base files which has a log file. This usually means
+no-intelligence on compaction.</li><li>`UnBoundedPartitionAwareCompactionStrategy`:UnBoundedPartitionAwareCompactionStrategy is a custom UnBounded Strategy. This will filter all the partitions that
+are eligible to be compacted by a \{@link BoundedPartitionAwareCompactionStrategy} and return the result. This is done
+so that a long running UnBoundedPartitionAwareCompactionStrategy does not step over partitions in a shorter running
+BoundedPartitionAwareCompactionStrategy. Essentially, this is an inverse of the partitions chosen in
+BoundedPartitionAwareCompactionStrategy</li></ul>
+
+:::note
+Please refer to [advanced configs](https://hudi.apache.org/docs/next/configurations#Compaction-Configs) for more details.
+:::
+
+## Ways to trigger Compaction
+
+### Inline
+By default, compaction is run asynchronously.
+
+If latency of ingesting records is important for you, you are most likely using Merge-On-Read tables.
+Merge-On-Read tables store data using a combination of columnar (e.g parquet) + row based (e.g avro) file formats.
+Updates are logged to delta files & later compacted to produce new versions of columnar files.
+To improve ingestion latency, Async Compaction is the default configuration.
+
+If immediate read performance of a new commit is important for you, or you want simplicity of not managing separate compaction jobs,
+you may want synchronous inline compaction, which means that as a commit is written it is also compacted by the same job.
+
+For this deployment mode, please use `hoodie.compact.inline = true` for Spark Datasource and Spark SQL writers. For
+HoodieStreamer sync once mode inline compaction can be achieved by passing the flag `--disable-compaction` (Meaning to
+disable async compaction). Further in HoodieStreamer when both
+ingestion and compaction is running in the same spark context, you can use resource allocation configuration
+in Hudi Streamer CLI such as (`--delta-sync-scheduling-weight`,
+`--compact-scheduling-weight`, `--delta-sync-scheduling-minshare`, and `--compact-scheduling-minshare`)
+to control executor allocation between ingestion and compaction.
+
+
+### Async & Offline Compaction models
+
+There are a couple of ways here to trigger compaction .
+
+#### Async execution within the same process
+In streaming ingestion write models like HoodieStreamer
+continuous mode, Flink and Spark Streaming, async compaction is enabled by default and runs alongside without blocking
+regular ingestion.
+
+##### Spark Structured Streaming
+
+Compactions are scheduled and executed asynchronously inside the
+streaming job.Here is an example snippet in java
+
+```properties
+import org.apache.hudi.DataSourceWriteOptions;
+import org.apache.hudi.HoodieDataSourceHelpers;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import org.apache.spark.sql.streaming.OutputMode;
+import org.apache.spark.sql.streaming.ProcessingTime;
+
+
+ DataStreamWriter<Row> writer = streamingInput.writeStream().format("org.apache.hudi")
+ .option("hoodie.datasource.write.operation", operationType)
+ .option("hoodie.datasource.write.table.type", tableType)
+ .option("hoodie.datasource.write.recordkey.field", "_row_key")
+ .option("hoodie.datasource.write.partitionpath.field", "partition")
+ .option("hoodie.datasource.write.precombine.field"(), "timestamp")
+ .option("hoodie.compact.inline.max.delta.commits", "10")
+ .option("hoodie.datasource.compaction.async.enable", "true")
+ .option("hoodie.table.name", tableName).option("checkpointLocation", checkpointLocation)
+ .outputMode(OutputMode.Append());
+ writer.trigger(new ProcessingTime(30000)).start(tablePath);
+```
+
+##### Hudi Streamer Continuous Mode
+Hudi Streamer provides continuous ingestion mode where a single long running spark application
+ingests data to Hudi table continuously from upstream sources. In this mode, Hudi supports managing asynchronous
+compactions. Here is an example snippet for running in continuous mode with async compactions
+
+```properties
+spark-submit --packages org.apache.hudi:hudi-utilities-bundle_2.11:0.6.0 \
+--class org.apache.hudi.utilities.streamer.HoodieStreamer \
+--table-type MERGE_ON_READ \
+--target-base-path <hudi_base_path> \
+--target-table <hudi_table> \
+--source-class org.apache.hudi.utilities.sources.JsonDFSSource \
+--source-ordering-field ts \
+--props /path/to/source.properties \
+--continous
+```
+
+#### Scheduling and Execution by a separate process
+For some use cases with long running table services, instead of having the regular writes block, users have the option to run
+both steps of the compaction ([scheduling and execution](#compaction-architecture)) offline in a separate process altogether.
+This allows for regular writers to not bother about these compaction steps and allows users to provide more resources for
+the compaction job as needed.
+
+:::note
+This model needs a lock provider configured for all jobs - the regular writer as well as the offline compaction job.
+:::
+
+#### Scheduling inline and executing async
+
+In this model, it is possible for a Spark Datasource writer or a Flink job to just schedule the compaction inline ( that
+will serialize the compaction plan in the timeline but will not execute it). And then a separate utility like
+HudiCompactor or HoodieFlinkCompactor can take care of periodically executing the compaction plan.
+
+:::note
+This model may need a lock provider **if** metadata table is enabled.
+:::
+
+#### Hudi Compactor Utility
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli.md#compactions)
+The compactor utility allows to do scheduling and execution of compaction.
+
+Example:
+```properties
+spark-submit --packages org.apache.hudi:hudi-utilities-bundle_2.11:0.6.0 \
+--class org.apache.hudi.utilities.HoodieCompactor \
+--base-path <base_path> \
+--table-name <table_name> \
+--schema-file <schema_file> \
+--instant-time <compaction_instant>
+```
+
+Note, the `instant-time` parameter is now optional for the Hudi Compactor Utility. If using the utility without `--instant time`,
+the spark-submit will execute the earliest scheduled compaction on the Hudi timeline.
+
+#### Hudi CLI
+Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli.md#compactions)
+
+Example:
+```properties
+hudi:trips->compaction run --tableName <table_name> --parallelism <parallelism> --compactionInstant <InstantTime>
+...
+```
+
+#### Flink Offline Compaction
+Offline compaction needs to submit the Flink task on the command line. The program entry is as follows: `hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar` :
+`org.apache.hudi.sink.compact.HoodieFlinkCompactor`
+
+```bash
+# Command line
+./bin/flink run -c org.apache.hudi.sink.compact.HoodieFlinkCompactor lib/hudi-flink-bundle_2.11-0.9.0.jar --path hdfs://xxx:9000/table
+```
+
+#### Options
+
+| Option Name | Default | Description |
+| ----------- |-------------------------------------------------------------------------------------------------------------------------------| ------- |
+| `--path` | `n/a **(Required)**` | The path where the target table is stored on Hudi |
+| `--compaction-max-memory` | `100` (Optional) | The index map size of log data during compaction, 100 MB by default. If you have enough memory, you can turn up this parameter |
+| `--schedule` | `false` (Optional) | whether to execute the operation of scheduling compaction plan. When the write process is still writing， turning on this parameter have a risk of losing data. Therefore, it must be ensured that there are no write tasks currently writing data to this table when this parameter is turned on |
+| `--seq` | `LIFO` (Optional) | The order in which compaction tasks are executed. Executing from the latest compaction plan by default. `LIFO`: executing from the latest plan. `FIFO`: executing from the oldest plan. |
+| `--service` | `false` (Optional) | Whether to start a monitoring service that checks and schedules new compaction task in configured interval. |
+| `--min-compaction-interval-seconds` | `600(s)` (optional) | The checking interval for service mode, by default 10 minutes. |
diff --git a/website/versioned_docs/version-0.15.1/comparison.md b/website/versioned_docs/version-0.15.1/comparison.md
new file mode 100644
index 000000000000..30ededd13a83
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/comparison.md
@@ -0,0 +1,56 @@
+---
+title: "Comparison"
+keywords: [ apache, hudi, kafka, kudu, hive, hbase, stream processing]
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+
+Apache Hudi fills a big void for processing data on top of DFS, and thus mostly co-exists nicely with these technologies. However,
+it would be useful to understand how Hudi fits into the current big data ecosystem, contrasting it with a few related systems
+and bring out the different tradeoffs these systems have accepted in their design.
+
+## Kudu
+
+[Apache Kudu](https://kudu.apache.org) is a storage system that has similar goals as Hudi, which is to bring real-time analytics on petabytes of data via first
+class support for `upserts`. A key differentiator is that Kudu also attempts to serve as a datastore for OLTP workloads, something that Hudi does not aspire to be.
+Consequently, Kudu does not support incremental pulling (as of early 2017), something Hudi does to enable incremental processing use cases.
+
+
+Kudu diverges from a distributed file system abstraction and HDFS altogether, with its own set of storage servers talking to each other via RAFT.
+Hudi, on the other hand, is designed to work with an underlying Hadoop compatible filesystem (HDFS,S3 or Ceph) and does not have its own fleet of storage servers,
+instead relying on Apache Spark to do the heavy-lifting. Thus, Hudi can be scaled easily, just like other Spark jobs, while Kudu would require hardware
+& operational support, typical to datastores like HBase or Vertica. We have not at this point, done any head to head benchmarks against Kudu (given RTTable is WIP).
+But, if we were to go with results shared by [CERN](https://db-blog.web.cern.ch/blog/zbigniew-baranowski/2017-01-performance-comparison-different-file-formats-and-storage-engines) ,
+we expect Hudi to positioned at something that ingests parquet with superior performance.
+
+
+## Hive Transactions
+
+[Hive Transactions/ACID](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions) is another similar effort, which tries to implement storage like
+`merge-on-read`, on top of ORC file format. Understandably, this feature is heavily tied to Hive and other efforts like [LLAP](https://cwiki.apache.org/confluence/display/Hive/LLAP).
+Hive transactions does not offer the read-optimized storage option or the incremental pulling, that Hudi does. In terms of implementation choices, Hudi leverages
+the full power of a processing framework like Spark, while Hive transactions feature is implemented underneath by Hive tasks/queries kicked off by user or the Hive metastore.
+Based on our production experience, embedding Hudi as a library into existing Spark pipelines was much easier and less operationally heavy, compared with the other approach.
+Hudi is also designed to work with non-hive engines like PrestoDB/Spark and will incorporate file formats other than parquet over time.
+
+## HBase
+
+Even though [HBase](https://hbase.apache.org) is ultimately a key-value store for OLTP workloads, users often tend to associate HBase with analytics given the proximity to Hadoop.
+Given HBase is heavily write-optimized, it supports sub-second upserts out-of-box and Hive-on-HBase lets users query that data. However, in terms of actual performance for analytical workloads,
+hybrid columnar storage formats like Parquet/ORC handily beat HBase, since these workloads are predominantly read-heavy. Hudi bridges this gap between faster data and having
+analytical storage formats. From an operational perspective, arming users with a library that provides faster data, is more scalable, than managing a big farm of HBase region servers,
+just for analytics. Finally, HBase does not support incremental processing primitives like `commit times`, `incremental pull` as first class citizens like Hudi.
+
+## Stream Processing
+
+A popular question, we get is : "How does Hudi relate to stream processing systems?", which we will try to answer here. Simply put, Hudi can integrate with
+batch (`copy-on-write table`) and streaming (`merge-on-read table`) jobs of today, to store the computed results in Hadoop. For Spark apps, this can happen via direct
+integration of Hudi library with Spark/Spark streaming DAGs. In case of Non-Spark processing systems (eg: Flink, Hive), the processing can be done in the respective systems
+and later sent into a Hudi table via a Kafka topic/DFS intermediate file. In more conceptual level, data processing
+pipelines just consist of three components : `source`, `processing`, `sink`, with users ultimately running queries against the sink to use the results of the pipeline.
+Hudi can act as either a source or sink, that stores data on DFS. Applicability of Hudi to a given stream processing pipeline ultimately boils down to suitability
+of PrestoDB/SparkSQL/Hive for your queries.
+
+More advanced use cases revolve around the concepts of [incremental processing](https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop), which effectively
+uses Hudi even inside the `processing` engine to speed up typical batch pipelines. For e.g: Hudi can be used as a state store inside a processing DAG (similar
+to how [rocksDB](https://ci.apache.org/projects/flink/flink-docs-release-1.2/ops/state_backends.html#the-rocksdbstatebackend) is used by Flink). This is an item on the roadmap
+and will eventually happen as a [Beam Runner](https://github.com/apache/hudi/issues/14452)
diff --git a/website/versioned_docs/version-0.15.1/concepts.md b/website/versioned_docs/version-0.15.1/concepts.md
new file mode 100644
index 000000000000..8d0adf8dd5a1
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/concepts.md
@@ -0,0 +1,172 @@
+---
+version: 0.6.0
+title: "Concepts"
+keywords: [ hudi, design, table, queries, timeline]
+summary: "Here we introduce some basic concepts & give a broad technical overview of Hudi"
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+
+Apache Hudi (pronounced “Hudi”) provides the following streaming primitives over hadoop compatible storages
+
+ * Update/Delete Records (how do I change records in a table?)
+ * Change Streams (how do I fetch records that changed?)
+
+In this section, we will discuss key concepts & terminologies that are important to understand, to be able to effectively use these primitives.
+
+## Timeline
+At its core, Hudi maintains a `timeline` of all actions performed on the table at different `instants` of time that helps provide instantaneous views of the table,
+while also efficiently supporting retrieval of data in the order of arrival. A Hudi instant consists of the following components
+
+ * `Instant action` : Type of action performed on the table
+ * `Instant time` : Instant time is typically a timestamp (e.g: 20190117010349), which monotonically increases in the order of action's begin time.
+ * `state` : current state of the instant
+
+Hudi guarantees that the actions performed on the timeline are atomic & timeline consistent based on the instant time.
+
+Key actions performed include
+
+ * `COMMITS` - A commit denotes an **atomic write** of a batch of records into a table.
+ * `CLEANS` - Background activity that gets rid of older versions of files in the table, that are no longer needed.
+ * `DELTA_COMMIT` - A delta commit refers to an **atomic write** of a batch of records into a MergeOnRead type table, where some/all of the data could be just written to delta logs.
+ * `COMPACTION` - Background activity to reconcile differential data structures within Hudi e.g: moving updates from row based log files to columnar formats. Internally, compaction manifests as a special commit on the timeline
+ * `ROLLBACK` - Indicates that a commit/delta commit was unsuccessful & rolled back, removing any partial files produced during such a write
+ * `SAVEPOINT` - Marks certain file groups as "saved", such that cleaner will not delete them. It helps restore the table to a point on the timeline, in case of disaster/data recovery scenarios.
+
+Any given instant can be
+in one of the following states
+
+ * `REQUESTED` - Denotes an action has been scheduled, but has not initiated
+ * `INFLIGHT` - Denotes that the action is currently being performed
+ * `COMPLETED` - Denotes completion of an action on the timeline
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_timeline.png" alt="hudi_timeline.png" />
+</figure>
+
+Example above shows upserts happenings between 10:00 and 10:20 on a Hudi table, roughly every 5 mins, leaving commit metadata on the Hudi timeline, along
+with other background cleaning/compactions. One key observation to make is that the commit time indicates the `arrival time` of the data (10:20AM), while the actual data
+organization reflects the actual time or `event time`, the data was intended for (hourly buckets from 07:00). These are two key concepts when reasoning about tradeoffs between latency and completeness of data.
+
+When there is late arriving data (data intended for 9:00 arriving >1 hr late at 10:20), we can see the upsert producing new data into even older time buckets/folders.
+With the help of the timeline, an incremental query attempting to get all new data that was committed successfully since 10:00 hours, is able to very efficiently consume
+only the changed files without say scanning all the time buckets > 07:00.
+
+## File management
+Hudi organizes a table into a directory structure under a `basepath` on DFS. Table is broken up into partitions, which are folders containing data files for that partition,
+very similar to Hive tables. Each partition is uniquely identified by its `partitionpath`, which is relative to the basepath.
+
+Within each partition, files are organized into `file groups`, uniquely identified by a `file id`. Each file group contains several
+`file slices`, where each slice contains a base file (`*.parquet`) produced at a certain commit/compaction instant time,
+ along with set of log files (`*.log.*`) that contain inserts/updates to the base file since the base file was produced.
+Hudi adopts a MVCC design, where compaction action merges logs and base files to produce new file slices and cleaning action gets rid of
+unused/older file slices to reclaim space on DFS.
+
+## Index
+Hudi provides efficient upserts, by mapping a given hoodie key (record key + partition path) consistently to a file id, via an indexing mechanism.
+This mapping between record key and file group/file id, never changes once the first version of a record has been written to a file. In short, the
+mapped file group contains all versions of a group of records.
+
+## Table Types & Queries
+Hudi table types define how data is indexed & laid out on the DFS and how the above primitives and timeline activities are implemented on top of such organization (i.e how data is written).
+In turn, `query types` define how the underlying data is exposed to the queries (i.e how data is read).
+
+| Table Type | Supported Query types |
+|-------------- |------------------|
+| Copy On Write | Snapshot Queries + Incremental Queries |
+| Merge On Read | Snapshot Queries + Incremental Queries + Read Optimized Queries |
+
+### Table Types
+Hudi supports the following table types.
+
+ - [Copy On Write](#copy-on-write-table) : Stores data using exclusively columnar file formats (e.g parquet). Updates simply version & rewrite the files by performing a synchronous merge during write.
+ - [Merge On Read](#merge-on-read-table) : Stores data using a combination of columnar (e.g parquet) + row based (e.g avro) file formats. Updates are logged to delta files & later compacted to produce new versions of columnar files synchronously or asynchronously.
+
+Following table summarizes the trade-offs between these two table types
+
+| Trade-off | CopyOnWrite | MergeOnRead |
+|-------------- |------------------| ------------------|
+| Data Latency | Higher | Lower |
+| Update cost (I/O) | Higher (rewrite entire parquet) | Lower (append to delta log) |
+| Parquet File Size | Smaller (high update(I/0) cost) | Larger (low update cost) |
+| Write Amplification | Higher | Lower (depending on compaction strategy) |
+
+
+### Query types
+Hudi supports the following query types
+
+ - **Snapshot Queries** : Queries see the latest snapshot of the table as of a given commit or compaction action. In case of merge on read table, it exposes near-real time data(few mins) by merging
+ the base and delta files of the latest file slice on-the-fly. For copy on write table, it provides a drop-in replacement for existing parquet tables, while providing upsert/delete and other write side features.
+ - **Incremental Queries** : Queries only see new data written to the table, since a given commit/compaction. This effectively provides change streams to enable incremental data pipelines.
+ - **Read Optimized Queries** : Queries see the latest snapshot of table as of a given commit/compaction action. Exposes only the base/columnar files in latest file slices and guarantees the
+ same columnar query performance compared to a non-hudi columnar table.
+
+Following table summarizes the trade-offs between the different query types.
+
+| Trade-off | Snapshot | Read Optimized |
+|-------------- |-------------| ------------------|
+| Data Latency | Lower | Higher
+| Query Latency | Higher (merge base / columnar file + row based delta / log files) | Lower (raw base / columnar file performance)
+
+
+## Copy On Write Table
+
+File slices in Copy-On-Write table only contain the base/columnar file and each commit produces new versions of base files.
+In other words, we implicitly compact on every commit, such that only columnar data exists. As a result, the write amplification
+(number of bytes written for 1 byte of incoming data) is much higher, where read amplification is zero.
+This is a much desired property for analytical workloads, which is predominantly read-heavy.
+
+Following illustrates how this works conceptually, when data written into copy-on-write table and two queries running on top of it.
+
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_cow.png" alt="hudi_cow.png" />
+</figure>
+
+
+As data gets written, updates to existing file groups produce a new slice for that file group stamped with the commit instant time,
+while inserts allocate a new file group and write its first slice for that file group. These file slices and their commit instant times are color coded above.
+SQL queries running against such a table (eg: `select count(*)` counting the total records in that partition), first checks the timeline for the latest commit
+and filters all but latest file slices of each file group. As you can see, an old query does not see the current inflight commit's files color coded in pink,
+but a new query starting after the commit picks up the new data. Thus queries are immune to any write failures/partial writes and only run on committed data.
+
+The intention of copy on write table, is to fundamentally improve how tables are managed today through
+
+ - First class support for atomically updating data at file-level, instead of rewriting whole tables/partitions
+ - Ability to incremental consume changes, as opposed to wasteful scans or fumbling with heuristics
+ - Tight control of file sizes to keep query performance excellent (small files hurt query performance considerably).
+
+
+## Merge On Read Table
+
+Merge on read table is a superset of copy on write, in the sense it still supports read optimized queries of the table by exposing only the base/columnar files in latest file slices.
+Additionally, it stores incoming upserts for each file group, onto a row based delta log, to support snapshot queries by applying the delta log,
+onto the latest version of each file id on-the-fly during query time. Thus, this table type attempts to balance read and write amplification intelligently, to provide near real-time data.
+The most significant change here, would be to the compactor, which now carefully chooses which delta log files need to be compacted onto
+their columnar base file, to keep the query performance in check (larger delta log files would incur longer merge times with merge data on query side)
+
+Following illustrates how the table works, and shows two types of queries - snapshot query and read optimized query.
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_mor.png" alt="hudi_mor.png" />
+</figure>
+
+There are lot of interesting things happening in this example, which bring out the subtleties in the approach.
+
+ - We now have commits every 1 minute or so, something we could not do in the other table type.
+ - Within each file id group, now there is an delta log file, which holds incoming updates to the records already present in the base columnar files. In the example, the delta log files hold
+ all the data from 10:05 to 10:10. The base columnar files are still versioned with the commit, as before.
+ Thus, if one were to simply look at base files alone, then the table layout looks exactly like a copy on write table.
+ - A periodic compaction process reconciles these changes from the delta log and produces a new version of base file, just like what happened at 10:05 in the example.
+ - There are two ways of querying the same underlying table: Read Optimized query and Snapshot query, depending on whether we chose query performance or freshness of data.
+ - The semantics around when data from a commit is available to a query changes in a subtle way for a read optimized query. Note, that such a query
+ running at 10:10, wont see data after 10:05 above, while a snapshot query always sees the freshest data.
+ - When we trigger compaction & what it decides to compact hold all the key to solving these hard problems. By implementing a compacting
+ strategy, where we aggressively compact the latest partitions compared to older partitions, we could ensure the read optimized queries see data
+ published within X minutes in a consistent fashion.
+
+The intention of merge on read table is to enable near real-time processing directly on top of DFS, as opposed to copying
+data out to specialized systems, which may not be able to handle the data volume. There are also a few secondary side benefits to
+this table such as reduced write amplification by avoiding synchronous merge of data, i.e, the amount of data written per 1 bytes of data in a batch
+
+
diff --git a/website/versioned_docs/version-0.15.1/concurrency_control.md b/website/versioned_docs/version-0.15.1/concurrency_control.md
new file mode 100644
index 000000000000..6181614ea986
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/concurrency_control.md
@@ -0,0 +1,304 @@
+---
+title: "Concurrency Control"
+summary: In this page, we will discuss how to perform concurrent writes to Hudi Tables.
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+last_modified_at: 2021-03-19T15:59:57-04:00
+---
+Concurrency control defines how different writers/readers coordinate access to the table. Hudi ensures atomic writes, by way of publishing commits atomically to the timeline, stamped with an instant time that denotes the time at which the action is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between writer processes (that issue user’s upserts/deletes), table services (that write data/metadata to optimize/perform bookkeeping) and read [...]
+
+In this section, we will discuss the different concurrency controls supported by Hudi and how they are leveraged to provide flexible deployment models; we will cover multi-writing, a popular deployment model; finally, we’ll describe ways to ingest data into a Hudi Table from multiple writers using different writers, like Hudi Streamer, Hudi datasource, Spark Structured Streaming and Spark SQL.
+
+:::note
+Hudi has introduced a new concurrency mode `NON_BLOCKING_CONCURRENCY_CONTROL`, where unlike OCC, multiple writers can
+operate on the table with non-blocking conflict resolution. The writers can write into the same file group with the
+conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is currently
+available for preview in version 1.0.0-beta only. You can read more about it under section [Model C: Multi-writer](#model-c-multi-writer).
+:::
+
+
+## Deployment models with supported concurrency controls
+
+### Model A: Single writer with inline table services
+
+This is the simplest form of concurrency, meaning there is no concurrency at all in the write processes. In this model, Hudi eliminates the need for concurrency control and maximizes throughput by supporting these table services out-of-box and running inline after every write to the table. Execution plans are idempotent, persisted to the timeline and auto-recover from failures. For most simple use-cases, this means just writing is sufficient to get a well-managed table that needs no conc [...]
+
+There is no actual concurrent writing in this model. **MVCC** is leveraged to provide snapshot isolation guarantees between ingestion writer and multiple readers and also between multiple table service writers and readers. Writes to the table either from ingestion or from table services produce versioned data that are available to readers only after the writes are committed. Until then, readers can access only the previous version of the data.
+
+A single writer with all table services such as cleaning, clustering, compaction, etc can be configured to be inline (such as Hudi Streamer sync-once mode and Spark Datasource with default configs) without any additional configs.
+
+#### Single Writer Guarantees
+
+In this model, the following are the guarantees on [write operations](https://hudi.apache.org/docs/write_operations/) to expect:
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table wilL NEVER have duplicates if dedup: [`hoodie.datasource.write.insert.drop.duplicates`](https://hudi.apache.org/docs/configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](https://hudi.apache.org/docs/configurations/#hoodiecombinebeforeinsert), is enabled.
+- *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if dedup: [`hoodie.datasource.write.insert.drop.duplicates`](https://hudi.apache.org/docs/configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](https://hudi.apache.org/docs/configurations/#hoodiecombinebeforeinsert), is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order.
+
+
+### Model B: Single writer with async table services
+
+Hudi provides the option of running the table services in an async fashion, where most of the heavy lifting (e.g actually rewriting the columnar data by compaction service) is done asynchronously. In this model, the async deployment eliminates any repeated wasteful retries and optimizes the table using clustering techniques while a single writer consumes the writes to the table without having to be blocked by such table services. This model avoids the need for taking an [external lock](# [...]
+
+A single writer along with async table services runs in the same process. For example, you can have a Hudi Streamer in continuous mode write to a MOR table using async compaction; you can use Spark Streaming (where [compaction](https://hudi.apache.org/docs/compaction) is async by default), and you can use Flink streaming or your own job setup and enable async table services inside the same writer.
+
+Hudi leverages **MVCC** in this model to support running any number of table service jobs concurrently, without any concurrency conflict. This is made possible by ensuring Hudi 's ingestion writer and async table services coordinate among themselves to ensure no conflicts and avoid race conditions. The same single write guarantees described in Model A above can be achieved in this model as well.
+With this model users don't need to spin up different spark jobs and manage the orchestration among it. For larger deployments, this model can ease the operational burden significantly while getting the table services running without blocking the writers.
+
+### Model C: Multi-writer
+
+It is not always possible to serialize all write operations to a table (such as UPSERT, INSERT or DELETE) into the same write process and therefore, multi-writing capability may be required. In multi-writing, disparate distributed processes run in parallel or overlapping time windows to write to the same table. In such cases, an external locking mechanism becomes necessary to coordinate concurrent accesses. Here are few different scenarios that would all fall under multi-writing:
+
+- Multiple ingestion writers to the same table:For instance, two Spark Datasource writers working on different sets of partitions form a source kafka topic.
+- Multiple ingestion writers to the same table, including one writer with async table services: For example, a Hudi Streamer with async compaction for regular ingestion & a Spark Datasource writer for backfilling.
+- A single ingestion writer and a separate compaction (HoodieCompactor) or clustering (HoodieClusteringJob) job apart from the ingestion writer: This is considered as multi-writing as they are not running in the same process.
+
+Hudi's concurrency model intelligently differentiates actual writing to the table from table services that manage or optimize the table. Hudi offers similar **optimistic concurrency control across multiple writers**, but **table services can still execute completely lock-free and async** as long as they run in the same process as one of the writers.
+For multi-writing, Hudi leverages file level optimistic concurrency control(OCC). For example, when two writers write to non overlapping files, both writes are allowed to succeed. However, when the writes from different writers overlap (touch the same set of files), only one of them will succeed. Please note that this feature is currently experimental and requires external lock providers to acquire locks briefly at critical sections during the write. More on lock providers below.
+
+#### Multi Writer Guarantees
+
+With multiple writers using OCC, these are the write guarantees to expect:
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if dedup is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if dedup is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order. If there are inflight commits
+ (due to multi-writing), incremental queries will not expose the completed commits following the inflight commits.
+
+#### Non-Blocking Concurrency Control Mode (Experimental)
+
+`NON_BLOCKING_CONCURRENCY_CONTROL`, offers the same set of guarantees as mentioned in the case of OCC but without
+explicit locks for serializing the writes. Lock is only needed for writing the commit metadata to the Hudi timeline. The
+completion time for the commits reflects the serialization order and file slicing is done based on completion time.
+Multiple writers can operate on the table with non-blocking conflict resolution. The writers can write into the same
+file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
+currently available for preview in version 1.0.0-beta only with the caveat that conflict resolution is not supported yet
+between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
+writers [here](sql_dml.md#non-blocking-concurrency-control-experimental).
+
+## Enabling Multi Writing
+
+The following properties are needed to be set appropriately to turn on optimistic concurrency control to achieve multi writing.
+
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.write.lock.provider=<lock-provider-classname>
+hoodie.cleaner.policy.failed.writes=LAZY
+```
+
+| Config Name | Default | Description [...]
+|-------------------------------------|-------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.write.concurrency.mode | SINGLE_WRITER (Optional) | <u>[Concurrency modes](https://github.com/apache/hudi/blob/00ece7bce0a4a8d0019721a28049723821e01842/hudi-common/src/main/java/org/apache/hudi/common/model/WriteConcurrencyMode.java)</u> for write operations.<br />Possible values:<br /><ul><li>`SINGLE_WRITER`: Only one active writer to the table. Maximizes throughput.</li><li>`OPTIMISTIC_CONCURRENCY_CONTROL`: Multiple wr [...]
+| hoodie.write.lock.provider | org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider (Optional) | Lock provider class name, user can provide their own implementation of LockProvider which should be subclass of org.apache.hudi.common.lock.LockProvider<br /><br />`Config Param: LOCK_PROVIDER_CLASS_NAME`<br />`Since Version: 0.8.0` [...]
+| hoodie.cleaner.policy.failed.writes | EAGER (Optional) | org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy: Policy that controls how to clean up failed writes. Hudi will delete any files written by failed writes to re-claim space. EAGER(default): Clean failed writes inline after every write operation. LAZY: Clean failed writes lazily after heartbeat timeout when the cleaning service runs. This policy is re [...]
+
+
+### External Locking and lock providers
+
+As can be seen above, a lock provider needs to be configured in muti-writing scenarios. External locking is typically used in conjunction with optimistic concurrency control because it provides a way to prevent conflicts that might occur when two or more transactions (commits in our case) attempt to modify the same resource concurrently. When a transaction attempts to modify a resource that is currently locked by another transaction, it must wait until the lock is released before proceeding.
+
+In case of multi-writing in Hudi, the locks are acquired on the Hudi table for a very short duration during specific phases (such as just before committing the writes or before scheduling table services) instead of locking for the entire span of time. This approach allows multiple writers to work on the same table simultaneously, increasing concurrency and avoids conflicts.
+
+There are 4 different lock providers that require different configurations to be set. Please refer to comprehensive locking configs [here](https://hudi.apache.org/docs/next/configurations#LOCK).
+
+#### Zookeeper based lock provider
+```
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
+```
+Following are the basic configs required to setup this lock provider:
+
+| Config Name| Default| Description |
+| ---------------------------------------------------------------------------- | ------------------------ |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.write.lock.zookeeper.base_path | N/A **(Required)** | The base path on Zookeeper under which to create lock related ZNodes. This should be same for all concurrent writers to the same table<br /><br />`Config Param: ZK_BASE_PATH`<br />`Since Version: 0.8.0` |
+| hoodie.write.lock.zookeeper.port | N/A **(Required)** | Zookeeper port to connect to.<br /><br />`Config Param: ZK_PORT`<br />`Since Version: 0.8.0` |
+| hoodie.write.lock.zookeeper.url | N/A **(Required)** | Zookeeper URL to connect to.<br /><br />`Config Param: ZK_CONNECT_URL`<br />`Since Version: 0.8.0` |
+
+#### HiveMetastore based lock provider
+
+```
+hoodie.write.lock.provider=org.apache.hudi.hive.transaction.lock.HiveMetastoreBasedLockProvider
+```
+Following are the basic configs required to setup this lock provider:
+
+| Config Name| Default| Description |
+| ----------------------------------------------------------------------- | ------------------------ |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.write.lock.hivemetastore.database | N/A **(Required)** | For Hive based lock provider, the Hive database to acquire lock against<br /><br />`Config Param: HIVE_DATABASE_NAME`<br />`Since Version: 0.8.0` |
+| hoodie.write.lock.hivemetastore.table | N/A **(Required)** | For Hive based lock provider, the Hive table to acquire lock against<br /><br />`Config Param: HIVE_TABLE_NAME`<br />`Since Version: 0.8.0` |
+
+`The HiveMetastore URI's are picked up from the hadoop configuration file loaded during runtime.`
+
+#### Amazon DynamoDB based lock provider
+```
+hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
+```
+Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters. You can refer to the
+[DynamoDB based Locks Configurations](https://hudi.apache.org/docs/configurations#DynamoDB-based-Locks-Configurations)
+section for the details of each related configuration knob. Following are the basic configs required to setup this lock provider:
+
+| Config Name| Default| Description |
+| ----------------------------------------------------------------------- | ------------------------ |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.write.lock.dynamodb.endpoint_url| N/A **(Required)** | For DynamoDB based lock provider, the url endpoint used for Amazon DynamoDB service. Useful for development with a local dynamodb instance.<br /><br />`Config Param: DYNAMODB_ENDPOINT_URL`<br />`Since Version: 0.10.1`|
+
+For advanced configs refer [here](https://hudi.apache.org/docs/next/configurations#DynamoDB-based-Locks-Configurations)
+
+
+When using the DynamoDB-based lock provider, the name of the DynamoDB table acting as the lock table for Hudi is
+specified by the config `hoodie.write.lock.dynamodb.table`. This DynamoDB table is automatically created by Hudi, so you
+don't have to create the table yourself. If you want to use an existing DynamoDB table, make sure that an attribute with
+the name `key` is present in the table. The `key` attribute should be the partition key of the DynamoDB table. The
+config `hoodie.write.lock.dynamodb.partition_key` specifies the value to put for the `key` attribute (not the attribute
+name), which is used for the lock on the same table. By default, `hoodie.write.lock.dynamodb.partition_key` is set to
+the table name, so that multiple writers writing to the same table share the same lock. If you customize the name, make
+sure it's the same across multiple writers.
+
+Also, to set up the credentials for accessing AWS resources, customers can pass the following props to Hudi jobs:
+```
+hoodie.aws.access.key
+hoodie.aws.secret.key
+hoodie.aws.session.token
+```
+If not configured, Hudi falls back to use [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html).
+
+IAM policy for your service instance will need to add the following permissions:
+
+```json
+{
+ "Sid":"DynamoDBLocksTable",
+ "Effect": "Allow",
+ "Action": [
+ "dynamodb:CreateTable",
+ "dynamodb:DeleteItem",
+ "dynamodb:DescribeTable",
+ "dynamodb:GetItem",
+ "dynamodb:PutItem",
+ "dynamodb:Scan",
+ "dynamodb:UpdateItem"
+ ],
+ "Resource": "arn:${Partition}:dynamodb:${Region}:${Account}:table/${TableName}"
+}
+```
+- `TableName` : same as `hoodie.write.lock.dynamodb.partition_key`
+- `Region`: same as `hoodie.write.lock.dynamodb.region`
+
+AWS SDK dependencies are not bundled with Hudi from v0.10.x and will need to be added to your classpath.
+Add the following Maven packages (check the latest versions at time of install):
+```
+com.amazonaws:dynamodb-lock-client
+com.amazonaws:aws-java-sdk-dynamodb
+com.amazonaws:aws-java-sdk-core
+```
+
+#### FileSystem based lock provider (Experimental)
+
+FileSystem based lock provider supports multiple writers cross different jobs/applications based on atomic create/delete operations of the underlying filesystem.
+
+```
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.FileSystemBasedLockProvider
+```
+
+When using the FileSystem based lock provider, by default, the lock file will store into `hoodie.base.path`+`/.hoodie/lock`. You may use a custom folder to store the lock file by specifying `hoodie.write.lock.filesystem.path`.
+
+In case the lock cannot release during job crash, you can set `hoodie.write.lock.filesystem.expire` (lock will never expire by default) to a desired expire time in minutes. You may also delete lock file manually in such situation.
+:::note
+FileSystem based lock provider is not supported with cloud storage like S3 or GCS.
+:::
+
+## Multi Writing via Spark Datasource Writer
+
+The `hudi-spark` module offers the DataSource API to write (and read) a Spark DataFrame into a Hudi table.
+
+Following is an example of how to use optimistic_concurrency_control via spark datasource
+
+```java
+inputDF.write.format("hudi")
+ .options(getQuickstartWriteConfigs)
+ .option("hoodie.datasource.write.precombine.field", "ts")
+ .option("hoodie.cleaner.policy.failed.writes", "LAZY")
+ .option("hoodie.write.concurrency.mode", "optimistic_concurrency_control")
+ .option("hoodie.write.lock.zookeeper.url", "zookeeper")
+ .option("hoodie.write.lock.zookeeper.port", "2181")
+ .option("hoodie.write.lock.zookeeper.base_path", "/test")
+ .option("hoodie.datasource.write.recordkey.field", "uuid")
+ .option("hoodie.datasource.write.partitionpath.field", "partitionpath")
+ .option("hoodie.table.name", tableName)
+ .mode(Overwrite)
+ .save(basePath)
+```
+
+## Multi Writing via Hudi Streamer
+
+The `HoodieStreamer` utility (part of hudi-utilities-bundle) provides ways to ingest from different sources such as DFS or Kafka, with the following capabilities.
+
+Using optimistic_concurrency_control via Hudi Streamer requires adding the above configs to the properties file that can be passed to the
+job. For example below, adding the configs to kafka-source.properties file and passing them to Hudi Streamer will enable optimistic concurrency.
+A Hudi Streamer job can then be triggered as follows:
+
+```java
+[hoodie]$ spark-submit --class org.apache.hudi.utilities.streamer.HoodieStreamer `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` \
+ --props file://${PWD}/hudi-utilities/src/test/resources/streamer-config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --target-base-path file:\/\/\/tmp/hudi-streamer-op \
+ --target-table tableName \
+ --op BULK_INSERT
+```
+
+## Early conflict Detection
+
+Multi writing using OCC allows multiple writers to concurrently write and atomically commit to the Hudi table if there is no overlapping data file to be written, to guarantee data consistency, integrity and correctness. Prior to 0.13.0 release, as the OCC (optimistic concurrency control) name suggests, each writer will optimistically proceed with ingestion and towards the end, just before committing will go about conflict resolution flow to deduce overlapping writes and abort one if need [...]
+
+To improve the concurrency control, the [0.13.0 release](https://hudi.apache.org/releases/release-0.13.0#early-conflict-detection-for-multi-writer) introduced a new feature, early conflict detection in OCC, to detect the conflict during the data writing phase and abort the writing early on once a conflict is detected, using Hudi's marker mechanism. Hudi can now stop a conflicting writer much earlier because of the early conflict detection and release computing resources necessary to clus [...]
+
+By default, this feature is turned off. To try this out, a user needs to set `hoodie.write.concurrency.early.conflict.detection.enable` to true, when using OCC for concurrency control (Refer [configs](https://hudi.apache.org/docs/next/configurations#Write-Configurations-advanced-configs) page for all relevant configs).
+:::note
+Early conflict Detection in OCC is an **EXPERIMENTAL** feature
+:::
+
+## Best Practices when using Optimistic Concurrency Control
+
+Concurrent Writing to Hudi tables requires acquiring a lock with one of the lock providers mentioned above. Due to several reasons you might want to configure retries to allow your application to acquire the lock.
+1. Network connectivity or excessive load on servers increasing time for lock acquisition resulting in timeouts
+2. Running a large number of concurrent jobs that are writing to the same hudi table can result in contention during lock acquisition can cause timeouts
+3. In some scenarios of conflict resolution, Hudi commit operations might take upto 10's of seconds while the lock is being held. This can result in timeouts for other jobs waiting to acquire a lock.
+
+Set the correct native lock provider client retries.
+:::note
+Please note that sometimes these settings are set on the server once and all clients inherit the same configs. Please check your settings before enabling optimistic concurrency.
+:::
+
+```
+hoodie.write.lock.wait_time_ms
+hoodie.write.lock.num_retries
+```
+
+Set the correct hudi client retries for Zookeeper & HiveMetastore. This is useful in cases when native client retry settings cannot be changed. Please note that these retries will happen in addition to any native client retries that you may have set.
+
+```
+hoodie.write.lock.client.wait_time_ms
+hoodie.write.lock.client.num_retries
+```
+
+*Setting the right values for these depends on a case by case basis; some defaults have been provided for general cases.*
+
+## Disabling Multi Writing
+
+Remove the following settings that were used to enable multi-writer or override with default values.
+
+```
+hoodie.write.concurrency.mode=single_writer
+hoodie.cleaner.policy.failed.writes=EAGER
+```
+
+## Caveats
+
+If you are using the `WriteClient` API, please note that multiple writes to the table need to be initiated from 2 different instances of the write client.
+It is **NOT** recommended to use the same instance of the write client to perform multi writing.
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Hands on Lab with using DynamoDB as lock table for Apache Hudi Data Lakes](https://youtu.be/JP0orl9_0yQ)
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.15.1/configurations.md b/website/versioned_docs/version-0.15.1/configurations.md
new file mode 100644
index 000000000000..d90509a4e147
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/configurations.md
@@ -0,0 +1,2187 @@
+---
+title: All Configurations
+keywords: [ configurations, default, flink options, spark, configs, parameters ]
+permalink: /docs/configurations.html
+summary: This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+last_modified_at: 2024-06-06T12:05:06.405
+---
+
+
+This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
+
+- [**Hudi Table Config**](#TABLE_CONFIG): Basic Hudi Table configuration parameters.
+- [**Environment Config**](#ENVIRONMENT_CONFIG): Hudi supports passing configurations via a configuration file `hudi-defaults.conf` in which each line consists of a key and a value separated by whitespace or = sign. For example:
+```
+hoodie.datasource.hive_sync.mode jdbc
+hoodie.datasource.hive_sync.jdbcurl jdbc:hive2://localhost:10000
+hoodie.datasource.hive_sync.support_timestamp false
+```
+It helps to have a central configuration file for your common cross job configurations/tunings, so all the jobs on your cluster can utilize it. It also works with Spark SQL DML/DDL, and helps avoid having to pass configs inside the SQL statements.
+
+By default, Hudi would load the configuration file under `/etc/hudi/conf` directory. You can specify a different configuration directory location by setting the `HUDI_CONF_DIR` environment variable.
+- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
+- [**Flink Sql Configs**](#FLINK_SQL): These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
+- [**Write Client Configs**](#WRITE_CLIENT): Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.
+- [**Reader Configs**](#READER): Please fill in the description for Config Group Name: Reader Configs
+- [**Metastore and Catalog Sync Configs**](#META_SYNC): Configurations used by the Hudi to sync metadata to external metastores and catalogs.
+- [**Metrics Configs**](#METRICS): These set of configs are used to enable monitoring and reporting of key Hudi stats and metrics.
+- [**Record Payload Config**](#RECORD_PAYLOAD): This is the lowest level of customization offered by Hudi. Record payloads define how to produce new values to upsert based on incoming new record and stored old record. Hudi provides default implementations such as OverwriteWithLatestAvroPayload which simply update table with the latest/last-written record. This can be overridden to a custom class extending HoodieRecordPayload class, on both datasource and WriteClient levels.
+- [**Kafka Connect Configs**](#KAFKA_CONNECT): These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables
+- [**Amazon Web Services Configs**](#AWS): Configurations specific to Amazon Web Services.
+- [**Hudi Streamer Configs**](#HUDI_STREAMER): These set of configs are used for Hudi Streamer utility which provides the way to ingest from different sources such as DFS or Kafka.
+
+:::note
+In the tables below **(N/A)** means there is no default value set
+:::
+
+## Externalized Config File
+Instead of directly passing configuration settings to every Hudi job, you can also centrally set them in a configuration
+file `hudi-defaults.conf`. By default, Hudi would load the configuration file under `/etc/hudi/conf` directory. You can
+specify a different configuration directory location by setting the `HUDI_CONF_DIR` environment variable. This can be
+useful for uniformly enforcing repeated configs (like Hive sync or write/index tuning), across your entire data lake.
+
+## Hudi Table Config {#TABLE_CONFIG}
+Basic Hudi Table configuration parameters.
+
+
+### Hudi Table Basic Configs {#Hudi-Table-Basic-Configs}
+Configurations of the Hudi Table like type of ingestion, storage formats, hive table name etc. Configurations are loaded from hoodie.properties, these properties are usually set during initializing a path as hoodie base path and never changes during the lifetime of a hoodie table.
+
+
+
+[**Basic Configs**](#Hudi-Table-Basic-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BOOTSTRAP_BASE_PATH` [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name that will be used for incremental query.If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME` [...]
+| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.<br />`Config Param: TABLE_CHECKSUM`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.create.schema](#hoodietablecreateschema) | (N/A) | Schema used when creating the table, for the first time.<br />`Config Param: CREATE_SCHEMA` [...]
+| [hoodie.table.keygenerator.class](#hoodietablekeygeneratorclass) | (N/A) | Key Generator class property for the hoodie table<br />`Config Param: KEY_GENERATOR_CLASS_NAME` [...]
+| [hoodie.table.metadata.partitions](#hoodietablemetadatapartitions) | (N/A) | Comma-separated list of metadata partitions that have been completely built and in-sync with data table. These partitions are ready for use by the readers<br />`Config Param: TABLE_METADATA_PARTITIONS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.metadata.partitions.inflight](#hoodietablemetadatapartitionsinflight) | (N/A) | Comma-separated list of metadata partitions whose building is in progress. These partitions are not yet ready for use by the readers.<br />`Config Param: TABLE_METADATA_PARTITIONS_INFLIGHT`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: NAME` [...]
+| [hoodie.table.partition.fields](#hoodietablepartitionfields) | (N/A) | Fields used to partition the table. Concatenated values of these fields are used as the partition path, by invoking toString()<br />`Config Param: PARTITION_FIELDS` [...]
+| [hoodie.table.precombine.field](#hoodietableprecombinefield) | (N/A) | Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.table.recordkey.fields](#hoodietablerecordkeyfields) | (N/A) | Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br />`Config Param: RECORDKEY_FIELDS` [...]
+| [hoodie.table.secondary.indexes.metadata](#hoodietablesecondaryindexesmetadata) | (N/A) | The metadata of secondary indexes<br />`Config Param: SECONDARY_INDEXES_METADATA`<br />`Since Version: 0.13.0` [...]
+| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | (N/A) | Version of timeline used, by the table.<br />`Config Param: TIMELINE_LAYOUT_VERSION` [...]
+| [hoodie.archivelog.folder](#hoodiearchivelogfolder) | archived | path under the meta folder, to store archived timeline instants at.<br />`Config Param: ARCHIVELOG_FOLDER` [...]
+| [hoodie.bootstrap.index.class](#hoodiebootstrapindexclass) | org.apache.hudi.common.bootstrap.index.hfile.HFileBootstrapIndex | Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_CLASS_NAME` [...]
+| [hoodie.bootstrap.index.enable](#hoodiebootstrapindexenable) | true | Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined, default true.<br />`Config Param: BOOTSTRAP_INDEX_ENABLE` [...]
+| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | org.apache.hudi.common.model.OverwriteWithLatestAvroPayload | Payload class to use for performing compactions, i.e merge delta logs with current base file and then produce a new base file.<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
+| [hoodie.compaction.record.merger.strategy](#hoodiecompactionrecordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in hoodie.datasource.write.record.merger.impls which has the same merger strategy id<br />`Config Param: RECORD_MERGER_STRATEGY`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` [...]
+| [hoodie.partition.metafile.use.base.format](#hoodiepartitionmetafileusebaseformat) | false | If true, partition metafiles are saved in the same format as base-files for this dataset (e.g. Parquet / ORC). If false (default) partition metafiles are saved as properties files.<br />`Config Param: PARTITION_METAFILE_USE_BASE_FORMAT` [...]
+| [hoodie.populate.meta.fields](#hoodiepopulatemetafields) | true | When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br />`Config Param: POPULATE_META_FIELDS` [...]
+| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | Base file format to store all the base file data.<br />`Config Param: BASE_FILE_FORMAT` [...]
+| [hoodie.table.cdc.enabled](#hoodietablecdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode.<br />`Config Param: CDC_ENABLED`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.cdc.supplemental.logging.mode](#hoodietablecdcsupplementalloggingmode) | DATA_BEFORE_AFTER | org.apache.hudi.common.table.cdc.HoodieCDCSupplementalLoggingMode: Change log capture supplemental logging mode. The supplemental log is used for accelerating the generation of change log details. OP_KEY_ONLY: Only keeping record keys in the supplemental logs, so the reader needs to figure out the update before image [...]
+| [hoodie.table.log.file.format](#hoodietablelogfileformat) | HOODIE_LOG | Log format used for the delta logs.<br />`Config Param: LOG_FILE_FORMAT` [...]
+| [hoodie.table.timeline.timezone](#hoodietabletimelinetimezone) | LOCAL | User can set hoodie commit timeline timezone, such as utc, local and so on. local is default<br />`Config Param: TIMELINE_TIMEZONE` [...]
+| [hoodie.table.type](#hoodietabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TYPE` [...]
+| [hoodie.table.version](#hoodietableversion) | ZERO | Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br />`Config Param: VERSION` [...]
+
+[**Advanced Configs**](#Hudi-Table-Basic-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.write.drop.partition.columns](#hoodiedatasourcewritedroppartitioncolumns) | false | When set to true, will not write the partition columns into hudi. By default, false.<br />`Config Param: DROP_PARTITION_COLUMNS` |
+| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Should we url encode the partition path value, before creating the folder structure.<br />`Config Param: URL_ENCODE_PARTITIONING` |
+---
+
+## Spark Datasource Configs {#SPARK_DATASOURCE}
+These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
+
+
+### Read Options {#Read-Options}
+Options useful for reading tables via `read.format.option(...)`
+
+
+
+
+[**Basic Configs**](#Read-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.read.begin.instanttime](#hoodiedatasourcereadbegininstanttime) | (N/A) | Required when `hoodie.datasource.query.type` is set to `incremental`. Represents the instant time to start incrementally pulling data from. The instanttime here need not necessarily correspond to an instant on the timeline. New data written with an instant_time > BEGIN_INSTANTTIME are fetched out. For e.g: ‘20170901080000’ will get all new data written after Sep 1, 2017 08:00AM. Note that [...]
+| [hoodie.datasource.read.end.instanttime](#hoodiedatasourcereadendinstanttime) | (N/A) | Used when `hoodie.datasource.query.type` is set to `incremental`. Represents the instant time to limit incrementally fetched data to. When not specified latest commit time from timeline is assumed by default. When specified, new data written with an instant_time <= END_INSTANTTIME are fetched out. Point in time type queries make more sense with begin and end instant times specified. Note [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files)<br />`Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: READ_PRE_COMBINE_FIELD` [...]
+
+[**Advanced Configs**](#Read-Options-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [as.of.instant](#asofinstant) | (N/A) | The query instant for time travel. Without specified this option, we query the latest snapshot.<br />`Config Param: TIME_TRAVEL_AS_OF_INSTANT` [...]
+| [hoodie.datasource.read.paths](#hoodiedatasourcereadpaths) | (N/A) | Comma separated list of file paths to read within a Hudi table.<br />`Config Param: READ_PATHS` [...]
+| [hoodie.datasource.merge.type](#hoodiedatasourcemergetype) | payload_combine | For Snapshot query on merge on read table, control whether we invoke the record payload implementation to merge (payload_combine) or skip merging altogetherskip_merge<br />`Config Param: REALTIME_MERGE` [...]
+| [hoodie.datasource.query.incremental.format](#hoodiedatasourcequeryincrementalformat) | latest_state | This config is used alone with the 'incremental' query type.When set to 'latest_state', it returns the latest records' values.When set to 'cdc', it returns the cdc data.<br />`Config Param: INCREMENTAL_FORMAT`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.read.extract.partition.values.from.path](#hoodiedatasourcereadextractpartitionvaluesfrompath) | false | When set to true, values for partition columns (partition values) will be extracted from physical partition path (default Spark behavior). When set to false partition values will be read from the data file (in Hudi partition columns are persisted by default). This config is a fallback allowing to preserve exis [...]
+| [hoodie.datasource.read.file.index.listing.mode](#hoodiedatasourcereadfileindexlistingmode) | lazy | Overrides Hudi's file-index implementation's file listing mode: when set to 'eager', file-index will list all partition paths and corresponding file slices w/in them eagerly, during initialization, prior to partition-pruning kicking in, meaning that all partitions will be listed including ones that might be s [...]
+| [hoodie.datasource.read.file.index.listing.partition-path-prefix.analysis.enabled](#hoodiedatasourcereadfileindexlistingpartition-path-prefixanalysisenabled) | true | Controls whether partition-path prefix analysis is enabled w/in the file-index, allowing to avoid necessity to recursively list deep folder structures of partitioned tables w/ multiple partition columns, by carefully analyzing provided partition-column predicates and deducing corresponding partition-path prefi [...]
+| [hoodie.datasource.read.incr.fallback.fulltablescan.enable](#hoodiedatasourcereadincrfallbackfulltablescanenable) | false | When doing an incremental query whether we should fall back to full table scans if file does not exist.<br />`Config Param: INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN` [...]
+| [hoodie.datasource.read.incr.filters](#hoodiedatasourcereadincrfilters) | | For use-cases like DeltaStreamer which reads from Hoodie Incremental table and applies opaque map functions, filters appearing late in the sequence of transformations cannot be automatically pushed down. This option allows setting filters directly on Hoodie Source.<br />`Config Param: PUSH_DOWN_INCR_FILTERS` [...]
+| [hoodie.datasource.read.incr.path.glob](#hoodiedatasourcereadincrpathglob) | | For the use-cases like users only want to incremental pull from certain partitions instead of the full table. This option allows using glob pattern to directly filter on path.<br />`Config Param: INCR_PATH_GLOB` [...]
+| [hoodie.datasource.read.schema.use.end.instanttime](#hoodiedatasourcereadschemauseendinstanttime) | false | Uses end instant schema when incrementally fetched data to. Default: users latest instant schema.<br />`Config Param: INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME` [...]
+| [hoodie.datasource.read.use.new.parquet.file.format](#hoodiedatasourcereadusenewparquetfileformat) | false | Read using the new Hudi parquet file format. The new Hudi parquet file format is introduced as an experimental feature in 0.14.0. Currently, the new Hudi parquet file format only applies to bootstrap and MOR queries. Schema evolution is also not supported by the new file format.<br />`Config Param: USE_NEW_HUD [...]
+| [hoodie.datasource.streaming.startOffset](#hoodiedatasourcestreamingstartOffset) | earliest | Start offset to pull data from hoodie streaming source. allow earliest, latest, and specified start instant time<br />`Config Param: START_OFFSET`<br />`Since Version: 0.13.0` [...]
+| [hoodie.enable.data.skipping](#hoodieenabledataskipping) | false | Enables data-skipping allowing queries to leverage indexes to reduce the search space by skipping over files<br />`Config Param: ENABLE_DATA_SKIPPING`<br />`Since Version: 0.10.0` [...]
+| [hoodie.file.index.enable](#hoodiefileindexenable) | true | Enables use of the spark file index implementation for Hudi, that speeds up listing of large tables.<br />`Config Param: ENABLE_HOODIE_FILE_INDEX` [...]
+| [hoodie.read.timeline.holes.resolution.policy](#hoodiereadtimelineholesresolutionpolicy) | FAIL | When doing incremental queries, there could be hollow commits (requested or inflight commits that are not the latest) that are produced by concurrent writers and could lead to potential data loss. This config allows users to have different ways of handling this situation. The valid values are [FAIL, BLOCK, USE [...]
+| [hoodie.schema.on.read.enable](#hoodieschemaonreadenable) | false | Enables support for Schema Evolution feature<br />`Config Param: SCHEMA_EVOLUTION_ENABLED` [...]
+---
+
+
+### Write Options {#Write-Options}
+You can pass down any of the WriteClient level configs directly using `options()` or `option(k,v)` methods.
+
+```java
+inputDF.write()
+.format("org.apache.hudi")
+.options(clientOpts) // any of the Hudi client opts can be passed in as well
+.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")
+.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")
+.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")
+.option(HoodieWriteConfig.TABLE_NAME, tableName)
+.mode(SaveMode.Append)
+.save(basePath);
+```
+
+Options useful for writing tables via `write.format.option(...)`
+
+
+
+
+[**Basic Configs**](#Write-Options-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD` |
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD` |
+| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING` |
+| [hoodie.datasource.write.operation](#hoodiedatasourcewriteoperation) | upsert | Whether to do upsert, insert or bulk_insert for the write operation. Use bulk_insert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br />`Config Param: OPERATION` |
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD` |
+| [hoodie.datasource.write.table.type](#hoodiedatasourcewritetabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TABLE_TYPE` |
+
+[**Advanced Configs**](#Write-Options-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` [...]
+| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` [...]
+| [hoodie.datasource.overwrite.mode](#hoodiedatasourceoverwritemode) | (N/A) | Controls whether overwrite use dynamic or static mode, if not configured, respect spark.sql.sources.partitionOverwriteMode<br />`Config Param: OVERWRITE_MODE`<br />`Since Version: 0.14.0` [...]
+| [hoodie.datasource.write.partitions.to.delete](#hoodiedatasourcewritepartitionstodelete) | (N/A) | Comma separated list of partitions to delete. Allows use of wildcard *<br />`Config Param: PARTITIONS_TO_DELETE` [...]
+| [hoodie.datasource.write.table.name](#hoodiedatasourcewritetablename) | (N/A) | Table name for the datasource write. Also used to register the table into meta stores.<br />`Config Param: TABLE_NAME` [...]
+| [hoodie.datasource.compaction.async.enable](#hoodiedatasourcecompactionasyncenable) | true | Controls whether async compaction should be turned on for MOR table writing.<br />`Config Param: ASYNC_COMPACT_ENABLE` [...]
+| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: HIVE_ASSUME_DATE_PARTITION` [...]
+| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` [...]
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: HIVE_BASE_FILE_FORMAT` [...]
+| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` [...]
+| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` [...]
+| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` [...]
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: HIVE_DATABASE` [...]
+| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` [...]
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: HIVE_PARTITION_EXTRACTOR_CLASS` [...]
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: HIVE_PARTITION_FIELDS` [...]
+| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` [...]
+| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` [...]
+| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` [...]
+| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` [...]
+| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` [...]
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: HIVE_TABLE` [...]
+| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` [...]
+| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Co [...]
+| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` [...]
+| [hoodie.datasource.insert.dup.policy](#hoodiedatasourceinsertduppolicy) | none | **Note** This is only applicable to Spark SQL writing.<br />When operation type is set to "insert", users can optionally enforce a dedup policy. This policy will be employed when records being ingested already exists in storage. Default policy is none and no action will be [...]
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: HIVE_CONDITIONAL_SYNC` [...]
+| [hoodie.datasource.write.commitmeta.key.prefix](#hoodiedatasourcewritecommitmetakeyprefix) | _ | Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata. This is useful to store checkpointing information, in a consistent way with the hudi timeline<br />`Config Param: COMMIT_METADATA_KEYPREFIX` [...]
+| [hoodie.datasource.write.drop.partition.columns](#hoodiedatasourcewritedroppartitioncolumns) | false | When set to true, will not write the partition columns into hudi. By default, false.<br />`Config Param: DROP_PARTITION_COLUMNS` [...]
+| [hoodie.datasource.write.insert.drop.duplicates](#hoodiedatasourcewriteinsertdropduplicates) | false | If set to true, records from the incoming dataframe will not overwrite existing records with the same key during the write operation. <br /> **Note** Just for Insert operation in Spark SQL writing since 0.14.0, users can switch to the config `hoodie.datasource.insert.dup.po [...]
+| [hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass) | org.apache.hudi.keygen.SimpleKeyGenerator | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`<br />`Config Param: KEYGENERATOR_CLASS_NAME` [...]
+| [hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled](#hoodiedatasourcewritekeygeneratorconsistentlogicaltimestampenabled) | false | When set to true, consistent value will be generated for a logical timestamp type column, like timestamp-millis and timestamp-micros, irrespective of whether row-writer is enabled. Disabled by default so as not to break the pipeline that deploy either fully row-writer path or non [...]
+| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Should we url encode the partition path value, before creating the folder structure.<br />`Config Param: URL_ENCODE_PARTITIONING` [...]
+| [hoodie.datasource.write.payload.class](#hoodiedatasourcewritepayloadclass) | org.apache.hudi.common.model.OverwriteWithLatestAvroPayload | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
+| [hoodie.datasource.write.reconcile.schema](#hoodiedatasourcewritereconcileschema) | false | This config controls how writer's schema will be selected based on the incoming batch's schema as well as existing table's one. When schema reconciliation is DISABLED, incoming batch's schema will be picked as a writer-schema (therefore updating table's schema). When schema recon [...]
+| [hoodie.datasource.write.record.merger.impls](#hoodiedatasourcewriterecordmergerimpls) | org.apache.hudi.common.model.HoodieAvroRecordMerger | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These merger impls will filter by hoodie.datasource.write.record.merger.strategy Hudi will pick most efficient implementation to perform merging/combining of the records (during [...]
+| [hoodie.datasource.write.record.merger.strategy](#hoodiedatasourcewriterecordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in hoodie.datasource.write.record.merger.impls which has the same merger strategy id<br />`Config Param: RECORD_MERGER_STRATEGY`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.write.row.writer.enable](#hoodiedatasourcewriterowwriterenable) | true | When set to true, will perform write operations directly using the spark native `Row` representation, avoiding any additional conversion costs.<br />`Config Param: ENABLE_ROW_WRITER` [...]
+| [hoodie.datasource.write.streaming.checkpoint.identifier](#hoodiedatasourcewritestreamingcheckpointidentifier) | default_single_writer | A stream identifier used for HUDI to fetch the right checkpoint(`batch id` to be more specific) corresponding this writer. Please note that keep the identifier an unique value for different writer if under multi-writer scenario. If the value is not set, will only keep the checkpo [...]
+| [hoodie.datasource.write.streaming.disable.compaction](#hoodiedatasourcewritestreamingdisablecompaction) | false | By default for MOR table, async compaction is enabled with spark streaming sink. By setting this config to true, we can disable it and the expectation is that, users will schedule and execute compaction in a different process/job altogether. Some users may wish to run it separate [...]
+| [hoodie.datasource.write.streaming.ignore.failed.batch](#hoodiedatasourcewritestreamingignorefailedbatch) | false | Config to indicate whether to ignore any non exception error (e.g. writestatus error) within a streaming microbatch. Turning this on, could hide the write status errors while the spark checkpoint moves ahead.So, would recommend users to use this with caution.<br />`Config Param: [...]
+| [hoodie.datasource.write.streaming.retry.count](#hoodiedatasourcewritestreamingretrycount) | 3 | Config to indicate how many times streaming job should retry for a failed micro batch.<br />`Config Param: STREAMING_RETRY_CNT` [...]
+| [hoodie.datasource.write.streaming.retry.interval.ms](#hoodiedatasourcewritestreamingretryintervalms) | 2000 | Config to indicate how long (by millisecond) before a retry should issued for failed microbatch<br />`Config Param: STREAMING_RETRY_INTERVAL_MS` [...]
+| [hoodie.meta.sync.client.tool.class](#hoodiemetasyncclienttoolclass) | org.apache.hudi.hive.HiveSyncTool | Sync tool class name used to sync to metastore. Defaults to Hive.<br />`Config Param: META_SYNC_CLIENT_TOOL_CLASS_NAME` [...]
+| [hoodie.spark.sql.insert.into.operation](#hoodiesparksqlinsertintooperation) | insert | Sql write operation to use with INSERT_INTO spark sql command. This comes with 3 possible values, bulk_insert, insert and upsert. bulk_insert is generally meant for initial loads and is known to be performant compared to insert. But bulk_insert may not do small file management. I [...]
+| [hoodie.spark.sql.optimized.writes.enable](#hoodiesparksqloptimizedwritesenable) | true | Controls whether spark sql prepped update, delete, and merge are enabled.<br />`Config Param: SPARK_SQL_OPTIMIZED_WRITES`<br />`Since Version: 0.14.0` [...]
+| [hoodie.sql.bulk.insert.enable](#hoodiesqlbulkinsertenable) | false | When set to true, the sql insert statement will use bulk insert. This config is deprecated as of 0.14.0. Please use hoodie.spark.sql.insert.into.operation instead.<br />`Config Param: SQL_ENABLE_BULK_INSERT` [...]
+| [hoodie.sql.insert.mode](#hoodiesqlinsertmode) | upsert | Insert mode when insert data to pk-table. The optional modes are: upsert, strict and non-strict.For upsert mode, insert statement do the upsert operation for the pk-table which will update the duplicate record.For strict mode, insert statement will keep the primary key uniqueness [...]
+| [hoodie.streamer.source.kafka.value.deserializer.class](#hoodiestreamersourcekafkavaluedeserializerclass) | io.confluent.kafka.serializers.KafkaAvroDeserializer | This class is used by kafka client to deserialize the records<br />`Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.9.0` [...]
+| [hoodie.write.set.null.for.missing.columns](#hoodiewritesetnullformissingcolumns) | false | When a nullable column is missing from incoming batch during a write operation, the write operation will fail schema compatibility check. Set this option to true will make the missing column be filled with null values to successfully complete the write operation.<br />`Config P [...]
+---
+
+
+### PreCommit Validator Configurations {#PreCommit-Validator-Configurations}
+The following set of configurations help validate new data before commits.
+
+
+
+[**Advanced Configs**](#PreCommit-Validator-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.precommit.validators](#hoodieprecommitvalidators) | | Comma separated list of class names that can be invoked to validate commit<br />`Config Param: VALIDATOR_CLASS_NAMES` |
+| [hoodie.precommit.validators.equality.sql.queries](#hoodieprecommitvalidatorsequalitysqlqueries) | | Spark SQL queries to run on table before committing new data to validate state before and after commit. Multiple queries separated by ';' delimiter are supported. Example: "select count(*) from \<TABLE_NAME\> Note \<TABLE_NAME\>is replaced by table state before and after commit.<br />`Config Param: EQUALITY_SQL_QUERIES` |
+| [hoodie.precommit.validators.inequality.sql.queries](#hoodieprecommitvalidatorsinequalitysqlqueries) | | Spark SQL queries to run on table before committing new data to validate state before and after commit.Multiple queries separated by ';' delimiter are supported.Example query: 'select count(*) from \<TABLE_NAME\> where col=null'Note \<TABLE_NAME\> variable is expected to be present in query.<br />`Config Param: INEQUALITY_SQL_QUERIES` |
+| [hoodie.precommit.validators.single.value.sql.queries](#hoodieprecommitvalidatorssinglevaluesqlqueries) | | Spark SQL queries to run on table before committing new data to validate state after commit.Multiple queries separated by ';' delimiter are supported.Expected result is included as part of query separated by '#'. Example query: 'query1#result1:query2#result2'Note \<TABLE_NAME\> variable is expected to be present in query.<br />`Config Param: SINGLE_VALUE_SQL_QUERIES` |
+---
+
+## Flink Sql Configs {#FLINK_SQL}
+These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
+
+
+### Flink Options {#Flink-Options}
+Flink jobs using the SQL can be configured through the options in WITH clause. The actual datasource level configs are listed below.
+
+
+
+[**Basic Configs**](#Flink-Options-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name to register to Hive metastore<br /> `Config Param: DATABASE_NAME` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name to register to Hive metastore<br /> `Config Param: TABLE_NAME` [...]
+| [path](#path) | (N/A) | Base path for the target hoodie table. The path would be created if it does not exist, otherwise a Hoodie table expects to be initialized successfully<br /> `Config Param: PATH` [...]
+| [read.end-commit](#readend-commit) | (N/A) | End commit instant for reading, the commit time format should be 'yyyyMMddHHmmss'<br /> `Config Param: READ_END_COMMIT` [...]
+| [read.start-commit](#readstart-commit) | (N/A) | Start commit instant for reading, the commit time format should be 'yyyyMMddHHmmss', by default reading from the latest instant for streaming read<br /> `Config Param: READ_START_COMMIT` [...]
+| [archive.max_commits](#archivemax_commits) | 50 | Max number of commits to keep before archiving older commits into a sequential log, default 50<br /> `Config Param: ARCHIVE_MAX_COMMITS` [...]
+| [archive.min_commits](#archivemin_commits) | 40 | Min number of commits to keep before archiving older commits into a sequential log, default 40<br /> `Config Param: ARCHIVE_MIN_COMMITS` [...]
+| [cdc.enabled](#cdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode<br /> `Config Param: CDC_ENABLED` [...]
+| [cdc.supplemental.logging.mode](#cdcsupplementalloggingmode) | DATA_BEFORE_AFTER | Setting 'op_key_only' persists the 'op' and the record key only, setting 'data_before' persists the additional 'before' image, and setting 'data_before_after' persists the additional 'before' and 'after' images.<br /> `Config Param: SUPPLEMENTAL_LOGGING_MODE` [...]
+| [changelog.enabled](#changelogenabled) | false | Whether to keep all the intermediate changes, we try to keep all the changes of a record when enabled: 1). The sink accept the UPDATE_BEFORE message; 2). The source try to emit every changes of a record. The semantics is best effort because the compaction job would finally merge all changes of a record into one. default false to have UPSERT semantics<br /> [...]
+| [clean.async.enabled](#cleanasyncenabled) | true | Whether to cleanup the old commits immediately on new commits, enabled by default<br /> `Config Param: CLEAN_ASYNC_ENABLED` [...]
+| [clean.retain_commits](#cleanretain_commits) | 30 | Number of commits to retain. So data will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much you can incrementally pull on this table, default 30<br /> `Config Param: CLEAN_RETAIN_COMMITS` [...]
+| [clustering.async.enabled](#clusteringasyncenabled) | false | Async Clustering, default false<br /> `Config Param: CLUSTERING_ASYNC_ENABLED` [...]
+| [clustering.plan.strategy.small.file.limit](#clusteringplanstrategysmallfilelimit) | 600 | Files smaller than the size specified here are candidates for clustering, default 600 MB<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_SMALL_FILE_LIMIT` [...]
+| [clustering.plan.strategy.target.file.max.bytes](#clusteringplanstrategytargetfilemaxbytes) | 1073741824 | Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups, default 1 GB<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_TARGET_FILE_MAX_BYTES` [...]
+| [compaction.async.enabled](#compactionasyncenabled) | true | Async Compaction, enabled by default for MOR<br /> `Config Param: COMPACTION_ASYNC_ENABLED` [...]
+| [compaction.delta_commits](#compactiondelta_commits) | 5 | Max delta commits needed to trigger compaction, default 5 commits<br /> `Config Param: COMPACTION_DELTA_COMMITS` [...]
+| [hive_sync.enabled](#hive_syncenabled) | false | Asynchronously sync Hive meta to HMS, default false<br /> `Config Param: HIVE_SYNC_ENABLED` [...]
+| [hive_sync.jdbc_url](#hive_syncjdbc_url) | jdbc:hive2://localhost:10000 | Jdbc URL for hive sync, default 'jdbc:hive2://localhost:10000'<br /> `Config Param: HIVE_SYNC_JDBC_URL` [...]
+| [hive_sync.metastore.uris](#hive_syncmetastoreuris) | | Metastore uris for hive sync, default ''<br /> `Config Param: HIVE_SYNC_METASTORE_URIS` [...]
+| [hive_sync.mode](#hive_syncmode) | HMS | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql, default 'hms'<br /> `Config Param: HIVE_SYNC_MODE` [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Decides how data files need to be read, in 1) Snapshot mode (obtain latest view, based on row & columnar data); 2) incremental mode (new data since an instantTime); 3) Read Optimized mode (obtain latest view, based on columnar data) .Default: snapshot<br /> `Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br /> `Config Param: HIVE_STYLE_PARTITIONING` [...]
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | | Partition path field. Value to be used at the `partitionPath` component of `HoodieKey`. Actual value obtained by invoking .toString(), default ''<br /> `Config Param: PARTITION_PATH_FIELD` [...]
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | uuid | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br /> `Config Param: RECORD_KEY_FIELD` [...]
+| [index.type](#indextype) | FLINK_STATE | Index type of Flink write job, default is using state backed index.<br /> `Config Param: INDEX_TYPE` [...]
+| [metadata.compaction.delta_commits](#metadatacompactiondelta_commits) | 10 | Max delta commits for metadata table to trigger compaction, default 10<br /> `Config Param: METADATA_COMPACTION_DELTA_COMMITS` [...]
+| [metadata.enabled](#metadataenabled) | false | Enable the internal metadata table which serves table metadata like level file listings, default disabled<br /> `Config Param: METADATA_ENABLED` [...]
+| [precombine.field](#precombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br /> `Config Param: PRECOMBINE_FIELD` [...]
+| [read.streaming.enabled](#readstreamingenabled) | false | Whether to read as streaming source, default false<br /> `Config Param: READ_AS_STREAMING` [...]
+| [table.type](#tabletype) | COPY_ON_WRITE | Type of table to write. COPY_ON_WRITE (or) MERGE_ON_READ<br /> `Config Param: TABLE_TYPE` [...]
+| [write.operation](#writeoperation) | upsert | The write operation, that this write should do<br /> `Config Param: OPERATION` [...]
+| [write.parquet.max.file.size](#writeparquetmaxfilesize) | 120 | Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /> `Config Param: WRITE_PARQUET_MAX_FILE_SIZE` [...]
+
+[**Advanced Configs**](#Flink-Options-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [clustering.tasks](#clusteringtasks) | (N/A) | Parallelism of tasks that do actual clustering, default same as the write task parallelism<br /> `Config Param: CLUSTERING_TASKS` [...]
+| [compaction.tasks](#compactiontasks) | (N/A) | Parallelism of tasks that do actual compaction, default same as the write task parallelism<br /> `Config Param: COMPACTION_TASKS` [...]
+| [hive_sync.conf.dir](#hive_syncconfdir) | (N/A) | The hive configuration directory, where the hive-site.xml lies in, the file should be put on the client machine<br /> `Config Param: HIVE_SYNC_CONF_DIR` [...]
+| [hive_sync.serde_properties](#hive_syncserde_properties) | (N/A) | Serde properties to hive table, the data format is k1=v1 k2=v2<br /> `Config Param: HIVE_SYNC_TABLE_SERDE_PROPERTIES` [...]
+| [hive_sync.table_properties](#hive_synctable_properties) | (N/A) | Additional properties to store with table, the data format is k1=v1 k2=v2<br /> `Config Param: HIVE_SYNC_TABLE_PROPERTIES` [...]
+| [hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass) | (N/A) | Key generator class, that implements will extract the key out of incoming record<br /> `Config Param: KEYGEN_CLASS_NAME` [...]
+| [read.tasks](#readtasks) | (N/A) | Parallelism of tasks that do actual read, default is the parallelism of the execution environment<br /> `Config Param: READ_TASKS` [...]
+| [source.avro-schema](#sourceavro-schema) | (N/A) | Source avro schema string, the parsed schema is used for deserialization<br /> `Config Param: SOURCE_AVRO_SCHEMA` [...]
+| [source.avro-schema.path](#sourceavro-schemapath) | (N/A) | Source avro schema file path, the parsed schema is used for deserialization<br /> `Config Param: SOURCE_AVRO_SCHEMA_PATH` [...]
+| [write.bucket_assign.tasks](#writebucket_assigntasks) | (N/A) | Parallelism of tasks that do bucket assign, default same as the write task parallelism<br /> `Config Param: BUCKET_ASSIGN_TASKS` [...]
+| [write.index_bootstrap.tasks](#writeindex_bootstraptasks) | (N/A) | Parallelism of tasks that do index bootstrap, default same as the write task parallelism<br /> `Config Param: INDEX_BOOTSTRAP_TASKS` [...]
+| [write.partition.format](#writepartitionformat) | (N/A) | Partition path format, only valid when 'write.datetime.partitioning' is true, default is: 1) 'yyyyMMddHH' for timestamp(3) WITHOUT TIME ZONE, LONG, FLOAT, DOUBLE, DECIMAL; 2) 'yyyyMMdd' for DATE and INT.<br /> `Config Param: PARTITION_FORMAT` [...]
+| [write.tasks](#writetasks) | (N/A) | Parallelism of tasks that do actual write, default is the parallelism of the execution environment<br /> `Config Param: WRITE_TASKS` [...]
+| [clean.policy](#cleanpolicy) | KEEP_LATEST_COMMITS | Clean policy to manage the Hudi table. Available option: KEEP_LATEST_COMMITS, KEEP_LATEST_FILE_VERSIONS, KEEP_LATEST_BY_HOURS.Default is KEEP_LATEST_COMMITS.<br /> `Config Param: CLEAN_POLICY` [...]
+| [clean.retain_file_versions](#cleanretain_file_versions) | 5 | Number of file versions to retain. default 5<br /> `Config Param: CLEAN_RETAIN_FILE_VERSIONS` [...]
+| [clean.retain_hours](#cleanretain_hours) | 24 | Number of hours for which commits need to be retained. This config provides a more flexible option ascompared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times [...]
+| [clustering.delta_commits](#clusteringdelta_commits) | 4 | Max delta commits needed to trigger clustering, default 4 commits<br /> `Config Param: CLUSTERING_DELTA_COMMITS` [...]
+| [clustering.plan.partition.filter.mode](#clusteringplanpartitionfiltermode) | NONE | Partition filter mode used in the creation of clustering plan. Available values are - NONE: do not filter table partition and thus the clustering plan will include all partitions that have clustering candidate.RECENT_DAYS: keep a continuous range of partitions, worked together wit [...]
+| [clustering.plan.strategy.class](#clusteringplanstrategyclass) | org.apache.hudi.client.clustering.plan.strategy.FlinkSizeBasedClusteringPlanStrategy | Config to provide a strategy class (subclass of ClusteringPlanStrategy) to create clustering plan i.e select what file groups are being clustered. Default strategy, looks at the last N (determined by clustering.plan.strategy.daybased.lookback.partitions) day based partitions picks [...]
+| [clustering.plan.strategy.cluster.begin.partition](#clusteringplanstrategyclusterbeginpartition) | | Begin partition used to filter partition (inclusive)<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_CLUSTER_BEGIN_PARTITION` [...]
+| [clustering.plan.strategy.cluster.end.partition](#clusteringplanstrategyclusterendpartition) | | End partition used to filter partition (inclusive)<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_CLUSTER_END_PARTITION` [...]
+| [clustering.plan.strategy.daybased.lookback.partitions](#clusteringplanstrategydaybasedlookbackpartitions) | 2 | Number of partitions to list to create ClusteringPlan, default is 2<br /> `Config Param: CLUSTERING_TARGET_PARTITIONS` [...]
+| [clustering.plan.strategy.daybased.skipfromlatest.partitions](#clusteringplanstrategydaybasedskipfromlatestpartitions) | 0 | Number of partitions to skip from latest when choosing partitions to create ClusteringPlan<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST` [...]
+| [clustering.plan.strategy.max.num.groups](#clusteringplanstrategymaxnumgroups) | 30 | Maximum number of groups to create as part of ClusteringPlan. Increasing groups will increase parallelism, default is 30<br /> `Config Param: CLUSTERING_MAX_NUM_GROUPS` [...]
+| [clustering.plan.strategy.partition.regex.pattern](#clusteringplanstrategypartitionregexpattern) | | Filter clustering partitions that matched regex pattern<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_PARTITION_REGEX_PATTERN` [...]
+| [clustering.plan.strategy.partition.selected](#clusteringplanstrategypartitionselected) | | Partitions to run clustering<br /> `Config Param: CLUSTERING_PLAN_STRATEGY_PARTITION_SELECTED` [...]
+| [clustering.plan.strategy.sort.columns](#clusteringplanstrategysortcolumns) | | Columns to sort the data by when clustering<br /> `Config Param: CLUSTERING_SORT_COLUMNS` [...]
+| [clustering.schedule.enabled](#clusteringscheduleenabled) | false | Schedule the cluster plan, default false<br /> `Config Param: CLUSTERING_SCHEDULE_ENABLED` [...]
+| [compaction.delta_seconds](#compactiondelta_seconds) | 3600 | Max delta seconds time needed to trigger compaction, default 1 hour<br /> `Config Param: COMPACTION_DELTA_SECONDS` [...]
+| [compaction.max_memory](#compactionmax_memory) | 100 | Max memory in MB for compaction spillable map, default 100MB<br /> `Config Param: COMPACTION_MAX_MEMORY` [...]
+| [compaction.schedule.enabled](#compactionscheduleenabled) | true | Schedule the compaction plan, enabled by default for MOR<br /> `Config Param: COMPACTION_SCHEDULE_ENABLED` [...]
+| [compaction.target_io](#compactiontarget_io) | 512000 | Target IO in MB for per compaction (both read and write), default 500 GB<br /> `Config Param: COMPACTION_TARGET_IO` [...]
+| [compaction.timeout.seconds](#compactiontimeoutseconds) | 1200 | Max timeout time in seconds for online compaction to rollback, default 20 minutes<br /> `Config Param: COMPACTION_TIMEOUT_SECONDS` [...]
+| [compaction.trigger.strategy](#compactiontriggerstrategy) | num_commits | Strategy to trigger compaction, options are 'num_commits': trigger compaction when there are at least N delta commits after last completed compaction; 'num_commits_after_last_request': trigger compaction when there are at least N delta commits after last completed/requested compac [...]
+| [hive_sync.assume_date_partitioning](#hive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/mm/dd, default false<br /> `Config Param: HIVE_SYNC_ASSUME_DATE_PARTITION` [...]
+| [hive_sync.auto_create_db](#hive_syncauto_create_db) | true | Auto create hive database if it does not exists, default true<br /> `Config Param: HIVE_SYNC_AUTO_CREATE_DB` [...]
+| [hive_sync.db](#hive_syncdb) | default | Database name for hive sync, default 'default'<br /> `Config Param: HIVE_SYNC_DB` [...]
+| [hive_sync.file_format](#hive_syncfile_format) | PARQUET | File format for hive sync, default 'PARQUET'<br /> `Config Param: HIVE_SYNC_FILE_FORMAT` [...]
+| [hive_sync.ignore_exceptions](#hive_syncignore_exceptions) | false | Ignore exceptions during hive synchronization, default false<br /> `Config Param: HIVE_SYNC_IGNORE_EXCEPTIONS` [...]
+| [hive_sync.partition_extractor_class](#hive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Tool to extract the partition value from HDFS path, default 'MultiPartKeysValueExtractor'<br /> `Config Param: HIVE_SYNC_PARTITION_EXTRACTOR_CLASS_NAME` [...]
+| [hive_sync.partition_fields](#hive_syncpartition_fields) | | Partition fields for hive sync, default ''<br /> `Config Param: HIVE_SYNC_PARTITION_FIELDS` [...]
+| [hive_sync.password](#hive_syncpassword) | hive | Password for hive sync, default 'hive'<br /> `Config Param: HIVE_SYNC_PASSWORD` [...]
+| [hive_sync.skip_ro_suffix](#hive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table when registering, default false<br /> `Config Param: HIVE_SYNC_SKIP_RO_SUFFIX` [...]
+| [hive_sync.support_timestamp](#hive_syncsupport_timestamp) | true | INT64 with original type TIMESTAMP_MICROS is converted to hive timestamp type. Disabled by default for backward compatibility.<br /> `Config Param: HIVE_SYNC_SUPPORT_TIMESTAMP` [...]
+| [hive_sync.table](#hive_synctable) | unknown | Table name for hive sync, default 'unknown'<br /> `Config Param: HIVE_SYNC_TABLE` [...]
+| [hive_sync.table.strategy](#hive_synctablestrategy) | ALL | Hive table synchronization strategy. Available option: RO, RT, ALL.<br /> `Config Param: HIVE_SYNC_TABLE_STRATEGY` [...]
+| [hive_sync.use_jdbc](#hive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled, default true<br /> `Config Param: HIVE_SYNC_USE_JDBC` [...]
+| [hive_sync.username](#hive_syncusername) | hive | Username for hive sync, default 'hive'<br /> `Config Param: HIVE_SYNC_USERNAME` [...]
+| [hoodie.bucket.index.hash.field](#hoodiebucketindexhashfield) | | Index key field. Value to be used as hashing to find the bucket ID. Should be a subset of or equal to the recordKey fields. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br /> `Config Par [...]
+| [hoodie.bucket.index.num.buckets](#hoodiebucketindexnumbuckets) | 4 | Hudi bucket number per partition. Only affected if using Hudi bucket index.<br /> `Config Param: BUCKET_INDEX_NUM_BUCKETS` [...]
+| [hoodie.datasource.merge.type](#hoodiedatasourcemergetype) | payload_combine | For Snapshot query on merge on read table. Use this key to define how the payloads are merged, in 1) skip_merge: read the base file records plus the log file records; 2) payload_combine: read the base file records first, for each record in base file, checks whether the key is in t [...]
+| [hoodie.datasource.write.keygenerator.type](#hoodiedatasourcewritekeygeneratortype) | SIMPLE | Key generator type, that implements will extract the key out of incoming record. **Note** This is being actively worked on. Please use `hoodie.datasource.write.keygenerator.class` instead.<br /> `Config Param: KEYGEN_TYPE` [...]
+| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Whether to encode the partition path url, default false<br /> `Config Param: URL_ENCODE_PARTITIONING` [...]
+| [hoodie.index.bucket.engine](#hoodieindexbucketengine) | SIMPLE | Type of bucket index engine. Available options: [SIMPLE | CONSISTENT_HASHING]<br /> `Config Param: BUCKET_INDEX_ENGINE_TYPE` [...]
+| [index.bootstrap.enabled](#indexbootstrapenabled) | false | Whether to bootstrap the index state from existing hoodie table, default false<br /> `Config Param: INDEX_BOOTSTRAP_ENABLED` [...]
+| [index.global.enabled](#indexglobalenabled) | true | Whether to update index for the old partition path if same key record with different partition path came in, default true<br /> `Config Param: INDEX_GLOBAL_ENABLED` [...]
+| [index.partition.regex](#indexpartitionregex) | .* | Whether to load partitions in state if partition path matching， default `*`<br /> `Config Param: INDEX_PARTITION_REGEX` [...]
+| [index.state.ttl](#indexstatettl) | 0.0 | Index state ttl in days, default stores the index permanently<br /> `Config Param: INDEX_STATE_TTL` [...]
+| [partition.default_name](#partitiondefault_name) | __HIVE_DEFAULT_PARTITION__ | The default partition name in case the dynamic partition column value is null/empty string<br /> `Config Param: PARTITION_DEFAULT_NAME` [...]
+| [payload.class](#payloadclass) | org.apache.hudi.common.model.EventTimeAvroPayload | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for the option in-effective<br /> `Config Param: PAYLOAD_CLASS_NAME` [...]
+| [read.data.skipping.enabled](#readdataskippingenabled) | false | Enables data-skipping allowing queries to leverage indexes to reduce the search space byskipping over files<br /> `Config Param: READ_DATA_SKIPPING_ENABLED` [...]
+| [read.streaming.check-interval](#readstreamingcheck-interval) | 60 | Check interval for streaming read of SECOND, default 1 minute<br /> `Config Param: READ_STREAMING_CHECK_INTERVAL` [...]
+| [read.streaming.skip_clustering](#readstreamingskip_clustering) | true | Whether to skip clustering instants to avoid reading base files of clustering operations for streaming read to improve read performance.<br /> `Config Param: READ_STREAMING_SKIP_CLUSTERING` [...]
+| [read.streaming.skip_compaction](#readstreamingskip_compaction) | true | Whether to skip compaction instants and avoid reading compacted base files for streaming read to improve read performance. This option can be used to avoid reading duplicates when changelog mode is enabled, it is a solution to keep data integrity <br /> `Config Param: READ_STREAMI [...]
+| [read.utc-timezone](#readutc-timezone) | true | Use UTC timezone or local timezone to the conversion between epoch time and LocalDateTime. Hive 0.x/1.x/2.x use local timezone. But Hive 3.x use UTC timezone, by default true<br /> `Config Param: READ_UTC_TIMEZONE` [...]
+| [record.merger.impls](#recordmergerimpls) | org.apache.hudi.common.model.HoodieAvroRecordMerger | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These merger impls will filter by record.merger.strategy. Hudi will pick most efficient implementation to perform merging/combining of the records (during update, reading MOR tab [...]
+| [record.merger.strategy](#recordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in record.merger.impls which has the same merger strategy id<br /> `Config Param: RECORD_MERGER_STRATEGY` [...]
+| [write.batch.size](#writebatchsize) | 256.0 | Batch buffer size in MB to flush data into the underneath filesystem, default 256MB<br /> `Config Param: WRITE_BATCH_SIZE` [...]
+| [write.bulk_insert.shuffle_input](#writebulk_insertshuffle_input) | true | Whether to shuffle the inputs by specific fields for bulk insert tasks, default true<br /> `Config Param: WRITE_BULK_INSERT_SHUFFLE_INPUT` [...]
+| [write.bulk_insert.sort_input](#writebulk_insertsort_input) | true | Whether to sort the inputs by specific fields for bulk insert tasks, default true<br /> `Config Param: WRITE_BULK_INSERT_SORT_INPUT` [...]
+| [write.bulk_insert.sort_input.by_record_key](#writebulk_insertsort_inputby_record_key) | false | Whether to sort the inputs by record keys for bulk insert tasks, default false<br /> `Config Param: WRITE_BULK_INSERT_SORT_INPUT_BY_RECORD_KEY` [...]
+| [write.client.id](#writeclientid) | | Unique identifier used to distinguish different writer pipelines for concurrent mode<br /> `Config Param: WRITE_CLIENT_ID` [...]
+| [write.commit.ack.timeout](#writecommitacktimeout) | -1 | Timeout limit for a writer task after it finishes a checkpoint and waits for the instant commit success, only for internal use<br /> `Config Param: WRITE_COMMIT_ACK_TIMEOUT` [...]
+| [write.ignore.failed](#writeignorefailed) | false | Flag to indicate whether to ignore any non exception error (e.g. writestatus error). within a checkpoint batch. By default false. Turning this on, could hide the write status errors while the flink checkpoint moves ahead. So, would recommend users to use this with caution.<br /> [...]
+| [write.insert.cluster](#writeinsertcluster) | false | Whether to merge small files for insert mode, if true, the write throughput will decrease because the read/write of existing small file, only valid for COW table, default false<br /> `Config Param: INSERT_CLUSTER` [...]
+| [write.log.max.size](#writelogmaxsize) | 1024 | Maximum size allowed in MB for a log file before it is rolled over to the next version, default 1GB<br /> `Config Param: WRITE_LOG_MAX_SIZE` [...]
+| [write.log_block.size](#writelog_blocksize) | 128 | Max log block size in MB for log file, default 128MB<br /> `Config Param: WRITE_LOG_BLOCK_SIZE` [...]
+| [write.merge.max_memory](#writemergemax_memory) | 100 | Max memory in MB for merge, default 100MB<br /> `Config Param: WRITE_MERGE_MAX_MEMORY` [...]
+| [write.parquet.block.size](#writeparquetblocksize) | 120 | Parquet RowGroup size. It's recommended to make this large enough that scan costs can be amortized by packing enough column values into a single row group.<br /> `Config Param: WRITE_PARQUET_BLOCK_SIZE` [...]
+| [write.parquet.page.size](#writeparquetpagesize) | 1 | Parquet page size. Page is the unit of read within a parquet file. Within a block, pages are compressed separately.<br /> `Config Param: WRITE_PARQUET_PAGE_SIZE` [...]
+| [write.partition.overwrite.mode](#writepartitionoverwritemode) | STATIC | When INSERT OVERWRITE a partitioned data source table, we currently support 2 modes: static and dynamic. Static mode deletes all the partitions that match the partition specification(e.g. PARTITION(a=1,b)) in the INSERT statement, before overwriting. Dynamic mode doesn't delete pa [...]
+| [write.precombine](#writeprecombine) | false | Flag to indicate whether to drop duplicates before insert/upsert. By default these cases will accept duplicates, to gain extra performance: 1) insert operation; 2) upsert for MOR table, the MOR table deduplicate on reading<br /> `Config Param: PRE_COMBINE` [...]
+| [write.rate.limit](#writeratelimit) | 0 | Write record rate limit per second to prevent traffic jitter and improve stability, default 0 (no limit)<br /> `Config Param: WRITE_RATE_LIMIT` [...]
+| [write.retry.interval.ms](#writeretryintervalms) | 2000 | Flag to indicate how long (by millisecond) before a retry should issued for failed checkpoint batch. By default 2000 and it will be doubled by every retry<br /> `Config Param: RETRY_INTERVAL_MS` [...]
+| [write.retry.times](#writeretrytimes) | 3 | Flag to indicate how many times streaming job should retry for a failed checkpoint batch. By default 3<br /> `Config Param: RETRY_TIMES` [...]
+| [write.sort.memory](#writesortmemory) | 128 | Sort memory in MB, default 128MB<br /> `Config Param: WRITE_SORT_MEMORY` [...]
+| [write.task.max.size](#writetaskmaxsize) | 1024.0 | Maximum memory in MB for a write task, when the threshold hits, it flushes the max size data bucket to avoid OOM, default 1GB<br /> `Config Param: WRITE_TASK_MAX_SIZE` [...]
+| [write.utc-timezone](#writeutc-timezone) | true | Use UTC timezone or local timezone to the conversion between epoch time and LocalDateTime. Default value is utc timezone for forward compatibility.<br /> `Config Param: WRITE_UTC_TIMEZONE` [...]
+---
+
+## Write Client Configs {#WRITE_CLIENT}
+Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.
+
+
+### Common Configurations {#Common-Configurations}
+The following set of configurations are common across Hudi.
+
+
+
+[**Basic Configs**](#Common-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` |
+
+[**Advanced Configs**](#Common-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [as.of.instant](#asofinstant) | (N/A) | The query instant for time travel. Without specified this option, we query the latest snapshot.<br />`Config Param: TIMESTAMP_AS_OF` [...]
+| [hoodie.memory.compaction.max.size](#hoodiememorycompactionmaxsize) | (N/A) | Maximum amount of memory used in bytes for compaction operations in bytes , before spilling to local storage.<br />`Config Param: MAX_MEMORY_FOR_COMPACTION` [...]
+| [hoodie.common.diskmap.compression.enabled](#hoodiecommondiskmapcompressionenabled) | true | Turn on compression for BITCASK disk map used by the External Spillable Map<br />`Config Param: DISK_MAP_BITCASK_COMPRESSION_ENABLED` [...]
+| [hoodie.common.spillable.diskmap.type](#hoodiecommonspillablediskmaptype) | BITCASK | When handling input data that cannot be held in memory, to merge with a file on storage, a spillable diskmap is employed. By default, we use a persistent hashmap based loosely on bitcask, that offers O(1) inserts, lookups. Change this to `ROCKS_DB` to prefer using rocksDB, for handling the spill.<br />`Config Param: SPILLABLE_DISK_MAP_TYPE` [...]
+| [hoodie.datasource.write.reconcile.schema](#hoodiedatasourcewritereconcileschema) | false | This config controls how writer's schema will be selected based on the incoming batch's schema as well as existing table's one. When schema reconciliation is DISABLED, incoming batch's schema will be picked as a writer-schema (therefore updating table's schema). When schema reconciliation is ENABLED, writer-schema will be picked such that table's schema (after txn) is either kept the [...]
+| [hoodie.fs.atomic_creation.support](#hoodiefsatomic_creationsupport) | | This config is used to specify the file system which supports atomic file creation . atomic means that an operation either succeeds and has an effect or has fails and has no effect; now this feature is used by FileSystemLockProvider to guaranteeing that only one writer can create the lock file at a time. since some FS does not support atomic file creation (eg: S3), we decide the FileS [...]
+| [hoodie.memory.dfs.buffer.max.size](#hoodiememorydfsbuffermaxsize) | 16777216 | Property to control the max memory in bytes for dfs input stream buffer size<br />`Config Param: MAX_DFS_STREAM_BUFFER_SIZE` [...]
+| [hoodie.read.timeline.holes.resolution.policy](#hoodiereadtimelineholesresolutionpolicy) | FAIL | When doing incremental queries, there could be hollow commits (requested or inflight commits that are not the latest) that are produced by concurrent writers and could lead to potential data loss. This config allows users to have different ways of handling this situation. The valid values are [FAIL, BLOCK, USE_TRANSITION_TIME]: Use `FAIL` to throw an exception when hollow commit is de [...]
+| [hoodie.schema.on.read.enable](#hoodieschemaonreadenable) | false | Enables support for Schema Evolution feature<br />`Config Param: SCHEMA_EVOLUTION_ENABLE` [...]
+| [hoodie.write.set.null.for.missing.columns](#hoodiewritesetnullformissingcolumns) | false | When a nullable column is missing from incoming batch during a write operation, the write operation will fail schema compatibility check. Set this option to true will make the missing column be filled with null values to successfully complete the write operation.<br />`Config Param: SET_NULL_FOR_MISSING_COLUMNS`<br />`Since Version: 0.14.1` [...]
+---
+
+
+### Metadata Configs {#Metadata-Configs}
+Configurations used by the Hudi Metadata Table. This table maintains the metadata about a given Hudi table (e.g file listings) to avoid overhead of accessing cloud storage, during queries.
+
+
+
+[**Basic Configs**](#Metadata-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.metadata.enable](#hoodiemetadataenable) | true | Enable the internal metadata table which serves table metadata like level file listings<br />`Config Param: ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.metadata.index.bloom.filter.enable](#hoodiemetadataindexbloomfilterenable) | false | Enable indexing bloom filters of user data files under metadata table. When enabled, metadata table will have a partition to store the bloom filter index and will be used during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_BLOOM_FILTER`<br />`Since Version: 0.11.0` |
+| [hoodie.metadata.index.column.stats.enable](#hoodiemetadataindexcolumnstatsenable) | false | Enable indexing column ranges of user data files under metadata table key lookups. When enabled, metadata table will have a partition to store the column ranges and will be used for pruning files during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_COLUMN_STATS`<br />`Since Version: 0.11.0` |
+
+[**Advanced Configs**](#Metadata-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------ | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.metadata.index.bloom.filter.column.list](#hoodiemetadataindexbloomfiltercolumnlist) | (N/A) | Comma-separated list of columns for which bloom filter index will be built. If not set, only record key will be indexed.<br />`Config Param: BLOOM_FILTER_INDEX_FOR_COLUMNS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.column.stats.column.list](#hoodiemetadataindexcolumnstatscolumnlist) | (N/A) | Comma-separated list of columns for which column stats index will be built. If not set, all columns will be indexed<br />`Config Param: COLUMN_STATS_INDEX_FOR_COLUMNS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.column.stats.processing.mode.override](#hoodiemetadataindexcolumnstatsprocessingmodeoverride) | (N/A) | By default Column Stats Index is automatically determining whether it should be read and processed either'in-memory' (w/in executing process) or using Spark (on a cluster), based on some factors like the size of the Index and how many columns are read. This config allows to override this behavior.<br />`Config Param: COLUMN_STATS_INDEX_PROCESSIN [...]
+| [_hoodie.metadata.ignore.spurious.deletes](#_hoodiemetadataignorespuriousdeletes) | true | There are cases when extra files are requested to be deleted from metadata table which are never added before. This config determines how to handle such spurious deletes<br />`Config Param: IGNORE_SPURIOUS_DELETES`<br />`Since Version: 0.10.0` [...]
+| [hoodie.assume.date.partitioning](#hoodieassumedatepartitioning) | false | Should HoodieWriteClient assume the data is partitioned by dates, i.e three levels from base path. This is a stop-gap to support tables created by versions < 0.3.1. Will be removed eventually<br />`Config Param: ASSUME_DATE_PARTITIONING`<br />`Since Version: 0.3.0` [...]
+| [hoodie.file.listing.parallelism](#hoodiefilelistingparallelism) | 200 | Parallelism to use, when listing the table on lake storage.<br />`Config Param: FILE_LISTING_PARALLELISM_VALUE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.metadata.auto.initialize](#hoodiemetadataautoinitialize) | true | Initializes the metadata table by reading from the file system when the table is first created. Enabled by default. Warning: This should only be disabled when manually constructing the metadata table outside of typical Hudi writer flows.<br />`Config Param: AUTO_INITIALIZE`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.compact.max.delta.commits](#hoodiemetadatacompactmaxdeltacommits) | 10 | Controls how often the metadata table is compacted.<br />`Config Param: COMPACT_NUM_DELTA_COMMITS`<br />`Since Version: 0.7.0` [...]
+| [hoodie.metadata.dir.filter.regex](#hoodiemetadatadirfilterregex) | | Directories matching this regex, will be filtered out when initializing metadata table from lake storage for the first time.<br />`Config Param: DIR_FILTER_REGEX`<br />`Since Version: 0.7.0` [...]
+| [hoodie.metadata.index.async](#hoodiemetadataindexasync) | false | Enable asynchronous indexing of metadata table.<br />`Config Param: ASYNC_INDEX_ENABLE`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.bloom.filter.file.group.count](#hoodiemetadataindexbloomfilterfilegroupcount) | 4 | Metadata bloom filter index partition file group count. This controls the size of the base and log files and read parallelism in the bloom filter index partition. The recommendation is to size the file group count such that the base files are under 1GB.<br />`Config Param: METADATA_INDEX_BLOOM_FILTER_FILE_GROUP_COUNT`<br />`Since Version: 0.11.0 [...]
+| [hoodie.metadata.index.bloom.filter.parallelism](#hoodiemetadataindexbloomfilterparallelism) | 200 | Parallelism to use for generating bloom filter index in metadata table.<br />`Config Param: BLOOM_FILTER_INDEX_PARALLELISM`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.check.timeout.seconds](#hoodiemetadataindexchecktimeoutseconds) | 900 | After the async indexer has finished indexing upto the base instant, it will ensure that all inflight writers reliably write index updates as well. If this timeout expires, then the indexer will abort itself safely.<br />`Config Param: METADATA_INDEX_CHECK_TIMEOUT_SECONDS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.column.stats.file.group.count](#hoodiemetadataindexcolumnstatsfilegroupcount) | 2 | Metadata column stats partition file group count. This controls the size of the base and log files and read parallelism in the column stats index partition. The recommendation is to size the file group count such that the base files are under 1GB.<br />`Config Param: METADATA_INDEX_COLUMN_STATS_FILE_GROUP_COUNT`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.column.stats.inMemory.projection.threshold](#hoodiemetadataindexcolumnstatsinMemoryprojectionthreshold) | 100000 | When reading Column Stats Index, if the size of the expected resulting projection is below the in-memory threshold (counted by the # of rows), it will be attempted to be loaded "in-memory" (ie not using the execution engine like Spark, Flink, etc). If the value is above the threshold execution engine will be used to compose the projection.<br /> [...]
+| [hoodie.metadata.index.column.stats.parallelism](#hoodiemetadataindexcolumnstatsparallelism) | 200 | Parallelism to use, when generating column stats index.<br />`Config Param: COLUMN_STATS_INDEX_PARALLELISM`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.log.compaction.blocks.threshold](#hoodiemetadatalogcompactionblocksthreshold) | 5 | Controls the criteria to log compacted files groups in metadata table.<br />`Config Param: LOG_COMPACT_BLOCKS_THRESHOLD`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.log.compaction.enable](#hoodiemetadatalogcompactionenable) | false | This configs enables logcompaction for the metadata table.<br />`Config Param: ENABLE_LOG_COMPACTION_ON_METADATA_TABLE`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.max.deltacommits.when_pending](#hoodiemetadatamaxdeltacommitswhen_pending) | 1000 | When there is a pending instant in data table, this config limits the allowed number of deltacommits in metadata table to prevent the metadata table's timeline from growing unboundedly as compaction won't be triggered due to the pending data table instant.<br />`Config Param: METADATA_MAX_NUM_DELTACOMMITS_WHEN_PENDING`<br />`Since Version: 0.14. [...]
+| [hoodie.metadata.max.init.parallelism](#hoodiemetadatamaxinitparallelism) | 100000 | Maximum parallelism to use when initializing Record Index.<br />`Config Param: RECORD_INDEX_MAX_PARALLELISM`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.max.logfile.size](#hoodiemetadatamaxlogfilesize) | 2147483648 | Maximum size in bytes of a single log file. Larger log files can contain larger log blocks thereby reducing the number of blocks to search for keys<br />`Config Param: MAX_LOG_FILE_SIZE_BYTES_PROP`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.max.reader.buffer.size](#hoodiemetadatamaxreaderbuffersize) | 10485760 | Max memory to use for the reader buffer while merging log blocks<br />`Config Param: MAX_READER_BUFFER_SIZE_PROP`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.max.reader.memory](#hoodiemetadatamaxreadermemory) | 1073741824 | Max memory to use for the reader to read from metadata<br />`Config Param: MAX_READER_MEMORY_PROP`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.metrics.enable](#hoodiemetadatametricsenable) | false | Enable publishing of metrics around metadata table.<br />`Config Param: METRICS_ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.metadata.optimized.log.blocks.scan.enable](#hoodiemetadataoptimizedlogblocksscanenable) | false | Optimized log blocks scanner that addresses all the multi-writer use-cases while appending to log files. It also differentiates original blocks written by ingestion writers and compacted blocks written by log compaction.<br />`Config Param: ENABLE_OPTIMIZED_LOG_BLOCKS_SCAN`<br />`Since Version: 0.13.0` [...]
+| [hoodie.metadata.record.index.enable](#hoodiemetadatarecordindexenable) | false | Create the HUDI Record Index within the Metadata Table<br />`Config Param: RECORD_INDEX_ENABLE_PROP`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.record.index.growth.factor](#hoodiemetadatarecordindexgrowthfactor) | 2.0 | The current number of records are multiplied by this number when estimating the number of file groups to create automatically. This helps account for growth in the number of records in the dataset.<br />`Config Param: RECORD_INDEX_GROWTH_FACTOR_PROP`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.record.index.max.filegroup.count](#hoodiemetadatarecordindexmaxfilegroupcount) | 10000 | Maximum number of file groups to use for Record Index.<br />`Config Param: RECORD_INDEX_MAX_FILE_GROUP_COUNT_PROP`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.record.index.max.filegroup.size](#hoodiemetadatarecordindexmaxfilegroupsize) | 1073741824 | Maximum size in bytes of a single file group. Large file group takes longer to compact.<br />`Config Param: RECORD_INDEX_MAX_FILE_GROUP_SIZE_BYTES_PROP`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.record.index.min.filegroup.count](#hoodiemetadatarecordindexminfilegroupcount) | 10 | Minimum number of file groups to use for Record Index.<br />`Config Param: RECORD_INDEX_MIN_FILE_GROUP_COUNT_PROP`<br />`Since Version: 0.14.0` [...]
+| [hoodie.metadata.spillable.map.path](#hoodiemetadataspillablemappath) | | Path on local storage to use, when keys read from metadata are held in a spillable map.<br />`Config Param: SPILLABLE_MAP_DIR_PROP`<br />`Since Version: 0.14.0` [...]
+---
+
+
+### Metaserver Configs {#Metaserver-Configs}
+Configurations used by the Hudi Metaserver.
+
+
+
+[**Advanced Configs**](#Metaserver-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name that will be used for incremental query.If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME`<br />`Since Version: 0.13.0` |
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: TABLE_NAME`<br />`Since Version: 0.13.0` |
+| [hoodie.metaserver.connect.retries](#hoodiemetaserverconnectretries) | 3 | Number of retries while opening a connection to metaserver<br />`Config Param: METASERVER_CONNECTION_RETRIES`<br />`Since Version: 0.13.0` |
+| [hoodie.metaserver.connect.retry.delay](#hoodiemetaserverconnectretrydelay) | 1 | Number of seconds for the client to wait between consecutive connection attempts<br />`Config Param: METASERVER_CONNECTION_RETRY_DELAY`<br />`Since Version: 0.13.0` |
+| [hoodie.metaserver.enabled](#hoodiemetaserverenabled) | false | Enable Hudi metaserver for storing Hudi tables' metadata.<br />`Config Param: METASERVER_ENABLE`<br />`Since Version: 0.13.0` |
+| [hoodie.metaserver.uris](#hoodiemetaserveruris) | thrift://localhost:9090 | Metaserver server uris<br />`Config Param: METASERVER_URLS`<br />`Since Version: 0.13.0` |
+---
+
+
+### Storage Configs {#Storage-Configs}
+Configurations that control aspects around writing, sizing, reading base and log files.
+
+
+
+[**Basic Configs**](#Storage-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.parquet.compression.codec](#hoodieparquetcompressioncodec) | gzip | Compression Codec for parquet files<br />`Config Param: PARQUET_COMPRESSION_CODEC_NAME` |
+| [hoodie.parquet.max.file.size](#hoodieparquetmaxfilesize) | 125829120 | Target size in bytes for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br />`Config Param: PARQUET_MAX_FILE_SIZE` |
+
+[**Advanced Configs**](#Storage-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.logfile.data.block.format](#hoodielogfiledatablockformat) | (N/A) | Format of the data block within delta logs. Following formats are currently supported "avro", "hfile", "parquet"<br />`Config Param: LOGFILE_DATA_BLOCK_FORMAT` [...]
+| [hoodie.avro.write.support.class](#hoodieavrowritesupportclass) | org.apache.hudi.avro.HoodieAvroWriteSupport | Provided write support class should extend HoodieAvroWriteSupport class and it is loaded at runtime. This is only required when trying to override the existing write context.<br />`Config Param: HOODIE_AVRO_WRITE_SUPPORT_CLASS`<br />`Since Version: 0.14.0` [...]
+| [hoodie.bloom.index.filter.dynamic.max.entries](#hoodiebloomindexfilterdynamicmaxentries) | 100000 | The threshold for the maximum number of keys to record in a dynamic Bloom filter row. Only applies if filter type is BloomFilterTypeCode.DYNAMIC_V0.<br />`Config Param: BLOOM_FILTER_DYNAMIC_MAX_ENTRIES` [...]
+| [hoodie.bloom.index.filter.type](#hoodiebloomindexfiltertype) | DYNAMIC_V0 | org.apache.hudi.common.bloom.BloomFilterTypeCode: Filter type used by Bloom filter. SIMPLE: Bloom filter that is based on the configured size. DYNAMIC_V0(default): Bloom filter that is auto sized based on number of keys.<br />`Config Param: BLOOM_FILTER_TYPE` [...]
+| [hoodie.hfile.block.size](#hoodiehfileblocksize) | 1048576 | Lower values increase the size in bytes of metadata tracked within HFile, but can offer potentially faster lookup times.<br />`Config Param: HFILE_BLOCK_SIZE` [...]
+| [hoodie.hfile.compression.algorithm](#hoodiehfilecompressionalgorithm) | GZ | Compression codec to use for hfile base files.<br />`Config Param: HFILE_COMPRESSION_ALGORITHM_NAME` [...]
+| [hoodie.hfile.max.file.size](#hoodiehfilemaxfilesize) | 125829120 | Target file size in bytes for HFile base files.<br />`Config Param: HFILE_MAX_FILE_SIZE` [...]
+| [hoodie.index.bloom.fpp](#hoodieindexbloomfpp) | 0.000000001 | Only applies if index type is BLOOM. Error rate allowed given the number of entries. This is used to calculate how many bits should be assigned for the bloom filter and the number of hash functions. This is usually set very low (default: 0.000000001), we like to tradeoff disk space for lower false positives. If the number of entries [...]
+| [hoodie.index.bloom.num_entries](#hoodieindexbloomnum_entries) | 60000 | Only applies if index type is BLOOM. This is the number of entries to be stored in the bloom filter. The rationale for the default: Assume the maxParquetFileSize is 128MB and averageRecordSize is 1kb and hence we approx a total of 130K records in a file. The default (60000) is roughly half of this approximation. Warning: Setting this [...]
+| [hoodie.io.factory.class](#hoodieiofactoryclass) | org.apache.hudi.io.hadoop.HoodieHadoopIOFactory | The fully-qualified class name of the factory class to return readers and writers of files used by Hudi. The provided class should implement `org.apache.hudi.io.storage.HoodieIOFactory`.<br />`Config Param: HOODIE_IO_FACTORY_CLASS`<br />`Since Version: 0.15.0` [...]
+| [hoodie.logfile.data.block.max.size](#hoodielogfiledatablockmaxsize) | 268435456 | LogFile Data block max size in bytes. This is the maximum size allowed for a single data block to be appended to a log file. This helps to make sure the data appended to the log file is broken up into sizable blocks to prevent from OOM errors. This size should be greater than the JVM memory.<br />`Config Param: LOGFILE_DATA_BLOCK_MAX [...]
+| [hoodie.logfile.max.size](#hoodielogfilemaxsize) | 1073741824 | LogFile max size in bytes. This is the maximum size allowed for a log file before it is rolled over to the next version.<br />`Config Param: LOGFILE_MAX_SIZE` [...]
+| [hoodie.logfile.to.parquet.compression.ratio](#hoodielogfiletoparquetcompressionratio) | 0.35 | Expected additional compression as records move from log files to parquet. Used for merge_on_read table to send inserts into log files & control the size of compacted parquet file.<br />`Config Param: LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION` [...]
+| [hoodie.orc.block.size](#hoodieorcblocksize) | 125829120 | ORC block size, recommended to be aligned with the target file size.<br />`Config Param: ORC_BLOCK_SIZE` [...]
+| [hoodie.orc.compression.codec](#hoodieorccompressioncodec) | ZLIB | Compression codec to use for ORC base files.<br />`Config Param: ORC_COMPRESSION_CODEC_NAME` [...]
+| [hoodie.orc.max.file.size](#hoodieorcmaxfilesize) | 125829120 | Target file size in bytes for ORC base files.<br />`Config Param: ORC_FILE_MAX_SIZE` [...]
+| [hoodie.orc.stripe.size](#hoodieorcstripesize) | 67108864 | Size of the memory buffer in bytes for writing<br />`Config Param: ORC_STRIPE_SIZE` [...]
+| [hoodie.parquet.block.size](#hoodieparquetblocksize) | 125829120 | Parquet RowGroup size in bytes. It's recommended to make this large enough that scan costs can be amortized by packing enough column values into a single row group.<br />`Config Param: PARQUET_BLOCK_SIZE` [...]
+| [hoodie.parquet.bloom.filter.enabled](#hoodieparquetbloomfilterenabled) | true | Control whether to write bloom filter or not. Default true. We can set to false in non bloom index cases for CPU resource saving.<br />`Config Param: PARQUET_WITH_BLOOM_FILTER_ENABLED`<br />`Since Version: 0.15.0` [...]
+| [hoodie.parquet.compression.ratio](#hoodieparquetcompressionratio) | 0.1 | Expected compression of parquet data used by Hudi, when it tries to size new parquet files. Increase this value, if bulk_insert is producing smaller than expected sized files<br />`Config Param: PARQUET_COMPRESSION_RATIO_FRACTION` [...]
+| [hoodie.parquet.dictionary.enabled](#hoodieparquetdictionaryenabled) | true | Whether to use dictionary encoding<br />`Config Param: PARQUET_DICTIONARY_ENABLED` [...]
+| [hoodie.parquet.field_id.write.enabled](#hoodieparquetfield_idwriteenabled) | true | Would only be effective with Spark 3.3+. Sets spark.sql.parquet.fieldId.write.enabled. If enabled, Spark will write out parquet native field ids that are stored inside StructField's metadata as parquet.field.id to parquet files.<br />`Config Param: PARQUET_FIELD_ID_WRITE_ENABLED`<br />`Since Version: 0.12.0` [...]
+| [hoodie.parquet.outputtimestamptype](#hoodieparquetoutputtimestamptype) | TIMESTAMP_MICROS | Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.<br />`Config Param: PARQUET_OUTPUT_TIMESTAMP_TYPE` [...]
+| [hoodie.parquet.page.size](#hoodieparquetpagesize) | 1048576 | Parquet page size in bytes. Page is the unit of read within a parquet file. Within a block, pages are compressed separately.<br />`Config Param: PARQUET_PAGE_SIZE` [...]
+| [hoodie.parquet.spark.row.write.support.class](#hoodieparquetsparkrowwritesupportclass) | org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport | Provided write support class should extend HoodieRowParquetWriteSupport class and it is loaded at runtime. This is only required when trying to override the existing write context when `hoodie.datasource.write.row.writer.enable=true`.<br />`Config Param: HOODIE_PARQUET_SPARK_ROW_WRITE_SUPPORT_CLASS`<br />`Since Version: 0.15.0` [...]
+| [hoodie.parquet.writelegacyformat.enabled](#hoodieparquetwritelegacyformatenabled) | false | Sets spark.sql.parquet.writeLegacyFormat. If true, data will be written in a way of Spark 1.4 and earlier. For example, decimal values will be written in Parquet's fixed-length byte array format which other systems such as Apache Hive and Apache Impala use. If false, the newer format in Parquet will be used. For example, decimals wil [...]
+| [hoodie.storage.class](#hoodiestorageclass) | org.apache.hudi.storage.hadoop.HoodieHadoopStorage | The fully-qualified class name of the `HoodieStorage` implementation class to instantiate. The provided class should implement `org.apache.hudi.storage.HoodieStorage`<br />`Config Param: HOODIE_STORAGE_CLASS`<br />`Since Version: 0.15.0` [...]
+---
+
+
+### Consistency Guard Configurations {#Consistency-Guard-Configurations}
+The consistency guard related config options, to help talk to eventually consistent object storage.(Tip: S3 is NOT eventually consistent anymore!)
+
+
+
+[**Advanced Configs**](#Consistency-Guard-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [_hoodie.optimistic.consistency.guard.enable](#_hoodieoptimisticconsistencyguardenable) | false | Enable consistency guard, which optimistically assumes consistency is achieved after a certain time period.<br />`Config Param: OPTIMISTIC_CONSISTENCY_GUARD_ENABLE`<br />`Since Version: 0.6.0` |
+| [hoodie.consistency.check.enabled](#hoodieconsistencycheckenabled) | false | Enabled to handle S3 eventual consistency issue. This property is no longer required since S3 is now strongly consistent. Will be removed in the future releases.<br />`Config Param: ENABLE`<br />`Since Version: 0.5.0`<br />`Deprecated since: 0.7.0` |
+| [hoodie.consistency.check.initial_interval_ms](#hoodieconsistencycheckinitial_interval_ms) | 400 | Amount of time (in ms) to wait, before checking for consistency after an operation on storage.<br />`Config Param: INITIAL_CHECK_INTERVAL_MS`<br />`Since Version: 0.5.0`<br />`Deprecated since: 0.7.0` |
+| [hoodie.consistency.check.max_checks](#hoodieconsistencycheckmax_checks) | 6 | Maximum number of consistency checks to perform, with exponential backoff.<br />`Config Param: MAX_CHECKS`<br />`Since Version: 0.5.0`<br />`Deprecated since: 0.7.0` |
+| [hoodie.consistency.check.max_interval_ms](#hoodieconsistencycheckmax_interval_ms) | 20000 | Maximum amount of time (in ms), to wait for consistency checking.<br />`Config Param: MAX_CHECK_INTERVAL_MS`<br />`Since Version: 0.5.0`<br />`Deprecated since: 0.7.0` |
+| [hoodie.optimistic.consistency.guard.sleep_time_ms](#hoodieoptimisticconsistencyguardsleep_time_ms) | 500 | Amount of time (in ms), to wait after which we assume storage is consistent.<br />`Config Param: OPTIMISTIC_CONSISTENCY_GUARD_SLEEP_TIME_MS`<br />`Since Version: 0.6.0` |
+---
+
+
+### FileSystem Guard Configurations {#FileSystem-Guard-Configurations}
+The filesystem retry related config options, to help deal with runtime exception like list/get/put/delete performance issues.
+
+
+
+[**Advanced Configs**](#FileSystem-Guard-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.filesystem.operation.retry.enable](#hoodiefilesystemoperationretryenable) | false | Enabled to handle list/get/delete etc file system performance issue.<br />`Config Param: FILESYSTEM_RETRY_ENABLE`<br />`Since Version: 0.11.0` |
+| [hoodie.filesystem.operation.retry.exceptions](#hoodiefilesystemoperationretryexceptions) | | The class name of the Exception that needs to be retried, separated by commas. Default is empty which means retry all the IOException and RuntimeException from FileSystem<br />`Config Param: RETRY_EXCEPTIONS`<br />`Since Version: 0.11.0` |
+| [hoodie.filesystem.operation.retry.initial_interval_ms](#hoodiefilesystemoperationretryinitial_interval_ms) | 100 | Amount of time (in ms) to wait, before retry to do operations on storage.<br />`Config Param: INITIAL_RETRY_INTERVAL_MS`<br />`Since Version: 0.11.0` |
+| [hoodie.filesystem.operation.retry.max_interval_ms](#hoodiefilesystemoperationretrymax_interval_ms) | 2000 | Maximum amount of time (in ms), to wait for next retry.<br />`Config Param: MAX_RETRY_INTERVAL_MS`<br />`Since Version: 0.11.0` |
+| [hoodie.filesystem.operation.retry.max_numbers](#hoodiefilesystemoperationretrymax_numbers) | 4 | Maximum number of retry actions to perform, with exponential backoff.<br />`Config Param: MAX_RETRY_NUMBERS`<br />`Since Version: 0.11.0` |
+---
+
+
+### File System View Storage Configurations {#File-System-View-Storage-Configurations}
+Configurations that control how file metadata is stored by Hudi, for transaction processing and queries.
+
+
+
+[**Advanced Configs**](#File-System-View-Storage-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.filesystem.remote.backup.view.enable](#hoodiefilesystemremotebackupviewenable) | true | Config to control whether backup needs to be configured if clients were not able to reach timeline service.<br />`Config Param: REMOTE_BACKUP_VIEW_ENABLE` [...]
+| [hoodie.filesystem.view.incr.timeline.sync.enable](#hoodiefilesystemviewincrtimelinesyncenable) | false | Controls whether or not, the file system view is incrementally updated as new actions are performed on the timeline.<br />`Config Param: INCREMENTAL_TIMELINE_SYNC_ENABLE` [...]
+| [hoodie.filesystem.view.remote.host](#hoodiefilesystemviewremotehost) | localhost | We expect this to be rarely hand configured.<br />`Config Param: REMOTE_HOST_NAME` [...]
+| [hoodie.filesystem.view.remote.port](#hoodiefilesystemviewremoteport) | 26754 | Port to serve file system view queries, when remote. We expect this to be rarely hand configured.<br />`Config Param: REMOTE_PORT_NUM` [...]
+| [hoodie.filesystem.view.remote.retry.enable](#hoodiefilesystemviewremoteretryenable) | false | Whether to enable API request retry for remote file system view.<br />`Config Param: REMOTE_RETRY_ENABLE`<br />`Since Version: 0.12.1` [...]
+| [hoodie.filesystem.view.remote.retry.exceptions](#hoodiefilesystemviewremoteretryexceptions) | | The class name of the Exception that needs to be retried, separated by commas. Default is empty which means retry all the IOException and RuntimeException from Remote Request.<br />`Config Param: RETRY_EXCEPTIONS`<br />`Since Version: 0.12.1` [...]
+| [hoodie.filesystem.view.remote.retry.initial_interval_ms](#hoodiefilesystemviewremoteretryinitial_interval_ms) | 100 | Amount of time (in ms) to wait, before retry to do operations on storage.<br />`Config Param: REMOTE_INITIAL_RETRY_INTERVAL_MS`<br />`Since Version: 0.12.1` [...]
+| [hoodie.filesystem.view.remote.retry.max_interval_ms](#hoodiefilesystemviewremoteretrymax_interval_ms) | 2000 | Maximum amount of time (in ms), to wait for next retry.<br />`Config Param: REMOTE_MAX_RETRY_INTERVAL_MS`<br />`Since Version: 0.12.1` [...]
+| [hoodie.filesystem.view.remote.retry.max_numbers](#hoodiefilesystemviewremoteretrymax_numbers) | 3 | Maximum number of retry for API requests against a remote file system view. e.g timeline server.<br />`Config Param: REMOTE_MAX_RETRY_NUMBERS`<br />`Since Version: 0.12.1` [...]
+| [hoodie.filesystem.view.remote.timeout.secs](#hoodiefilesystemviewremotetimeoutsecs) | 300 | Timeout in seconds, to wait for API requests against a remote file system view. e.g timeline server.<br />`Config Param: REMOTE_TIMEOUT_SECS` [...]
+| [hoodie.filesystem.view.rocksdb.base.path](#hoodiefilesystemviewrocksdbbasepath) | /tmp/hoodie_timeline_rocksdb | Path on local storage to use, when storing file system view in embedded kv store/rocksdb.<br />`Config Param: ROCKSDB_BASE_PATH` [...]
+| [hoodie.filesystem.view.secondary.type](#hoodiefilesystemviewsecondarytype) | MEMORY | Specifies the secondary form of storage for file system view, if the primary (e.g timeline server) is unavailable.<br />`Config Param: SECONDARY_VIEW_TYPE` [...]
+| [hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction](#hoodiefilesystemviewspillablebootstrapbasefilememfraction) | 0.05 | Fraction of the file system view memory, to be used for holding mapping to bootstrap base files.<br />`Config Param: BOOTSTRAP_BASE_FILE_MEM_FRACTION` [...]
+| [hoodie.filesystem.view.spillable.clustering.mem.fraction](#hoodiefilesystemviewspillableclusteringmemfraction) | 0.01 | Fraction of the file system view memory, to be used for holding clustering related metadata.<br />`Config Param: SPILLABLE_CLUSTERING_MEM_FRACTION` [...]
+| [hoodie.filesystem.view.spillable.compaction.mem.fraction](#hoodiefilesystemviewspillablecompactionmemfraction) | 0.8 | Fraction of the file system view memory, to be used for holding compaction related metadata.<br />`Config Param: SPILLABLE_COMPACTION_MEM_FRACTION` [...]
+| [hoodie.filesystem.view.spillable.dir](#hoodiefilesystemviewspillabledir) | /tmp/ | Path on local storage to use, when file system view is held in a spillable map.<br />`Config Param: SPILLABLE_DIR` [...]
+| [hoodie.filesystem.view.spillable.log.compaction.mem.fraction](#hoodiefilesystemviewspillablelogcompactionmemfraction) | 0.8 | Fraction of the file system view memory, to be used for holding log compaction related metadata.<br />`Config Param: SPILLABLE_LOG_COMPACTION_MEM_FRACTION`<br />`Since Version: 0.13.0` [...]
+| [hoodie.filesystem.view.spillable.mem](#hoodiefilesystemviewspillablemem) | 104857600 | Amount of memory to be used in bytes for holding file system view, before spilling to disk.<br />`Config Param: SPILLABLE_MEMORY` [...]
+| [hoodie.filesystem.view.spillable.replaced.mem.fraction](#hoodiefilesystemviewspillablereplacedmemfraction) | 0.01 | Fraction of the file system view memory, to be used for holding replace commit related metadata.<br />`Config Param: SPILLABLE_REPLACED_MEM_FRACTION` [...]
+| [hoodie.filesystem.view.type](#hoodiefilesystemviewtype) | MEMORY | File system view provides APIs for viewing the files on the underlying lake storage, as file groups and file slices. This config controls how such a view is held. Options include MEMORY,SPILLABLE_DISK,EMBEDDED_KV_STORE,REMOTE_ONLY,REMOTE_FIRST which provide different trade offs for memory usage and API request performance.<br [...]
+---
+
+
+### Archival Configs {#Archival-Configs}
+Configurations that control archival.
+
+
+
+[**Basic Configs**](#Archival-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.keep.max.commits](#hoodiekeepmaxcommits) | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. <br />`Config Param: MAX_COMMITS_TO_KEEP` |
+| [hoodie.keep.min.commits](#hoodiekeepmincommits) | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline.<br />`Config Param: MIN_COMMITS_TO_KEEP` |
+
+[**Advanced Configs**](#Archival-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
+| [hoodie.archive.async](#hoodiearchiveasync) | false | Only applies when hoodie.archive.automatic is turned on. When turned on runs archiver async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_ARCHIVE`<br />`Since Version: 0.11.0` [...]
+| [hoodie.archive.automatic](#hoodiearchiveautomatic) | true | When enabled, the archival table service is invoked immediately after each commit, to archive commits if we cross a maximum value of commits. It's recommended to enable this, to ensure number of active commits is bounded.<br />`Config Param: AUTO_ARCHIVE` [...]
+| [hoodie.archive.beyond.savepoint](#hoodiearchivebeyondsavepoint) | false | If enabled, archival will proceed beyond savepoint, skipping savepoint commits. If disabled, archival will stop at the earliest savepoint commit.<br />`Config Param: ARCHIVE_BEYOND_SAVEPOINT`<br />`Since Version: 0.12.0` [...]
+| [hoodie.archive.delete.parallelism](#hoodiearchivedeleteparallelism) | 100 | When performing archival operation, Hudi needs to delete the files of the archived instants in the active timeline in .hoodie folder. The file deletion also happens after merging small archived files into larger ones if enabled. This config limits the Spark parallelism for deleting files in both cases, i.e., parallelism of deleting files does not go above the configured value and the par [...]
+| [hoodie.archive.merge.enable](#hoodiearchivemergeenable) | false | When enable, hoodie will auto merge several small archive files into larger one. It's useful when storage scheme doesn't support append operation.<br />`Config Param: ARCHIVE_MERGE_ENABLE` [...]
+| [hoodie.archive.merge.files.batch.size](#hoodiearchivemergefilesbatchsize) | 10 | The number of small archive files to be merged at once.<br />`Config Param: ARCHIVE_MERGE_FILES_BATCH_SIZE` [...]
+| [hoodie.archive.merge.small.file.limit.bytes](#hoodiearchivemergesmallfilelimitbytes) | 20971520 | This config sets the archive file size limit below which an archive file becomes a candidate to be selected as such a small file.<br />`Config Param: ARCHIVE_MERGE_SMALL_FILE_LIMIT_BYTES` [...]
+| [hoodie.commits.archival.batch](#hoodiecommitsarchivalbatch) | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size.<br />`Config Param: COMMITS_ARCHIVAL_BATCH_SIZE` [...]
+---
+
+
+### Bootstrap Configs {#Bootstrap-Configs}
+Configurations that control how you want to bootstrap your existing tables for the first time into hudi. The bootstrap operation can flexibly avoid copying data over before you can use Hudi and support running the existing writers and new hudi writers in parallel, to validate the migration.
+
+
+
+[**Basic Configs**](#Bootstrap-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BASE_PATH`<br />`Since Version: 0.6.0` |
+
+[**Advanced Configs**](#Bootstrap-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.bootstrap.data.queries.only](#hoodiebootstrapdataqueriesonly) | false | Improves query performance, but queries cannot use hudi metadata fields<br />`Config Param: DATA_QUERIES_ONLY`<br />`Since Version: 0.14.0` [...]
+| [hoodie.bootstrap.full.input.provider](#hoodiebootstrapfullinputprovider) | org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider | Class to use for reading the bootstrap dataset partitions/files, for Bootstrap mode FULL_RECORD<br />`Config Param: FULL_BOOTSTRAP_INPUT_PROVIDER_CLASS_NAME`<br />`Since Version: 0.6.0` [...]
+| [hoodie.bootstrap.index.class](#hoodiebootstrapindexclass) | org.apache.hudi.common.bootstrap.index.hfile.HFileBootstrapIndex | Implementation to use, for mapping a skeleton base file to a bootstrap base file.<br />`Config Param: INDEX_CLASS_NAME`<br />`Since Version: 0.6.0` [...]
+| [hoodie.bootstrap.mode.selector](#hoodiebootstrapmodeselector) | org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped<br />`Config Param: MODE_SELECTOR_CLASS_NAME`<br />`Since Version: 0.6.0` [...]
+| [hoodie.bootstrap.mode.selector.regex](#hoodiebootstrapmodeselectorregex) | .* | Matches each bootstrap dataset partition against this regex and applies the mode below to it.<br />`Config Param: PARTITION_SELECTOR_REGEX_PATTERN`<br />`Since Version: 0.6.0` [...]
+| [hoodie.bootstrap.mode.selector.regex.mode](#hoodiebootstrapmodeselectorregexmode) | METADATA_ONLY | org.apache.hudi.client.bootstrap.BootstrapMode: Bootstrap mode for importing an existing table into Hudi FULL_RECORD: In this mode, the full record data is copied into hudi and metadata columns are added. A full record bootstrap is functionally equivalent to a bulk-insert. After a full record bootst [...]
+| [hoodie.bootstrap.parallelism](#hoodiebootstrapparallelism) | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to [...]
+| [hoodie.bootstrap.partitionpath.translator.class](#hoodiebootstrappartitionpathtranslatorclass) | org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPartitionPathTranslator | Translates the partition paths from the bootstrapped data into how is laid out as a Hudi table.<br />`Config Param: PARTITION_PATH_TRANSLATOR_CLASS_NAME`<br />`Since Version: 0.6.0` [...]
+---
+
+
+### Clean Configs {#Clean-Configs}
+Cleaning (reclamation of older/unused file groups/slices).
+
+
+
+[**Basic Configs**](#Clean-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.clean.async](#hoodiecleanasync) | false | Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_CLEAN` |
+| [hoodie.cleaner.commits.retained](#hoodiecleanercommitsretained) | 10 | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br />`Config Param: CLEANER_COMMITS_RETAINED` |
+
+[**Advanced Configs**](#Clean-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.clean.allow.multiple](#hoodiecleanallowmultiple) | false | Allows scheduling/executing multiple cleans by enabling this config. If users prefer to strictly ensure clean requests should be mutually exclusive, .i.e. a 2nd clean will not be scheduled if another clean is not yet completed to avoid repeat cleaning of same files, they might want to disable this config.<br />`Config Param: ALLOW_MULTIPLE_CLEANS`<br />`Since Version: 0.11.0`<br [...]
+| [hoodie.clean.automatic](#hoodiecleanautomatic) | true | When enabled, the cleaner table service is invoked immediately after each commit, to delete older file slices. It's recommended to enable this, to ensure metadata and data storage growth is bounded.<br />`Config Param: AUTO_CLEAN` [...]
+| [hoodie.clean.max.commits](#hoodiecleanmaxcommits) | 1 | Number of commits after the last clean operation, before scheduling of a new clean is attempted.<br />`Config Param: CLEAN_MAX_COMMITS` [...]
+| [hoodie.clean.trigger.strategy](#hoodiecleantriggerstrategy) | NUM_COMMITS | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits`.<br />`Config Param: CLEAN_TRIGGER_STRATEGY` [...]
+| [hoodie.cleaner.delete.bootstrap.base.file](#hoodiecleanerdeletebootstrapbasefile) | false | When set to true, cleaner also deletes the bootstrap base file when it's skeleton base file is cleaned. Turn this to true, if you want to ensure the bootstrap dataset storage is reclaimed over time, as the table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap base files are also physically deleted, to comply with data privac [...]
+| [hoodie.cleaner.fileversions.retained](#hoodiecleanerfileversionsretained) | 3 | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning.<br />`Config Param: CLEANER_FILE_VERSIONS_RETAINED` [...]
+| [hoodie.cleaner.hours.retained](#hoodiecleanerhoursretained) | 24 | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of hours to [...]
+| [hoodie.cleaner.incremental.mode](#hoodiecleanerincrementalmode) | true | When enabled, the plans for each cleaner service run is computed incrementally off the events in the timeline, since the last cleaner run. This is much more efficient than obtaining listings for the full table for each planning (even with a metadata table).<br />`Config Param: CLEANER_INCREMENTAL_MODE_ENABLE` [...]
+| [hoodie.cleaner.parallelism](#hoodiecleanerparallelism) | 200 | This config controls the behavior of both the cleaning plan and cleaning execution. Deriving the cleaning plan is parallelized at the table partition level, i.e., each table partition is processed by one Spark task to figure out the files to clean. The cleaner picks the configured parallelism if the number of table partitions is larger than this configured value. The parallelism [...]
+| [hoodie.cleaner.policy](#hoodiecleanerpolicy) | KEEP_LATEST_COMMITS | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query execu [...]
+| [hoodie.cleaner.policy.failed.writes](#hoodiecleanerpolicyfailedwrites) | EAGER | org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy: Policy that controls how to clean up failed writes. Hudi will delete any files written by failed writes to re-claim space. EAGER(default): Clean failed writes inline after every write operation. LAZY: Clean failed writes lazily after heartbeat timeout when the cleaning service runs. This policy is required whe [...]
+---
+
+
+### Clustering Configs {#Clustering-Configs}
+Configurations that control the clustering table service in hudi, which optimizes the storage layout for better query performance by sorting and sizing data files.
+
+
+
+[**Basic Configs**](#Clustering-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.plan.strategy.small.file.limit](#hoodieclusteringplanstrategysmallfilelimit) | 314572800 | Files smaller than the size in bytes specified here are candidates for clustering<br />`Config Param: PLAN_STRATEGY_SMALL_FILE_LIMIT`<br />`Since Version: 0.7.0` |
+| [hoodie.clustering.plan.strategy.target.file.max.bytes](#hoodieclusteringplanstrategytargetfilemaxbytes) | 1073741824 | Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups<br />`Config Param: PLAN_STRATEGY_TARGET_FILE_MAX_BYTES`<br />`Since Version: 0.7.0` |
+
+[**Advanced Configs**](#Clustering-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.clustering.plan.strategy.cluster.begin.partition](#hoodieclusteringplanstrategyclusterbeginpartition) | (N/A) | Begin partition used to filter partition (inclusive), only effective when the filter mode 'hoodie.clustering.plan.partition.filter.mode' is SELECTED_PARTITIONS<br />`Config Param: PARTITION_FILTER_BEGIN_PARTITION`<br />`Since Version: 0.11.0` [...]
+| [hoodie.clustering.plan.strategy.cluster.end.partition](#hoodieclusteringplanstrategyclusterendpartition) | (N/A) | End partition used to filter partition (inclusive), only effective when the filter mode 'hoodie.clustering.plan.partition.filter.mode' is SELECTED_PARTITIONS<br />`Config Param: PARTITION_FILTER_END_PARTITION`<br />`Since Version: 0.11.0` [...]
+| [hoodie.clustering.plan.strategy.partition.regex.pattern](#hoodieclusteringplanstrategypartitionregexpattern) | (N/A) | Filter clustering partitions that matched regex pattern<br />`Config Param: PARTITION_REGEX_PATTERN`<br />`Since Version: 0.11.0` [...]
+| [hoodie.clustering.plan.strategy.partition.selected](#hoodieclusteringplanstrategypartitionselected) | (N/A) | Partitions to run clustering<br />`Config Param: PARTITION_SELECTED`<br />`Since Version: 0.11.0` [...]
+| [hoodie.clustering.plan.strategy.sort.columns](#hoodieclusteringplanstrategysortcolumns) | (N/A) | Columns to sort the data by when clustering<br />`Config Param: PLAN_STRATEGY_SORT_COLUMNS`<br />`Since Version: 0.7.0` [...]
+| [hoodie.clustering.async.max.commits](#hoodieclusteringasyncmaxcommits) | 4 | Config to control frequency of async clustering<br />`Config Param: ASYNC_CLUSTERING_MAX_COMMITS`<br />`Since Version: 0.9.0` [...]
+| [hoodie.clustering.execution.strategy.class](#hoodieclusteringexecutionstrategyclass) | org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy | Config to provide a strategy class (subclass of RunClusteringStrategy) to define how the clustering plan is executed. By default, we sort the file groups in th plan by the specified columns, while meeting the configured target file sizes.<br />`Config Param: EXECUTI [...]
+| [hoodie.clustering.inline.max.commits](#hoodieclusteringinlinemaxcommits) | 4 | Config to control frequency of clustering planning<br />`Config Param: INLINE_CLUSTERING_MAX_COMMITS`<br />`Since Version: 0.7.0` [...]
+| [hoodie.clustering.max.parallelism](#hoodieclusteringmaxparallelism) | 15 | Maximum number of parallelism jobs submitted in clustering operation. If the resource is sufficient(Like Spark engine has enough idle executors), increasing this value will let the clustering job run faster, while it will give additional pressure to the execution engi [...]
+| [hoodie.clustering.plan.partition.filter.mode](#hoodieclusteringplanpartitionfiltermode) | NONE | org.apache.hudi.table.action.cluster.ClusteringPlanPartitionFilterMode: Partition filter mode used in the creation of clustering plan. NONE(default): Do not filter partitions. The clustering plan will include all partitions that have clustering candidates. REC [...]
+| [hoodie.clustering.plan.strategy.class](#hoodieclusteringplanstrategyclass) | org.apache.hudi.client.clustering.plan.strategy.SparkSizeBasedClusteringPlanStrategy | Config to provide a strategy class (subclass of ClusteringPlanStrategy) to create clustering plan i.e select what file groups are being clustered. Default strategy, looks at the clustering small file size limit (determined by hoodie.clustering.plan.strategy.small.file [...]
+| [hoodie.clustering.plan.strategy.daybased.lookback.partitions](#hoodieclusteringplanstrategydaybasedlookbackpartitions) | 2 | Number of partitions to list to create ClusteringPlan<br />`Config Param: DAYBASED_LOOKBACK_PARTITIONS`<br />`Since Version: 0.7.0` [...]
+| [hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions](#hoodieclusteringplanstrategydaybasedskipfromlatestpartitions) | 0 | Number of partitions to skip from latest when choosing partitions to create ClusteringPlan<br />`Config Param: PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST`<br />`Since Version: 0.9.0` [...]
+| [hoodie.clustering.plan.strategy.max.bytes.per.group](#hoodieclusteringplanstrategymaxbytespergroup) | 2147483648 | Each clustering operation can create multiple output file groups. Total amount of data processed by clustering operation is defined by below two properties (CLUSTERING_MAX_BYTES_PER_GROUP * CLUSTERING_MAX_NUM_GROUPS). Max amount of data to be included in one group<br [...]
+| [hoodie.clustering.plan.strategy.max.num.groups](#hoodieclusteringplanstrategymaxnumgroups) | 30 | Maximum number of groups to create as part of ClusteringPlan. Increasing groups will increase parallelism<br />`Config Param: PLAN_STRATEGY_MAX_GROUPS`<br />`Since Version: 0.7.0` [...]
+| [hoodie.clustering.plan.strategy.single.group.clustering.enabled](#hoodieclusteringplanstrategysinglegroupclusteringenabled) | true | Whether to generate clustering plan when there is only one file group involved, by default true<br />`Config Param: PLAN_STRATEGY_SINGLE_GROUP_CLUSTERING_ENABLED`<br />`Since Version: 0.14.0` [...]
+| [hoodie.clustering.rollback.pending.replacecommit.on.conflict](#hoodieclusteringrollbackpendingreplacecommitonconflict) | false | If updates are allowed to file groups pending clustering, then set this config to rollback failed or pending clustering instants. Pending clustering will be rolled back ONLY IF there is conflict between incoming upsert and filegroup to be clustered. Please exercise ca [...]
+| [hoodie.clustering.schedule.inline](#hoodieclusteringscheduleinline) | false | When set to true, clustering service will be attempted for inline scheduling after each write. Users have to ensure they have a separate job to run async clustering(execution) for the one scheduled by this writer. Users can choose to set both `hoodie.clustering.inline [...]
+| [hoodie.clustering.updates.strategy](#hoodieclusteringupdatesstrategy) | org.apache.hudi.client.clustering.update.strategy.SparkRejectUpdateStrategy | Determines how to handle updates, deletes to file groups that are under clustering. Default strategy just rejects the update<br />`Config Param: UPDATES_STRATEGY`<br />`Since Version: 0.7.0` [...]
+| [hoodie.layout.optimize.build.curve.sample.size](#hoodielayoutoptimizebuildcurvesamplesize) | 200000 | Determines target sample size used by the Boundary-based Interleaved Index method of building space-filling curve. Larger sample size entails better layout optimization outcomes, at the expense of higher memory footprint.<br />`Config Param: LAYOUT_OPTIMIZE_BUILD_CURV [...]
+| [hoodie.layout.optimize.curve.build.method](#hoodielayoutoptimizecurvebuildmethod) | DIRECT | org.apache.hudi.config.HoodieClusteringConfig$SpatialCurveCompositionStrategyType: This configuration only has effect if `hoodie.layout.optimize.strategy` is set to either "z-order" or "hilbert" (i.e. leveraging space-filling curves). This configuration controls the t [...]
+| [hoodie.layout.optimize.data.skipping.enable](#hoodielayoutoptimizedataskippingenable) | true | Enable data skipping by collecting statistics once layout optimization is complete.<br />`Config Param: LAYOUT_OPTIMIZE_DATA_SKIPPING_ENABLE`<br />`Since Version: 0.10.0`<br />`Deprecated since: 0.11.0` [...]
+| [hoodie.layout.optimize.enable](#hoodielayoutoptimizeenable) | false | This setting has no effect. Please refer to clustering configuration, as well as LAYOUT_OPTIMIZE_STRATEGY config to enable advanced record layout optimization strategies<br />`Config Param: LAYOUT_OPTIMIZE_ENABLE`<br />`Since Version: 0.10.0`<br />`Deprecated since: 0 [...]
+| [hoodie.layout.optimize.strategy](#hoodielayoutoptimizestrategy) | LINEAR | org.apache.hudi.config.HoodieClusteringConfig$LayoutOptimizationStrategy: Determines ordering strategy for records layout optimization. LINEAR(default): Orders records lexicographically ZORDER: Orders records along Z-order spatial-curve. HILBERT: Orders re [...]
+---
+
+
+### Compaction Configs {#Compaction-Configs}
+Configurations that control compaction (merging of log files onto a new base files).
+
+
+
+[**Basic Configs**](#Compaction-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.compact.inline](#hoodiecompactinline) | false | When set to true, compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br />`Config Param: INLINE_COMPACT` |
+| [hoodie.compact.inline.max.delta.commits](#hoodiecompactinlinemaxdeltacommits) | 5 | Number of delta commits after the last compaction, before scheduling of a new compaction is attempted. This config takes effect only for the compaction triggering strategy based on the number of commits, i.e., NUM_COMMITS, NUM_COMMITS_AFTER_LAST_REQUEST, NUM_AND_TIME, and NUM_OR_TIME.<br />`Config Param: INLINE_COMPACT_NUM_DELTA_COMMITS` |
+
+[**Advanced Configs**](#Compaction-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ----------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.compact.inline.max.delta.seconds](#hoodiecompactinlinemaxdeltaseconds) | 3600 | Number of elapsed seconds after the last compaction, before scheduling a new one. This config takes effect only for the compaction triggering strategy based on the elapsed time, i.e., TIME_ELAPSED, NUM_AND_TIME, and NUM_OR_TIME.<br />`Config Param: INLINE_COMPACT_TIME_DELTA_SECONDS` [...]
+| [hoodie.compact.inline.trigger.strategy](#hoodiecompactinlinetriggerstrategy) | NUM_COMMITS | org.apache.hudi.table.action.compact.CompactionTriggerStrategy: Controls when compaction is scheduled. NUM_COMMITS(default): triggers compaction when there are at least N delta commits after last completed compaction. NUM_COMMITS_AFTER_LAST_REQUEST: triggers compaction when there are at least N delta comm [...]
+| [hoodie.compact.schedule.inline](#hoodiecompactscheduleinline) | false | When set to true, compaction service will be attempted for inline scheduling after each write. Users have to ensure they have a separate job to run async compaction(execution) for the one scheduled by this writer. Users can choose to set both `hoodie.compact.inline` and `hoodie.compact.schedule.inline` to false a [...]
+| [hoodie.compaction.daybased.target.partitions](#hoodiecompactiondaybasedtargetpartitions) | 10 | Used by org.apache.hudi.io.compact.strategy.DayBasedCompactionStrategy to denote the number of latest partitions to compact during a compaction run.<br />`Config Param: TARGET_PARTITIONS_PER_DAYBASED_COMPACTION` [...]
+| [hoodie.compaction.lazy.block.read](#hoodiecompactionlazyblockread) | true | When merging the delta log files, this config helps to choose whether the log blocks should be read lazily or not. Choose true to use lazy block reading (low memory usage, but incurs seeks to each block header) or false for immediate block read (higher memory usage)<br />`Config Param: COMPACTION_LAZY_BLOCK_READ_ [...]
+| [hoodie.compaction.logfile.num.threshold](#hoodiecompactionlogfilenumthreshold) | 0 | Only if the log file num is greater than the threshold, the file group will be compacted.<br />`Config Param: COMPACTION_LOG_FILE_NUM_THRESHOLD`<br />`Since Version: 0.13.0` [...]
+| [hoodie.compaction.logfile.size.threshold](#hoodiecompactionlogfilesizethreshold) | 0 | Only if the log file size is greater than the threshold in bytes, the file group will be compacted.<br />`Config Param: COMPACTION_LOG_FILE_SIZE_THRESHOLD` [...]
+| [hoodie.compaction.reverse.log.read](#hoodiecompactionreverselogread) | false | HoodieLogFormatReader reads a logfile in the forward direction starting from pos=0 to pos=file_length. If this config is set to true, the reader reads the logfile in reverse direction, from pos=file_length to pos=0<br />`Config Param: COMPACTION_REVERSE_LOG_READ_ENABLE` [...]
+| [hoodie.compaction.strategy](#hoodiecompactionstrategy) | org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy | Compaction strategy decides which file groups are picked up for compaction during each compaction run. By default. Hudi picks the log file with most accumulated unmerged data<br />`Config Param: COMPACTION_STRATEGY` [...]
+| [hoodie.compaction.target.io](#hoodiecompactiontargetio) | 512000 | Amount of MBs to spend during compaction run for the LogFileSizeBasedCompactionStrategy. This value helps bound ingestion latency while compaction is run inline mode.<br />`Config Param: TARGET_IO_PER_COMPACTION_IN_MB` [...]
+| [hoodie.copyonwrite.insert.auto.split](#hoodiecopyonwriteinsertautosplit) | true | Config to control whether we control insert split sizes automatically based on average record sizes. It's recommended to keep this turned on, since hand tuning is otherwise extremely cumbersome.<br />`Config Param: COPY_ON_WRITE_AUTO_SPLIT_INSERTS` [...]
+| [hoodie.copyonwrite.insert.split.size](#hoodiecopyonwriteinsertsplitsize) | 500000 | Number of inserts assigned for each partition/bucket for writing. We based the default on writing out 100MB files, with at least 1kb records (100K records per file), and over provision to 500K. As long as auto-tuning of splits is turned on, this only affects the first write, where there is no history to learn [...]
+| [hoodie.copyonwrite.record.size.estimate](#hoodiecopyonwriterecordsizeestimate) | 1024 | The average record size. If not explicitly specified, hudi will compute the record size estimate compute dynamically based on commit metadata. This is critical in computing the insert parallelism and bin-packing inserts into small files.<br />`Config Param: COPY_ON_WRITE_RECORD_SIZE_ESTIMATE` [...]
+| [hoodie.log.compaction.blocks.threshold](#hoodielogcompactionblocksthreshold) | 5 | Log compaction can be scheduled if the no. of log blocks crosses this threshold value. This is effective only when log compaction is enabled via hoodie.log.compaction.inline<br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` [...]
+| [hoodie.log.compaction.enable](#hoodielogcompactionenable) | false | By enabling log compaction through this config, log compaction will also get enabled for the metadata table.<br />`Config Param: ENABLE_LOG_COMPACTION`<br />`Since Version: 0.14.0` [...]
+| [hoodie.log.compaction.inline](#hoodielogcompactioninline) | false | When set to true, logcompaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` [...]
+| [hoodie.optimized.log.blocks.scan.enable](#hoodieoptimizedlogblocksscanenable) | false | New optimized scan for log blocks that handles all multi-writer use-cases while appending to log files. It also differentiates original blocks written by ingestion writers and compacted blocks written log compaction.<br />`Config Param: ENABLE_OPTIMIZED_LOG_BLOCKS_SCAN`<br />`Since Version: 0.13.0` [...]
+| [hoodie.parquet.small.file.limit](#hoodieparquetsmallfilelimit) | 104857600 | During upsert operation, we opportunistically expand existing small files on storage, instead of writing new files, to keep number of files to an optimum. This config sets the file size limit below which a file on storage becomes a candidate to be selected as such a `small file`. By default, treat any file <= [...]
+| [hoodie.record.size.estimation.threshold](#hoodierecordsizeestimationthreshold) | 1.0 | We use the previous commits' metadata to calculate the estimated record size and use it to bin pack records into partitions. If the previous commit is too small to make an accurate estimation, Hudi will search commits in the reverse order, until we find a commit that has totalBytesWritten larger than (PARQUET_ [...]
+---
+
+
+### Error table Configs {#Error-table-Configs}
+Configurations that are required for Error table configs
+
+
+
+[**Basic Configs**](#Error-table-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.errortable.base.path](#hoodieerrortablebasepath) | (N/A) | Base path for error table under which all error records would be stored.<br />`Config Param: ERROR_TABLE_BASE_PATH` |
+| [hoodie.errortable.target.table.name](#hoodieerrortabletargettablename) | (N/A) | Table name to be used for the error table<br />`Config Param: ERROR_TARGET_TABLE` |
+| [hoodie.errortable.write.class](#hoodieerrortablewriteclass) | (N/A) | Class which handles the error table writes. This config is used to configure a custom implementation for Error Table Writer. Specify the full class name of the custom error table writer as a value for this config<br />`Config Param: ERROR_TABLE_WRITE_CLASS` |
+| [hoodie.errortable.enable](#hoodieerrortableenable) | false | Config to enable error table. If the config is enabled, all the records with processing error in DeltaStreamer are transferred to error table.<br />`Config Param: ERROR_TABLE_ENABLED` |
+| [hoodie.errortable.insert.shuffle.parallelism](#hoodieerrortableinsertshuffleparallelism) | 200 | Config to set insert shuffle parallelism. The config is similar to hoodie.insert.shuffle.parallelism config but applies to the error table.<br />`Config Param: ERROR_TABLE_INSERT_PARALLELISM_VALUE` |
+| [hoodie.errortable.upsert.shuffle.parallelism](#hoodieerrortableupsertshuffleparallelism) | 200 | Config to set upsert shuffle parallelism. The config is similar to hoodie.upsert.shuffle.parallelism config but applies to the error table.<br />`Config Param: ERROR_TABLE_UPSERT_PARALLELISM_VALUE` |
+| [hoodie.errortable.validate.recordcreation.enable](#hoodieerrortablevalidaterecordcreationenable) | true | Records that fail to be created due to keygeneration failure or other issues will be sent to the Error Table<br />`Config Param: ERROR_ENABLE_VALIDATE_RECORD_CREATION`<br />`Since Version: 0.15.0` |
+| [hoodie.errortable.validate.targetschema.enable](#hoodieerrortablevalidatetargetschemaenable) | false | Records with schema mismatch with Target Schema are sent to Error Table.<br />`Config Param: ERROR_ENABLE_VALIDATE_TARGET_SCHEMA` |
+| [hoodie.errortable.write.failure.strategy](#hoodieerrortablewritefailurestrategy) | ROLLBACK_COMMIT | The config specifies the failure strategy if error table write fails. Use one of - [ROLLBACK_COMMIT (Rollback the corresponding base table write commit for which the error events were triggered) , LOG_ERROR (Error is logged but the base table write succeeds) ]<br />`Config Param: ERROR_TABLE_WRITE_FAILURE_STRATEGY` |
+---
+
+
+### Layout Configs {#Layout-Configs}
+Configurations that control storage layout and data distribution, which defines how the files are organized within a table.
+
+
+
+[**Advanced Configs**](#Layout-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.storage.layout.partitioner.class](#hoodiestoragelayoutpartitionerclass) | (N/A) | Partitioner class, it is used to distribute data in a specific way.<br />`Config Param: LAYOUT_PARTITIONER_CLASS_NAME` [...]
+| [hoodie.storage.layout.type](#hoodiestoragelayouttype) | DEFAULT | org.apache.hudi.table.storage.HoodieStorageLayout$LayoutType: Determines how the files are organized within a table. DEFAULT(default): Each file group contains records of a certain set of keys, without particular grouping criteria. BUCKET: Each file group contains records of a set of keys which map to a certain range of hash values, so that using the hash function can easily identify th [...]
+---
+
+
+### Memory Configurations {#Memory-Configurations}
+Controls memory usage for compaction and merges, performed internally by Hudi.
+
+
+
+[**Advanced Configs**](#Memory-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.memory.compaction.max.size](#hoodiememorycompactionmaxsize) | (N/A) | Maximum amount of memory used in bytes for compaction operations in bytes , before spilling to local storage.<br />`Config Param: MAX_MEMORY_FOR_COMPACTION` [...]
+| [hoodie.memory.spillable.map.path](#hoodiememoryspillablemappath) | (N/A) | Default file path for spillable map<br />`Config Param: SPILLABLE_MAP_BASE_PATH` [...]
+| [hoodie.memory.compaction.fraction](#hoodiememorycompactionfraction) | 0.6 | HoodieCompactedLogScanner reads logblocks, converts records to HoodieRecords and then merges these log blocks and records. At any point, the number of entries in a log block can be less than or equal to the number of entries in the corresponding parquet file. This can lead to OOM in the Scanner. Hence, a spillable map helps alleviate the memory pressure. Use this config to set the max [...]
+| [hoodie.memory.dfs.buffer.max.size](#hoodiememorydfsbuffermaxsize) | 16777216 | Property to control the max memory in bytes for dfs input stream buffer size<br />`Config Param: MAX_DFS_STREAM_BUFFER_SIZE` [...]
+| [hoodie.memory.merge.fraction](#hoodiememorymergefraction) | 0.6 | This fraction is multiplied with the user memory fraction (1 - spark.memory.fraction) to get a final fraction of heap space to use during merge<br />`Config Param: MAX_MEMORY_FRACTION_FOR_MERGE` [...]
+| [hoodie.memory.merge.max.size](#hoodiememorymergemaxsize) | 1073741824 | Maximum amount of memory used in bytes for merge operations, before spilling to local storage.<br />`Config Param: MAX_MEMORY_FOR_MERGE` [...]
+| [hoodie.memory.writestatus.failure.fraction](#hoodiememorywritestatusfailurefraction) | 0.1 | Property to control how what fraction of the failed record, exceptions we report back to driver. Default is 10%. If set to 100%, with lot of failures, this can cause memory pressure, cause OOMs and mask actual data errors.<br />`Config Param: WRITESTATUS_FAILURE_FRACTION` [...]
+---
+
+
+### Write Configurations {#Write-Configurations}
+Configurations that control write behavior on Hudi tables. These can be directly passed down from even higher level frameworks (e.g Spark datasources, Flink sink) and utilities (e.g Hudi Streamer).
+
+
+
+[**Basic Configs**](#Write-Configurations-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br />`Config Param: TBL_NAME` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD_NAME` [...]
+| [hoodie.write.concurrency.mode](#hoodiewriteconcurrencymode) | SINGLE_WRITER | org.apache.hudi.common.model.WriteConcurrencyMode: Concurrency modes for write operations. SINGLE_WRITER(default): Only one active writer to the table. Maximizes throughput. OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table with lazy conflict resolution using locks. This means that only one writer succeeds if multiple writers write to the same file group. [...]
+
+[**Advanced Configs**](#Write-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.avro.schema](#hoodieavroschema) | (N/A) | Schema string representing the current write schema of the table. Hudi passes this to implementations of HoodieRecordPayload to convert incoming records to avro. This is also used as the write schema evolving records during an update.<br />`Config Param: AVRO_SCHEMA_STRING` [...]
+| [hoodie.bulkinsert.user.defined.partitioner.class](#hoodiebulkinsertuserdefinedpartitionerclass) | (N/A) | If specified, this class will be used to re-partition records before they are bulk inserted. This can be used to sort, pack, cluster data optimally for common query patterns. For now we support a build-in user defined bulkinsert partitioner org.apache.hudi.execution.bulkinsert.RDDCustomColumnsS [...]
+| [hoodie.bulkinsert.user.defined.partitioner.sort.columns](#hoodiebulkinsertuserdefinedpartitionersortcolumns) | (N/A) | Columns to sort the data by when use org.apache.hudi.execution.bulkinsert.RDDCustomColumnsSortPartitioner as user defined partitioner during bulk_insert. For example 'column1,column2'<br />`Config Param: BULKINSERT_USER_DEFINED_PARTITIONER_SORT_COLUMNS` [...]
+| [hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass) | (N/A) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records.<br />`Config Param: KEYGENERATOR_CLASS_NAME` [...]
+| [hoodie.internal.schema](#hoodieinternalschema) | (N/A) | Schema string representing the latest schema of the table. Hudi passes this to implementations of evolution of schema<br />`Config Param: INTERNAL_SCHEMA_STRING` [...]
+| [hoodie.write.schema](#hoodiewriteschema) | (N/A) | Config allowing to override writer's schema. This might be necessary in cases when writer's schema derived from the incoming dataset might actually be different from the schema we actually want to use when writing. This, for ex, could be the case for'partial-update' use-cases (like `MERGE INTO` [...]
+| [_.hoodie.allow.multi.write.on.same.instant](#_hoodieallowmultiwriteonsameinstant) | false | <br />`Config Param: ALLOW_MULTI_WRITE_ON_SAME_INSTANT_ENABLE` [...]
+| [hoodie.allow.empty.commit](#hoodieallowemptycommit) | true | Whether to allow generation of empty commits, even if no data was written in the commit. It's useful in cases where extra metadata needs to be published regardless e.g tracking source offsets when ingesting data<br />`Config Param: ALLOW_EMPTY_COMMIT` [...]
+| [hoodie.allow.operation.metadata.field](#hoodieallowoperationmetadatafield) | false | Whether to include '_hoodie_operation' in the metadata fields. Once enabled, all the changes of a record are persisted to the delta log directly without merge<br />`Config Param: ALLOW_OPERATION_METADATA_FIELD`<br />`Since Version: 0.9.0` [...]
+| [hoodie.auto.adjust.lock.configs](#hoodieautoadjustlockconfigs) | false | Auto adjust lock configurations when metadata table is enabled and for async table services.<br />`Config Param: AUTO_ADJUST_LOCK_CONFIGS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.auto.commit](#hoodieautocommit) | true | Controls whether a write operation should auto commit. This can be turned off to perform inspection of the uncommitted write before deciding to commit.<br />`Config Param: AUTO_COMMIT_ENABLE` [...]
+| [hoodie.avro.schema.external.transformation](#hoodieavroschemaexternaltransformation) | false | When enabled, records in older schema are rewritten into newer schema during upsert,delete and background compaction,clustering operations.<br />`Config Param: AVRO_EXTERNAL_SCHEMA_TRANSFORMATION_ENABLE` [...]
+| [hoodie.avro.schema.validate](#hoodieavroschemavalidate) | false | Validate the schema used for the write against the latest schema, for backwards compatibility.<br />`Config Param: AVRO_SCHEMA_VALIDATE_ENABLE` [...]
+| [hoodie.bulkinsert.shuffle.parallelism](#hoodiebulkinsertshuffleparallelism) | 0 | For large initial imports using bulk_insert operation, controls the parallelism to use for sort modes or custom partitioning done before writing records to the table. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards [...]
+| [hoodie.bulkinsert.sort.mode](#hoodiebulkinsertsortmode) | NONE | org.apache.hudi.execution.bulkinsert.BulkInsertSortMode: Modes for sorting records during bulk insert. NONE(default): No sorting. Fastest and matches `spark.write.parquet()` in number of files and overhead. GLOBAL_SORT: This ensures best file sizes, with lowest memory overhead at cost o [...]
+| [hoodie.client.heartbeat.interval_in_ms](#hoodieclientheartbeatinterval_in_ms) | 60000 | Writers perform heartbeats to indicate liveness. Controls how often (in ms), such heartbeats are registered to lake storage.<br />`Config Param: CLIENT_HEARTBEAT_INTERVAL_IN_MS` [...]
+| [hoodie.client.heartbeat.tolerable.misses](#hoodieclientheartbeattolerablemisses) | 2 | Number of heartbeat misses, before a writer is deemed not alive and all pending writes are aborted.<br />`Config Param: CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES` [...]
+| [hoodie.client.init.callback.classes](#hoodieclientinitcallbackclasses) | | Fully-qualified class names of the Hudi client init callbacks to run at the initialization of the Hudi client. The class names are separated by `,`. The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`.By default, no Hudi client init callback is executed.<br />`C [...]
+| [hoodie.combine.before.delete](#hoodiecombinebeforedelete) | true | During delete operations, controls whether we should combine deletes (and potentially also upserts) before writing to storage.<br />`Config Param: COMBINE_BEFORE_DELETE` [...]
+| [hoodie.combine.before.insert](#hoodiecombinebeforeinsert) | false | When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage.<br />`Config Param: COMBINE_BEFORE_INSERT` [...]
+| [hoodie.combine.before.upsert](#hoodiecombinebeforeupsert) | true | When upserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage. This should be turned off only if you are absolutely certain that there are no duplicates incoming, otherwise it can lead to duplicate keys and violate the unique [...]
+| [hoodie.consistency.check.initial_interval_ms](#hoodieconsistencycheckinitial_interval_ms) | 2000 | Initial time between successive attempts to ensure written data's metadata is consistent on storage. Grows with exponential backoff after the initial value.<br />`Config Param: INITIAL_CONSISTENCY_CHECK_INTERVAL_MS` [...]
+| [hoodie.consistency.check.max_checks](#hoodieconsistencycheckmax_checks) | 7 | Maximum number of checks, for consistency of written data.<br />`Config Param: MAX_CONSISTENCY_CHECKS` [...]
+| [hoodie.consistency.check.max_interval_ms](#hoodieconsistencycheckmax_interval_ms) | 300000 | Max time to wait between successive attempts at performing consistency checks<br />`Config Param: MAX_CONSISTENCY_CHECK_INTERVAL_MS` [...]
+| [hoodie.datasource.write.keygenerator.type](#hoodiedatasourcewritekeygeneratortype) | SIMPLE | **Note** This is being actively worked on. Please use `hoodie.datasource.write.keygenerator.class` instead. org.apache.hudi.keygen.constant.KeyGeneratorType: Key generator type, indicating the key generator class to use, that implements `org.apache.hudi.keygen.KeyGenerator`. SIMPLE(default) [...]
+| [hoodie.datasource.write.payload.class](#hoodiedatasourcewritepayloadclass) | org.apache.hudi.common.model.OverwriteWithLatestAvroPayload | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br />`Config Param: WRITE_PAYLOAD_CLASS_NAME` [...]
+| [hoodie.datasource.write.record.merger.impls](#hoodiedatasourcewriterecordmergerimpls) | org.apache.hudi.common.model.HoodieAvroRecordMerger | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These merger impls will filter by hoodie.datasource.write.record.merger.strategy Hudi will pick most efficient implementation to perform merging/combining of the records (during update, readin [...]
+| [hoodie.datasource.write.record.merger.strategy](#hoodiedatasourcewriterecordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in hoodie.datasource.write.record.merger.impls which has the same merger strategy id<br />`Config Param: RECORD_MERGER_STRATEGY`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.write.schema.allow.auto.evolution.column.drop](#hoodiedatasourcewriteschemaallowautoevolutioncolumndrop) | false | Controls whether table's schema is allowed to automatically evolve when incoming batch's schema can have any of the columns dropped. By default, Hudi will not allow this kind of (auto) schema evolution. Set this config to true to allow table's schema to be updated automatically when columns are [...]
+| [hoodie.delete.shuffle.parallelism](#hoodiedeleteshuffleparallelism) | 0 | Parallelism used for delete operation. Delete operations also performs shuffles, similar to upsert operation. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards Hudi by default automatically uses the parallelism deduce [...]
+| [hoodie.embed.timeline.server](#hoodieembedtimelineserver) | true | When true, spins up an instance of the timeline server (meta server that serves cached file listings, statistics),running on each writer's driver process, accepting requests during the write from executors.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_ENABLE` [...]
+| [hoodie.embed.timeline.server.async](#hoodieembedtimelineserverasync) | false | Controls whether or not, the requests to the timeline server are processed in asynchronous fashion, potentially improving throughput.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_USE_ASYNC_ENABLE` [...]
+| [hoodie.embed.timeline.server.gzip](#hoodieembedtimelineservergzip) | true | Controls whether gzip compression is used, for large responses from the timeline server, to improve latency.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_COMPRESS_ENABLE` [...]
+| [hoodie.embed.timeline.server.port](#hoodieembedtimelineserverport) | 0 | Port at which the timeline server listens for requests. When running embedded in each writer, it picks a free port and communicates to all the executors. This should rarely be changed.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_PORT_NUM` [...]
+| [hoodie.embed.timeline.server.reuse.enabled](#hoodieembedtimelineserverreuseenabled) | false | Controls whether the timeline server instance should be cached and reused across the tablesto avoid startup costs and server overhead. This should only be used if you are running multiple writers in the same JVM.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_REUSE_ENABLED` [...]
+| [hoodie.embed.timeline.server.threads](#hoodieembedtimelineserverthreads) | -1 | Number of threads to serve requests in the timeline server. By default, auto configured based on the number of underlying cores.<br />`Config Param: EMBEDDED_TIMELINE_NUM_SERVER_THREADS` [...]
+| [hoodie.fail.on.timeline.archiving](#hoodiefailontimelinearchiving) | true | Timeline archiving removes older instants from the timeline, after each write operation, to minimize metadata overhead. Controls whether or not, the write should be failed as well, if such archiving fails.<br />`Config Param: FAIL_ON_TIMELINE_ARCHIVING_ENABLE` [...]
+| [hoodie.fail.writes.on.inline.table.service.exception](#hoodiefailwritesoninlinetableserviceexception) | true | Table services such as compaction and clustering can fail and prevent syncing to the metaclient. Set this to true to fail writes when table services fail<br />`Config Param: FAIL_ON_INLINE_TABLE_SERVICE_EXCEPTION`<br />`Since Version: 0.13.0` [...]
+| [hoodie.fileid.prefix.provider.class](#hoodiefileidprefixproviderclass) | org.apache.hudi.table.RandomFileIdPrefixProvider | File Id Prefix provider class, that implements `org.apache.hudi.fileid.FileIdPrefixProvider`<br />`Config Param: FILEID_PREFIX_PROVIDER_CLASS`<br />`Since Version: 0.10.0` [...]
+| [hoodie.finalize.write.parallelism](#hoodiefinalizewriteparallelism) | 200 | Parallelism for the write finalization internal operation, which involves removing any partially written files from lake storage, before committing the write. Reduce this value, if the high number of tasks incur delays for smaller tables or low latency writes.<br />`Config Param: FINALIZE_WRITE [...]
+| [hoodie.insert.shuffle.parallelism](#hoodieinsertshuffleparallelism) | 0 | Parallelism for inserting records into the table. Inserts can shuffle data before writing to tune file sizes and optimize the storage layout. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards Hudi by default automatic [...]
+| [hoodie.markers.delete.parallelism](#hoodiemarkersdeleteparallelism) | 100 | Determines the parallelism for deleting marker files, which are used to track all files (valid or invalid/partial) written during a write operation. Increase this value if delays are observed, with large batch writes.<br />`Config Param: MARKERS_DELETE_PARALLELISM_VALUE` [...]
+| [hoodie.markers.timeline_server_based.batch.interval_ms](#hoodiemarkerstimeline_server_basedbatchinterval_ms) | 50 | The batch interval in milliseconds for marker creation batch processing<br />`Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_INTERVAL_MS`<br />`Since Version: 0.9.0` [...]
+| [hoodie.markers.timeline_server_based.batch.num_threads](#hoodiemarkerstimeline_server_basedbatchnum_threads) | 20 | Number of threads to use for batch processing marker creation requests at the timeline server<br />`Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_NUM_THREADS`<br />`Since Version: 0.9.0` [...]
+| [hoodie.merge.allow.duplicate.on.inserts](#hoodiemergeallowduplicateoninserts) | true | When enabled, we allow duplicate keys even if inserts are routed to merge with an existing file (for ensuring file sizing). This is only relevant for insert operation, since upsert, delete operations will ensure unique key constraints are maintained.<br />`Config Param: MERGE_ALLOW_DUPLICATE_ON [...]
+| [hoodie.merge.data.validation.enabled](#hoodiemergedatavalidationenabled) | false | When enabled, data validation checks are performed during merges to ensure expected number of records after merge operation.<br />`Config Param: MERGE_DATA_VALIDATION_CHECK_ENABLE` [...]
+| [hoodie.merge.small.file.group.candidates.limit](#hoodiemergesmallfilegroupcandidateslimit) | 1 | Limits number of file groups, whose base file satisfies small-file limit, to consider for appending records during upsert operation. Only applicable to MOR tables<br />`Config Param: MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT` [...]
+| [hoodie.release.resource.on.completion.enable](#hoodiereleaseresourceoncompletionenable) | true | Control to enable release all persist rdds when the spark job finish.<br />`Config Param: RELEASE_RESOURCE_ENABLE`<br />`Since Version: 0.11.0` [...]
+| [hoodie.rollback.instant.backup.dir](#hoodierollbackinstantbackupdir) | .rollback_backup | Path where instants being rolled back are copied. If not absolute path then a directory relative to .hoodie folder is created.<br />`Config Param: ROLLBACK_INSTANT_BACKUP_DIRECTORY` [...]
+| [hoodie.rollback.instant.backup.enabled](#hoodierollbackinstantbackupenabled) | false | Backup instants removed during rollback and restore (useful for debugging)<br />`Config Param: ROLLBACK_INSTANT_BACKUP_ENABLED` [...]
+| [hoodie.rollback.parallelism](#hoodierollbackparallelism) | 100 | This config controls the parallelism for rollback of commits. Rollbacks perform deletion of files or logging delete blocks to file groups on storage in parallel. The configure value limits the parallelism so that the number of Spark tasks do not exceed the value. If rollback is slow due to the [...]
+| [hoodie.rollback.using.markers](#hoodierollbackusingmarkers) | true | Enables a more efficient mechanism for rollbacks based on the marker files generated during the writes. Turned on by default.<br />`Config Param: ROLLBACK_USING_MARKERS_ENABLE` [...]
+| [hoodie.schema.cache.enable](#hoodieschemacacheenable) | false | cache query internalSchemas in driver/executor side<br />`Config Param: ENABLE_INTERNAL_SCHEMA_CACHE` [...]
+| [hoodie.sensitive.config.keys](#hoodiesensitiveconfigkeys) | ssl,tls,sasl,auth,credentials | Comma separated list of filters for sensitive config keys. Hudi Streamer will not print any configuration which contains the configured filter. For example with a configured filter `ssl`, value for config `ssl.trustore.location` would be masked.<br />`Config Param: SENSITIVE_CONFIG_KEYS_FILTER` [...]
+| [hoodie.skip.default.partition.validation](#hoodieskipdefaultpartitionvalidation) | false | When table is upgraded from pre 0.12 to 0.12, we check for "default" partition and fail if found one. Users are expected to rewrite the data in those partitions. Enabling this config will bypass this validation<br />`Config Param: SKIP_DEFAULT_PARTITION_VALIDATION`<br />`Since Version: 0.12.0` [...]
+| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | File format to store all the base file data. org.apache.hudi.common.model.HoodieFileFormat: Hoodie file formats. PARQUET(default): Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data compression and [...]
+| [hoodie.table.services.enabled](#hoodietableservicesenabled) | true | Master control to disable all table services including archive, clean, compact, cluster, etc.<br />`Config Param: TABLE_SERVICES_ENABLED`<br />`Since Version: 0.11.0` [...]
+| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | 1 | Controls the layout of the timeline. Version 0 relied on renames, Version 1 (default) models the timeline as an immutable log relying only on atomic writes for object storage.<br />`Config Param: TIMELINE_LAYOUT_VERSION_NUM`<br />`Since Version: 0.5.1` [...]
+| [hoodie.upsert.shuffle.parallelism](#hoodieupsertshuffleparallelism) | 0 | Parallelism to use for upsert operation on the table. Upserts can shuffle data to perform index lookups, file sizing, bin packing records optimally into file groups. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards H [...]
+| [hoodie.write.buffer.limit.bytes](#hoodiewritebufferlimitbytes) | 4194304 | Size of in-memory buffer used for parallelizing network reads and lake storage writes.<br />`Config Param: WRITE_BUFFER_LIMIT_BYTES_VALUE` [...]
+| [hoodie.write.buffer.record.cache.limit](#hoodiewritebufferrecordcachelimit) | 131072 | Maximum queue size of in-memory buffer for parallelizing network reads and lake storage writes.<br />`Config Param: WRITE_BUFFER_RECORD_CACHE_LIMIT`<br />`Since Version: 0.15.0` [...]
+| [hoodie.write.buffer.record.sampling.rate](#hoodiewritebufferrecordsamplingrate) | 64 | Sampling rate of in-memory buffer used to estimate object size. Higher value lead to lower CPU usage.<br />`Config Param: WRITE_BUFFER_RECORD_SAMPLING_RATE`<br />`Since Version: 0.15.0` [...]
+| [hoodie.write.concurrency.async.conflict.detector.initial_delay_ms](#hoodiewriteconcurrencyasyncconflictdetectorinitial_delay_ms) | 0 | Used for timeline-server-based markers with `AsyncTimelineServerBasedDetectionStrategy`. The time in milliseconds to delay the first execution of async marker-based conflict detection.<br />`Config Param: ASYNC_CONFLICT_DETECTOR_INITIAL_DELAY_MS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.concurrency.async.conflict.detector.period_ms](#hoodiewriteconcurrencyasyncconflictdetectorperiod_ms) | 30000 | Used for timeline-server-based markers with `AsyncTimelineServerBasedDetectionStrategy`. The period in milliseconds between successive executions of async marker-based conflict detection.<br />`Config Param: ASYNC_CONFLICT_DETECTOR_PERIOD_MS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.concurrency.early.conflict.check.commit.conflict](#hoodiewriteconcurrencyearlyconflictcheckcommitconflict) | false | Whether to enable commit conflict checking or not during early conflict detection.<br />`Config Param: EARLY_CONFLICT_DETECTION_CHECK_COMMIT_CONFLICT`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.concurrency.early.conflict.detection.enable](#hoodiewriteconcurrencyearlyconflictdetectionenable) | false | Whether to enable early conflict detection based on markers. It eagerly detects writing conflict before create markers and fails fast if a conflict is detected, to release cluster compute resources as soon as possible.<br />`Config Param: EARLY_CONFLICT_DETECTION_ENABLE`<br />`Since Version: 0. [...]
+| [hoodie.write.concurrency.early.conflict.detection.strategy](#hoodiewriteconcurrencyearlyconflictdetectionstrategy) | | The class name of the early conflict detection strategy to use. This should be a subclass of `org.apache.hudi.common.conflict.detection.EarlyConflictDetectionStrategy`.<br />`Config Param: EARLY_CONFLICT_DETECTION_STRATEGY_CLASS_NAME`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.executor.disruptor.buffer.limit.bytes](#hoodiewriteexecutordisruptorbufferlimitbytes) | 1024 | The size of the Disruptor Executor ring buffer, must be power of 2<br />`Config Param: WRITE_EXECUTOR_DISRUPTOR_BUFFER_LIMIT_BYTES`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.executor.disruptor.wait.strategy](#hoodiewriteexecutordisruptorwaitstrategy) | BLOCKING_WAIT | org.apache.hudi.common.util.queue.DisruptorWaitStrategyType: Strategy employed for making Disruptor Executor wait on a cursor. BLOCKING_WAIT(default): The slowest of the available wait strategies. However, it is the most conservative with the respect to CPU usage and will give the most cons [...]
+| [hoodie.write.executor.type](#hoodiewriteexecutortype) | SIMPLE | org.apache.hudi.common.util.queue.ExecutorType: Types of executor that implements org.apache.hudi.common.util.queue.HoodieExecutor. The executor orchestrates concurrent producers and consumers communicating through a message queue. BOUNDED_IN_MEMORY: Executor which orchestrates concurrent p [...]
+| [hoodie.write.markers.type](#hoodiewritemarkerstype) | TIMELINE_SERVER_BASED | org.apache.hudi.common.table.marker.MarkerType: Marker type indicating how markers are stored in the file system, used for identifying the files written and cleaning up files not committed which should be deleted. DIRECT: Individual marker file corresponding to each data file is directly cr [...]
+| [hoodie.write.num.retries.on.conflict.failures](#hoodiewritenumretriesonconflictfailures) | 0 | Maximum number of times to retry a batch on conflict failure.<br />`Config Param: NUM_RETRIES_ON_CONFLICT_FAILURES`<br />`Since Version: 0.14.0` [...]
+| [hoodie.write.status.storage.level](#hoodiewritestatusstoragelevel) | MEMORY_AND_DISK_SER | Write status objects hold metadata about a write (stats, errors), that is not yet committed to storage. This controls the how that information is cached for inspection by clients. We rarely expect this to be changed.<br />`Config Param: WRITE_STATUS_STORAGE_LEVEL_VALUE` [...]
+| [hoodie.write.tagged.record.storage.level](#hoodiewritetaggedrecordstoragelevel) | MEMORY_AND_DISK_SER | Determine what level of persistence is used to cache write RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: TAGGED_RECORD_STORAGE_LEVEL_VALUE` [...]
+| [hoodie.writestatus.class](#hoodiewritestatusclass) | org.apache.hudi.client.WriteStatus | Subclass of org.apache.hudi.client.WriteStatus to be used to collect information about a write. Can be overridden to collection additional metrics/statistics about the data if needed.<br />`Config Param: WRITE_STATUS_CLASS_NAME` [...]
+---
+
+
+### Commit Callback Configs {#COMMIT_CALLBACK}
+Configurations controlling callback behavior into HTTP endpoints, to push notifications on commits on hudi tables.
+
+
+#### Write commit callback configs {#Write-commit-callback-configs}
+
+
+
+
+[**Advanced Configs**](#Write-commit-callback-configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.write.commit.callback.http.custom.headers](#hoodiewritecommitcallbackhttpcustomheaders) | (N/A) | Http callback custom headers. Format: HeaderName1:HeaderValue1;HeaderName2:HeaderValue2<br />`Config Param: CALLBACK_HTTP_CUSTOM_HEADERS`<br />`Since Version: 0.15.0` |
+| [hoodie.write.commit.callback.http.url](#hoodiewritecommitcallbackhttpurl) | (N/A) | Callback host to be sent along with callback messages<br />`Config Param: CALLBACK_HTTP_URL`<br />`Since Version: 0.6.0` |
+| [hoodie.write.commit.callback.class](#hoodiewritecommitcallbackclass) | org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback | Full path of callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default<br />`Config Param: CALLBACK_CLASS_NAME`<br />`Since Version: 0.6.0` |
+| [hoodie.write.commit.callback.http.api.key](#hoodiewritecommitcallbackhttpapikey) | hudi_write_commit_http_callback | Http callback API key. hudi_write_commit_http_callback by default<br />`Config Param: CALLBACK_HTTP_API_KEY_VALUE`<br />`Since Version: 0.6.0` |
+| [hoodie.write.commit.callback.http.timeout.seconds](#hoodiewritecommitcallbackhttptimeoutseconds) | 30 | Callback timeout in seconds.<br />`Config Param: CALLBACK_HTTP_TIMEOUT_IN_SECONDS`<br />`Since Version: 0.6.0` |
+| [hoodie.write.commit.callback.on](#hoodiewritecommitcallbackon) | false | Turn commit callback on/off. off by default.<br />`Config Param: TURN_CALLBACK_ON`<br />`Since Version: 0.6.0` |
+---
+
+
+#### Write commit Kafka callback configs {#Write-commit-Kafka-callback-configs}
+Controls notifications sent to Kafka, on events happening to a hudi table.
+
+
+
+[**Advanced Configs**](#Write-commit-Kafka-callback-configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.write.commit.callback.kafka.bootstrap.servers](#hoodiewritecommitcallbackkafkabootstrapservers) | (N/A) | Bootstrap servers of kafka cluster, to be used for publishing commit metadata.<br />`Config Param: BOOTSTRAP_SERVERS`<br />`Since Version: 0.7.0` |
+| [hoodie.write.commit.callback.kafka.partition](#hoodiewritecommitcallbackkafkapartition) | (N/A) | It may be desirable to serialize all changes into a single Kafka partition for providing strict ordering. By default, Kafka messages are keyed by table name, which guarantees ordering at the table level, but not globally (or when new partitions are added)<br />`Config Param: PARTITION`<br />`Since Version: 0.7.0` |
+| [hoodie.write.commit.callback.kafka.topic](#hoodiewritecommitcallbackkafkatopic) | (N/A) | Kafka topic name to publish timeline activity into.<br />`Config Param: TOPIC`<br />`Since Version: 0.7.0` |
+| [hoodie.write.commit.callback.kafka.acks](#hoodiewritecommitcallbackkafkaacks) | all | kafka acks level, all by default to ensure strong durability.<br />`Config Param: ACKS`<br />`Since Version: 0.7.0` |
+| [hoodie.write.commit.callback.kafka.retries](#hoodiewritecommitcallbackkafkaretries) | 3 | Times to retry the produce. 3 by default<br />`Config Param: RETRIES`<br />`Since Version: 0.7.0` |
+---
+
+
+#### Write commit pulsar callback configs {#Write-commit-pulsar-callback-configs}
+Controls notifications sent to pulsar, on events happening to a hudi table.
+
+
+
+[**Advanced Configs**](#Write-commit-pulsar-callback-configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.write.commit.callback.pulsar.broker.service.url](#hoodiewritecommitcallbackpulsarbrokerserviceurl) | (N/A) | Server's url of pulsar cluster, to be used for publishing commit metadata.<br />`Config Param: BROKER_SERVICE_URL`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.topic](#hoodiewritecommitcallbackpulsartopic) | (N/A) | pulsar topic name to publish timeline activity into.<br />`Config Param: TOPIC`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.connection-timeout](#hoodiewritecommitcallbackpulsarconnection-timeout) | 10s | Duration of waiting for a connection to a broker to be established.<br />`Config Param: CONNECTION_TIMEOUT`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.keepalive-interval](#hoodiewritecommitcallbackpulsarkeepalive-interval) | 30s | Duration of keeping alive interval for each client broker connection.<br />`Config Param: KEEPALIVE_INTERVAL`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.operation-timeout](#hoodiewritecommitcallbackpulsaroperation-timeout) | 30s | Duration of waiting for completing an operation.<br />`Config Param: OPERATION_TIMEOUT`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.producer.block-if-queue-full](#hoodiewritecommitcallbackpulsarproducerblock-if-queue-full) | true | When the queue is full, the method is blocked instead of an exception is thrown.<br />`Config Param: PRODUCER_BLOCK_QUEUE_FULL`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.producer.pending-queue-size](#hoodiewritecommitcallbackpulsarproducerpending-queue-size) | 1000 | The maximum size of a queue holding pending messages.<br />`Config Param: PRODUCER_PENDING_QUEUE_SIZE`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.producer.pending-total-size](#hoodiewritecommitcallbackpulsarproducerpending-total-size) | 50000 | The maximum number of pending messages across partitions.<br />`Config Param: PRODUCER_PENDING_SIZE`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.producer.route-mode](#hoodiewritecommitcallbackpulsarproducerroute-mode) | RoundRobinPartition | Message routing logic for producers on partitioned topics.<br />`Config Param: PRODUCER_ROUTE_MODE`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.producer.send-timeout](#hoodiewritecommitcallbackpulsarproducersend-timeout) | 30s | The timeout in each sending to pulsar.<br />`Config Param: PRODUCER_SEND_TIMEOUT`<br />`Since Version: 0.11.0` |
+| [hoodie.write.commit.callback.pulsar.request-timeout](#hoodiewritecommitcallbackpulsarrequest-timeout) | 60s | Duration of waiting for completing a request.<br />`Config Param: REQUEST_TIMEOUT`<br />`Since Version: 0.11.0` |
+---
+
+
+### Lock Configs {#LOCK}
+Configurations that control locking mechanisms required for concurrency control between writers to a Hudi table. Concurrency between Hudi's own table services are auto managed internally.
+
+
+#### Common Lock Configurations {#Common-Lock-Configurations}
+
+
+
+
+[**Basic Configs**](#Common-Lock-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.write.lock.heartbeat_interval_ms](#hoodiewritelockheartbeat_interval_ms) | 60000 | Heartbeat interval in ms, to send a heartbeat to indicate that hive client holding locks.<br />`Config Param: LOCK_HEARTBEAT_INTERVAL_MS`<br />`Since Version: 0.15.0` |
+
+[**Advanced Configs**](#Common-Lock-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.write.lock.filesystem.path](#hoodiewritelockfilesystempath) | (N/A) | For DFS based lock providers, path to store the locks under. use Table's meta path as default<br />`Config Param: FILESYSTEM_LOCK_PATH`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.hivemetastore.database](#hoodiewritelockhivemetastoredatabase) | (N/A) | For Hive based lock provider, the Hive database to acquire lock against<br />`Config Param: HIVE_DATABASE_NAME`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.hivemetastore.table](#hoodiewritelockhivemetastoretable) | (N/A) | For Hive based lock provider, the Hive table to acquire lock against<br />`Config Param: HIVE_TABLE_NAME`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.hivemetastore.uris](#hoodiewritelockhivemetastoreuris) | (N/A) | For Hive based lock provider, the Hive metastore URI to acquire locks against.<br />`Config Param: HIVE_METASTORE_URI`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.zookeeper.base_path](#hoodiewritelockzookeeperbase_path) | (N/A) | The base path on Zookeeper under which to create lock related ZNodes. This should be same for all concurrent writers to the same table<br />`Config Param: ZK_BASE_PATH`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.zookeeper.port](#hoodiewritelockzookeeperport) | (N/A) | Zookeeper port to connect to.<br />`Config Param: ZK_PORT`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.zookeeper.url](#hoodiewritelockzookeeperurl) | (N/A) | Zookeeper URL to connect to.<br />`Config Param: ZK_CONNECT_URL`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.client.num_retries](#hoodiewritelockclientnum_retries) | 50 | Maximum number of times to retry to acquire lock additionally from the lock manager.<br />`Config Param: LOCK_ACQUIRE_CLIENT_NUM_RETRIES`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.client.wait_time_ms_between_retry](#hoodiewritelockclientwait_time_ms_between_retry) | 5000 | Amount of time to wait between retries on the lock provider by the lock manager<br />`Config Param: LOCK_ACQUIRE_CLIENT_RETRY_WAIT_TIME_IN_MILLIS`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.conflict.resolution.strategy](#hoodiewritelockconflictresolutionstrategy) | org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy | Lock provider class name, this should be subclass of org.apache.hudi.client.transaction.ConflictResolutionStrategy<br />`Config Param: WRITE_CONFLICT_RESOLUTION_STRATEGY_CLASS_NAME`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.filesystem.expire](#hoodiewritelockfilesystemexpire) | 0 | For DFS based lock providers, expire time in minutes, must be a non-negative number, default means no expire<br />`Config Param: FILESYSTEM_LOCK_EXPIRE`<br />`Since Version: 0.12.0` |
+| [hoodie.write.lock.max_wait_time_ms_between_retry](#hoodiewritelockmax_wait_time_ms_between_retry) | 16000 | Maximum amount of time to wait between retries by lock provider client. This bounds the maximum delay from the exponential backoff. Currently used by ZK based lock provider only.<br />`Config Param: LOCK_ACQUIRE_RETRY_MAX_WAIT_TIME_IN_MILLIS`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.num_retries](#hoodiewritelocknum_retries) | 15 | Maximum number of times to retry lock acquire, at each lock provider<br />`Config Param: LOCK_ACQUIRE_NUM_RETRIES`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.provider](#hoodiewritelockprovider) | org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider | Lock provider class name, user can provide their own implementation of LockProvider which should be subclass of org.apache.hudi.common.lock.LockProvider<br />`Config Param: LOCK_PROVIDER_CLASS_NAME`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.wait_time_ms](#hoodiewritelockwait_time_ms) | 60000 | Timeout in ms, to wait on an individual lock acquire() call, at the lock provider.<br />`Config Param: LOCK_ACQUIRE_WAIT_TIMEOUT_MS`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.wait_time_ms_between_retry](#hoodiewritelockwait_time_ms_between_retry) | 1000 | Initial amount of time to wait between retries to acquire locks, subsequent retries will exponentially backoff.<br />`Config Param: LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.zookeeper.connection_timeout_ms](#hoodiewritelockzookeeperconnection_timeout_ms) | 15000 | Timeout in ms, to wait for establishing connection with Zookeeper.<br />`Config Param: ZK_CONNECTION_TIMEOUT_MS`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.zookeeper.lock_key](#hoodiewritelockzookeeperlock_key) | | Key name under base_path at which to create a ZNode and acquire lock. Final path on zk will look like base_path/lock_key. If this parameter is not set, we would set it as the table name<br />`Config Param: ZK_LOCK_KEY`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.zookeeper.session_timeout_ms](#hoodiewritelockzookeepersession_timeout_ms) | 60000 | Timeout in ms, to wait after losing connection to ZooKeeper, before the session is expired<br />`Config Param: ZK_SESSION_TIMEOUT_MS`<br />`Since Version: 0.8.0` |
+---
+
+
+#### DynamoDB based Locks Configurations {#DynamoDB-based-Locks-Configurations}
+Configs that control DynamoDB based locking mechanisms required for concurrency control between writers to a Hudi table. Concurrency between Hudi's own table services are auto managed internally.
+
+
+
+[**Advanced Configs**](#DynamoDB-based-Locks-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.write.lock.dynamodb.endpoint_url](#hoodiewritelockdynamodbendpoint_url) | (N/A) | For DynamoDB based lock provider, the url endpoint used for Amazon DynamoDB service. Useful for development with a local dynamodb instance.<br />`Config Param: DYNAMODB_ENDPOINT_URL`<br />`Since Version: 0.10.1` |
+| [hoodie.write.lock.dynamodb.billing_mode](#hoodiewritelockdynamodbbilling_mode) | PAY_PER_REQUEST | For DynamoDB based lock provider, by default it is `PAY_PER_REQUEST` mode. Alternative is `PROVISIONED`.<br />`Config Param: DYNAMODB_LOCK_BILLING_MODE`<br />`Since Version: 0.10.0` |
+| [hoodie.write.lock.dynamodb.partition_key](#hoodiewritelockdynamodbpartition_key) | | For DynamoDB based lock provider, the partition key for the DynamoDB lock table. Each Hudi dataset should has it's unique key so concurrent writers could refer to the same partition key. By default we use the Hudi table name specified to be the partition key<br />`Config Param: DYNAMODB_LOCK_PARTITION_KEY`<br />`Since Version: 0.10.0` |
+| [hoodie.write.lock.dynamodb.read_capacity](#hoodiewritelockdynamodbread_capacity) | 20 | For DynamoDB based lock provider, read capacity units when using PROVISIONED billing mode<br />`Config Param: DYNAMODB_LOCK_READ_CAPACITY`<br />`Since Version: 0.10.0` |
+| [hoodie.write.lock.dynamodb.region](#hoodiewritelockdynamodbregion) | us-east-1 | For DynamoDB based lock provider, the region used in endpoint for Amazon DynamoDB service. Would try to first get it from AWS_REGION environment variable. If not find, by default use us-east-1<br />`Config Param: DYNAMODB_LOCK_REGION`<br />`Since Version: 0.10.0` |
+| [hoodie.write.lock.dynamodb.table](#hoodiewritelockdynamodbtable) | hudi_locks | For DynamoDB based lock provider, the name of the DynamoDB table acting as lock table<br />`Config Param: DYNAMODB_LOCK_TABLE_NAME`<br />`Since Version: 0.10.0` |
+| [hoodie.write.lock.dynamodb.table_creation_timeout](#hoodiewritelockdynamodbtable_creation_timeout) | 120000 | For DynamoDB based lock provider, the maximum number of milliseconds to wait for creating DynamoDB table<br />`Config Param: DYNAMODB_LOCK_TABLE_CREATION_TIMEOUT`<br />`Since Version: 0.10.0` |
+| [hoodie.write.lock.dynamodb.write_capacity](#hoodiewritelockdynamodbwrite_capacity) | 10 | For DynamoDB based lock provider, write capacity units when using PROVISIONED billing mode<br />`Config Param: DYNAMODB_LOCK_WRITE_CAPACITY`<br />`Since Version: 0.10.0` |
+| [hoodie.write.lock.wait_time_ms](#hoodiewritelockwait_time_ms) | 60000 | Lock Acquire Wait Timeout in milliseconds<br />`Config Param: LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY`<br />`Since Version: 0.10.0` |
+---
+
+
+### Key Generator Configs {#KEY_GENERATOR}
+Hudi maintains keys (record key + partition path) for uniquely identifying a particular record. These configs allow developers to setup the Key generator class that extracts these out of incoming records.
+
+
+#### Key Generator Options {#Key-Generator-Options}
+
+
+
+
+[**Basic Configs**](#Key-Generator-Options-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD_NAME` |
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` |
+
+[**Advanced Configs**](#Key-Generator-Options-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled](#hoodiedatasourcewritekeygeneratorconsistentlogicaltimestampenabled) | false | When set to true, consistent value will be generated for a logical timestamp type column, like timestamp-millis and timestamp-micros, irrespective of whether row-writer is enabled. Disabled by default so as not to break the pipeline that deploy either fully row-writer path or non row-writer path. For example, if it is kept disabled [...]
+| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Should we url encode the partition path value, before creating the folder structure.<br />`Config Param: URL_ENCODE_PARTITIONING` [...]
+---
+
+
+#### Timestamp-based key generator configs {#Timestamp-based-key-generator-configs}
+Configs used for TimestampBasedKeyGenerator which relies on timestamps for the partition field. The field values are interpreted as timestamps and not just converted to string while generating partition path value for records. Record key is same as before where it is chosen by field name.
+
+
+
+[**Advanced Configs**](#Timestamp-based-key-generator-configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.keygen.timebased.timestamp.type](#hoodiekeygentimebasedtimestamptype) | (N/A) | Timestamp type of the field, which should be one of the timestamp types supported: `UNIX_TIMESTAMP`, `DATE_STRING`, `MIXED`, `EPOCHMILLISECONDS`, `SCALAR`.<br />`Config Param: TIMESTAMP_TYPE_FIELD` |
+| [hoodie.keygen.datetime.parser.class](#hoodiekeygendatetimeparserclass) | org.apache.hudi.keygen.parser.HoodieDateTimeParser | Date time parser class name.<br />`Config Param: DATE_TIME_PARSER` |
+| [hoodie.keygen.timebased.input.dateformat](#hoodiekeygentimebasedinputdateformat) | | Input date format such as `yyyy-MM-dd'T'HH:mm:ss.SSSZ`.<br />`Config Param: TIMESTAMP_INPUT_DATE_FORMAT` |
+| [hoodie.keygen.timebased.input.dateformat.list.delimiter.regex](#hoodiekeygentimebasedinputdateformatlistdelimiterregex) | , | The delimiter for allowed input date format list, usually `,`.<br />`Config Param: TIMESTAMP_INPUT_DATE_FORMAT_LIST_DELIMITER_REGEX` |
+| [hoodie.keygen.timebased.input.timezone](#hoodiekeygentimebasedinputtimezone) | UTC | Timezone of the input timestamp, such as `UTC`.<br />`Config Param: TIMESTAMP_INPUT_TIMEZONE_FORMAT` |
+| [hoodie.keygen.timebased.output.dateformat](#hoodiekeygentimebasedoutputdateformat) | | Output date format such as `yyyy-MM-dd'T'HH:mm:ss.SSSZ`.<br />`Config Param: TIMESTAMP_OUTPUT_DATE_FORMAT` |
+| [hoodie.keygen.timebased.output.timezone](#hoodiekeygentimebasedoutputtimezone) | UTC | Timezone of the output timestamp, such as `UTC`.<br />`Config Param: TIMESTAMP_OUTPUT_TIMEZONE_FORMAT` |
+| [hoodie.keygen.timebased.timestamp.scalar.time.unit](#hoodiekeygentimebasedtimestampscalartimeunit) | SECONDS | When timestamp type `SCALAR` is used, this specifies the time unit, with allowed unit specified by `TimeUnit` enums (`NANOSECONDS`, `MICROSECONDS`, `MILLISECONDS`, `SECONDS`, `MINUTES`, `HOURS`, `DAYS`).<br />`Config Param: INPUT_TIME_UNIT` |
+| [hoodie.keygen.timebased.timezone](#hoodiekeygentimebasedtimezone) | UTC | Timezone of both input and output timestamp if they are the same, such as `UTC`. Please use `hoodie.keygen.timebased.input.timezone` and `hoodie.keygen.timebased.output.timezone` instead if the input and output timezones are different.<br />`Config Param: TIMESTAMP_TIMEZONE_FORMAT` |
+---
+
+
+### Index Configs {#INDEX}
+Configurations that control indexing behavior, which tags incoming records as either inserts or updates to older records.
+
+
+#### Common Index Configs {#Common-Index-Configs}
+
+
+
+
+[**Basic Configs**](#Common-Index-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.index.type](#hoodieindextype) | (N/A) | org.apache.hudi.index.HoodieIndex$IndexType: Determines how input records are indexed, i.e., looked up based on the key for the location in the existing table. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java engines. HBASE: uses an external managed Apache HBase table to store record key to location mapping. HBase index is a global index, enforcing key uniqueness across all partitions in the table. INMEMORY: Use [...]
+
+[**Advanced Configs**](#Common-Index-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ----------------------------------------------------------------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.bucket.index.hash.field](#hoodiebucketindexhashfield) | (N/A) | Index key. It is used to index the record and find its file group. If not set, use record key field as default<br />`Config Param: BUCKET_INDEX_HASH_FIELD` [...]
+| [hoodie.bucket.index.max.num.buckets](#hoodiebucketindexmaxnumbuckets) | (N/A) | Only applies if bucket index engine is consistent hashing. Determine the upper bound of the number of buckets in the hudi table. Bucket resizing cannot be done higher than this max limit.<br />`Config Param: BUCKET_INDEX_MAX_NUM_BUCKETS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.bucket.index.min.num.buckets](#hoodiebucketindexminnumbuckets) | (N/A) | Only applies if bucket index engine is consistent hashing. Determine the lower bound of the number of buckets in the hudi table. Bucket resizing cannot be done lower than this min limit.<br />`Config Param: BUCKET_INDEX_MIN_NUM_BUCKETS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.bloom.index.bucketized.checking](#hoodiebloomindexbucketizedchecking) | true | Only applies if index type is BLOOM. When true, bucketized bloom filtering is enabled. This reduces skew seen in sort based bloom index lookup<br />`Config Param: BLOOM_INDEX_BUCKETIZED_CHECKING` [...]
+| [hoodie.bloom.index.input.storage.level](#hoodiebloomindexinputstoragelevel) | MEMORY_AND_DISK_SER | Only applies when #bloomIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: BLOOM_INDEX_INPUT_STORAGE_LEVEL_VALUE` [...]
+| [hoodie.bloom.index.keys.per.bucket](#hoodiebloomindexkeysperbucket) | 10000000 | Only applies if bloomIndexBucketizedChecking is enabled and index type is bloom. This configuration controls the “bucket” size which tracks the number of record-key checks made against a single file and is the unit of work allocated to each partition performing bloom filter lookup. A higher value would amortize the fixed cost of reading a bloom filter to memory.<br />`Config Par [...]
+| [hoodie.bloom.index.parallelism](#hoodiebloomindexparallelism) | 0 | Only applies if index type is BLOOM. This is the amount of parallelism for index lookup, which involves a shuffle. By default, this is auto computed based on input workload characteristics. If the parallelism is explicitly configured by the user, the user-configured value is used in defining the actual parallelism. If the indexing stage is slow due to the limited parallelism, yo [...]
+| [hoodie.bloom.index.prune.by.ranges](#hoodiebloomindexprunebyranges) | true | Only applies if index type is BLOOM. When true, range information from files to leveraged speed up index lookups. Particularly helpful, if the key has a monotonously increasing prefix, such as timestamp. If the record key is completely random, it is better to turn this off, since range pruning will only add extra overhead to the index lookup.<br />`Config Param: BLOOM_INDEX_PRU [...]
+| [hoodie.bloom.index.update.partition.path](#hoodiebloomindexupdatepartitionpath) | true | Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition<br />`Config Param: BLOOM_INDEX_UPDATE_PARTI [...]
+| [hoodie.bloom.index.use.caching](#hoodiebloomindexusecaching) | true | Only applies if index type is BLOOM.When true, the input RDD will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br />`Config Param: BLOOM_INDEX_USE_CACHING` [...]
+| [hoodie.bloom.index.use.metadata](#hoodiebloomindexusemetadata) | false | Only applies if index type is BLOOM.When true, the index lookup uses bloom filters and column stats from metadata table when available to speed up the process.<br />`Config Param: BLOOM_INDEX_USE_METADATA`<br />`Since Version: 0.11.0` [...]
+| [hoodie.bloom.index.use.treebased.filter](#hoodiebloomindexusetreebasedfilter) | true | Only applies if index type is BLOOM. When true, interval tree based file pruning optimization is enabled. This mode speeds-up file-pruning based on key ranges when compared with the brute-force mode<br />`Config Param: BLOOM_INDEX_TREE_BASED_FILTER` [...]
+| [hoodie.bucket.index.merge.threshold](#hoodiebucketindexmergethreshold) | 0.2 | Control if buckets should be merged when using consistent hashing bucket indexSpecifically, if a file slice size is smaller than `hoodie.xxxx.max.file.size` * threshold, then it will be consideredas a merge candidate.<br />`Config Param: BUCKET_MERGE_THRESHOLD`<br />`Since Version: 0.13.0` [...]
+| [hoodie.bucket.index.num.buckets](#hoodiebucketindexnumbuckets) | 256 | Only applies if index type is BUCKET. Determine the number of buckets in the hudi table, and each partition is divided to N buckets.<br />`Config Param: BUCKET_INDEX_NUM_BUCKETS` [...]
+| [hoodie.bucket.index.split.threshold](#hoodiebucketindexsplitthreshold) | 2.0 | Control if the bucket should be split when using consistent hashing bucket index.Specifically, if a file slice size reaches `hoodie.xxxx.max.file.size` * threshold, then split will be carried out.<br />`Config Param: BUCKET_SPLIT_THRESHOLD`<br />`Since Version: 0.13.0` [...]
+| [hoodie.global.index.reconcile.parallelism](#hoodieglobalindexreconcileparallelism) | 60 | Only applies if index type is GLOBAL_BLOOM or GLOBAL_SIMPLE. This controls the parallelism for deduplication during indexing where more than 1 record could be tagged due to partition update.<br />`Config Param: GLOBAL_INDEX_RECONCILE_PARALLELISM` [...]
+| [hoodie.global.simple.index.parallelism](#hoodieglobalsimpleindexparallelism) | 100 | Only applies if index type is GLOBAL_SIMPLE. This limits the parallelism of fetching records from the base files of all table partitions. The index picks the configured parallelism if the number of base files is larger than this configured value; otherwise, the number of base files is used as the parallelism. If the indexing stage is slow due to the limited parallelism, you can [...]
+| [hoodie.index.bucket.engine](#hoodieindexbucketengine) | SIMPLE | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. SIMPLE(default): Uses a fixed number of buckets for file groups which cannot shrink or expand. This works for both COW and MOR tables. CONSISTENT_HASHING: Supports dynamic number of buckets with bucket resizing to proper [...]
+| [hoodie.index.class](#hoodieindexclass) | | Full path of user-defined index class and must be a subclass of HoodieIndex class. It will take precedence over the hoodie.index.type configuration if specified<br />`Config Param: INDEX_CLASS_NAME` [...]
+| [hoodie.record.index.input.storage.level](#hoodierecordindexinputstoragelevel) | MEMORY_AND_DISK_SER | Only applies when #recordIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: RECORD_INDEX_INPUT_STORAGE_LEVEL_VALUE`<br />`Since Version: 0.14.0` [...]
+| [hoodie.record.index.update.partition.path](#hoodierecordindexupdatepartitionpath) | false | Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: true , isAdvanced: true , description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to fals [...]
+| [hoodie.record.index.use.caching](#hoodierecordindexusecaching) | true | Only applies if index type is RECORD_INDEX.When true, the input RDD will be cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br />`Config Param: RECORD_INDEX_USE_CACHING`<br />`Since Version: 0.14.0` [...]
+| [hoodie.simple.index.input.storage.level](#hoodiesimpleindexinputstoragelevel) | MEMORY_AND_DISK_SER | Only applies when #simpleIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: SIMPLE_INDEX_INPUT_STORAGE_LEVEL_VALUE` [...]
+| [hoodie.simple.index.parallelism](#hoodiesimpleindexparallelism) | 0 | Only applies if index type is SIMPLE. This limits the parallelism of fetching records from the base files of affected partitions. By default, this is auto computed based on input workload characteristics. If the parallelism is explicitly configured by the user, the user-configured value is used in defining the actual parallelism. If the indexing stage is slow due to the limited [...]
+| [hoodie.simple.index.update.partition.path](#hoodiesimpleindexupdatepartitionpath) | true | Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: true , isAdvanced: true , description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to fals [...]
+| [hoodie.simple.index.use.caching](#hoodiesimpleindexusecaching) | true | Only applies if index type is SIMPLE. When true, the incoming writes will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br />`Config Param: SIMPLE_INDEX_USE_CACHING` [...]
+---
+
+
+#### HBase Index Configs {#HBase-Index-Configs}
+Configurations that control indexing behavior (when HBase based indexing is enabled), which tags incoming records as either inserts or updates to older records.
+
+
+
+[**Advanced Configs**](#HBase-Index-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.index.hbase.kerberos.user.keytab](#hoodieindexhbasekerberosuserkeytab) | (N/A) | File name of the kerberos keytab file for connecting to the hbase cluster.<br />`Config Param: KERBEROS_USER_KEYTAB` [...]
+| [hoodie.index.hbase.kerberos.user.principal](#hoodieindexhbasekerberosuserprincipal) | (N/A) | The kerberos principal name for connecting to the hbase cluster.<br />`Config Param: KERBEROS_USER_PRINCIPAL` [...]
+| [hoodie.index.hbase.master.kerberos.principal](#hoodieindexhbasemasterkerberosprincipal) | (N/A) | The value of hbase.master.kerberos.principal in hbase cluster.<br />`Config Param: MASTER_PRINCIPAL` [...]
+| [hoodie.index.hbase.max.qps.fraction](#hoodieindexhbasemaxqpsfraction) | (N/A) | Maximum for HBASE_QPS_FRACTION_PROP to stabilize skewed write workloads<br />`Config Param: MAX_QPS_FRACTION` [...]
+| [hoodie.index.hbase.min.qps.fraction](#hoodieindexhbaseminqpsfraction) | (N/A) | Minimum for HBASE_QPS_FRACTION_PROP to stabilize skewed write workloads<br />`Config Param: MIN_QPS_FRACTION` [...]
+| [hoodie.index.hbase.regionserver.kerberos.principal](#hoodieindexhbaseregionserverkerberosprincipal) | (N/A) | The value of hbase.regionserver.kerberos.principal in hbase cluster.<br />`Config Param: REGIONSERVER_PRINCIPAL` [...]
+| [hoodie.index.hbase.sleep.ms.for.get.batch](#hoodieindexhbasesleepmsforgetbatch) | (N/A) | <br />`Config Param: SLEEP_MS_FOR_GET_BATCH` [...]
+| [hoodie.index.hbase.sleep.ms.for.put.batch](#hoodieindexhbasesleepmsforputbatch) | (N/A) | <br />`Config Param: SLEEP_MS_FOR_PUT_BATCH` [...]
+| [hoodie.index.hbase.table](#hoodieindexhbasetable) | (N/A) | Only applies if index type is HBASE. HBase Table name to use as the index. Hudi stores the row_key and [partition_path, fileID, commitTime] mapping in the table<br />`Config Param: TABLENAME` [...]
+| [hoodie.index.hbase.zknode.path](#hoodieindexhbasezknodepath) | (N/A) | Only applies if index type is HBASE. This is the root znode that will contain all the znodes created/used by HBase<br />`Config Param: ZK_NODE_PATH` [...]
+| [hoodie.index.hbase.zkport](#hoodieindexhbasezkport) | (N/A) | Only applies if index type is HBASE. HBase ZK Quorum port to connect to<br />`Config Param: ZKPORT` [...]
+| [hoodie.index.hbase.zkquorum](#hoodieindexhbasezkquorum) | (N/A) | Only applies if index type is HBASE. HBase ZK Quorum url to connect to<br />`Config Param: ZKQUORUM` [...]
+| [hoodie.hbase.index.update.partition.path](#hoodiehbaseindexupdatepartitionpath) | false | Only applies if index type is HBASE. When an already existing record is upserted to a new partition compared to whats in storage, this config when set, will delete old record in old partition and will insert it as new record in new partition.<br />`Config Param: UPDATE_PARTITION_PATH_ENABLE` [...]
+| [hoodie.index.hbase.bucket.number](#hoodieindexhbasebucketnumber) | 8 | Only applicable when using RebalancedSparkHoodieHBaseIndex, same as hbase regions count can get the best performance<br />`Config Param: BUCKET_NUMBER` [...]
+| [hoodie.index.hbase.desired_puts_time_in_secs](#hoodieindexhbasedesired_puts_time_in_secs) | 600 | <br />`Config Param: DESIRED_PUTS_TIME_IN_SECONDS` [...]
+| [hoodie.index.hbase.dynamic_qps](#hoodieindexhbasedynamic_qps) | false | Property to decide if HBASE_QPS_FRACTION_PROP is dynamically calculated based on write volume.<br />`Config Param: COMPUTE_QPS_DYNAMICALLY` [...]
+| [hoodie.index.hbase.get.batch.size](#hoodieindexhbasegetbatchsize) | 100 | Controls the batch size for performing gets against HBase. Batching improves throughput, by saving round trips.<br />`Config Param: GET_BATCH_SIZE` [...]
+| [hoodie.index.hbase.max.qps.per.region.server](#hoodieindexhbasemaxqpsperregionserver) | 1000 | Property to set maximum QPS allowed per Region Server. This should be same across various jobs. This is intended to limit the aggregate QPS generated across various jobs to an Hbase Region Server. It is recommended to set this value based on global indexing throughput needs and most importantly, how much the HBase insta [...]
+| [hoodie.index.hbase.put.batch.size](#hoodieindexhbaseputbatchsize) | 100 | Controls the batch size for performing puts against HBase. Batching improves throughput, by saving round trips.<br />`Config Param: PUT_BATCH_SIZE` [...]
+| [hoodie.index.hbase.put.batch.size.autocompute](#hoodieindexhbaseputbatchsizeautocompute) | false | Property to set to enable auto computation of put batch size<br />`Config Param: PUT_BATCH_SIZE_AUTO_COMPUTE` [...]
+| [hoodie.index.hbase.qps.allocator.class](#hoodieindexhbaseqpsallocatorclass) | org.apache.hudi.index.hbase.DefaultHBaseQPSResourceAllocator | Property to set which implementation of HBase QPS resource allocator to be used, whichcontrols the batching rate dynamically.<br />`Config Param: QPS_ALLOCATOR_CLASS_NAME` [...]
+| [hoodie.index.hbase.qps.fraction](#hoodieindexhbaseqpsfraction) | 0.5 | Property to set the fraction of the global share of QPS that should be allocated to this job. Let's say there are 3 jobs which have input size in terms of number of rows required for HbaseIndexing as x, 2x, 3x respectively. Then this fraction for the jobs would be (0.17) 1/6, 0.33 (2/6) and 0.5 (3/6) respectively. Default [...]
+| [hoodie.index.hbase.rollback.sync](#hoodieindexhbaserollbacksync) | false | When set to true, the rollback method will delete the last failed task index. The default value is false. Because deleting the index will add extra load on the Hbase cluster for each rollback<br />`Config Param: ROLLBACK_SYNC_ENABLE` [...]
+| [hoodie.index.hbase.security.authentication](#hoodieindexhbasesecurityauthentication) | simple | Property to decide if the hbase cluster secure authentication is enabled or not. Possible values are 'simple' (no authentication), and 'kerberos'.<br />`Config Param: SECURITY_AUTHENTICATION` [...]
+| [hoodie.index.hbase.zk.connection_timeout_ms](#hoodieindexhbasezkconnection_timeout_ms) | 15000 | Timeout to use for establishing connection with zookeeper, from HBase client.<br />`Config Param: ZK_CONNECTION_TIMEOUT_MS` [...]
+| [hoodie.index.hbase.zk.session_timeout_ms](#hoodieindexhbasezksession_timeout_ms) | 60000 | Session timeout value to use for Zookeeper failure detection, for the HBase client.Lower this value, if you want to fail faster.<br />`Config Param: ZK_SESSION_TIMEOUT_MS` [...]
+| [hoodie.index.hbase.zkpath.qps_root](#hoodieindexhbasezkpathqps_root) | /QPS_ROOT | chroot in zookeeper, to use for all qps allocation co-ordination.<br />`Config Param: ZKPATH_QPS_ROOT` [...]
+---
+
+## Metastore and Catalog Sync Configs {#META_SYNC}
+Configurations used by the Hudi to sync metadata to external metastores and catalogs.
+
+
+### Common Metadata Sync Configs {#Common-Metadata-Sync-Configs}
+
+
+
+
+[**Basic Configs**](#Common-Metadata-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+
+[**Advanced Configs**](#Common-Metadata-Sync-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` |
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` |
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` |
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` |
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` |
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` |
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` |
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` |
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` |
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` |
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` |
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` |
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` |
+---
+
+
+### Glue catalog sync based client Configurations {#Glue-catalog-sync-based-client-Configurations}
+Configs that control Glue catalog sync based client.
+
+
+
+[**Basic Configs**](#Glue-catalog-sync-based-client-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.glue.partition_index_fields](#hoodiedatasourcemetasyncgluepartition_index_fields) | | Specify the partitions fields to index on aws glue. Separate the fields by semicolon. By default, when the feature is enabled, all the partition will be indexed. You can create up to three indexes, separate them by comma. Eg: col1;col2;col3,col2,col3<br />`Config Param: META_SYNC_PARTITION_INDEX_FIELDS`<br />`Since Version: 0.15.0` |
+| [hoodie.datasource.meta.sync.glue.partition_index_fields.enable](#hoodiedatasourcemetasyncgluepartition_index_fieldsenable) | false | Enable aws glue partition index feature, to speedup partition based query pattern<br />`Config Param: META_SYNC_PARTITION_INDEX_FIELDS_ENABLE`<br />`Since Version: 0.15.0` |
+
+[**Advanced Configs**](#Glue-catalog-sync-based-client-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.glue.all_partitions_read_parallelism](#hoodiedatasourcemetasyncglueall_partitions_read_parallelism) | 1 | Parallelism for listing all partitions(first time sync). Should be in interval [1, 10].<br />`Config Param: ALL_PARTITIONS_READ_PARALLELISM`<br />`Since Version: 0.15.0` |
+| [hoodie.datasource.meta.sync.glue.changed_partitions_read_parallelism](#hoodiedatasourcemetasyncgluechanged_partitions_read_parallelism) | 1 | Parallelism for listing changed partitions(second and subsequent syncs).<br />`Config Param: CHANGED_PARTITIONS_READ_PARALLELISM`<br />`Since Version: 0.15.0` |
+| [hoodie.datasource.meta.sync.glue.metadata_file_listing](#hoodiedatasourcemetasyncgluemetadata_file_listing) | false | Makes athena use the metadata table to list partitions and files. Currently it won't benefit from other features such stats indexes<br />`Config Param: GLUE_METADATA_FILE_LISTING`<br />`Since Version: 0.14.0` |
+| [hoodie.datasource.meta.sync.glue.partition_change_parallelism](#hoodiedatasourcemetasyncgluepartition_change_parallelism) | 1 | Parallelism for change operations - such as create/update/delete.<br />`Config Param: PARTITION_CHANGE_PARALLELISM`<br />`Since Version: 0.15.0` |
+| [hoodie.datasource.meta.sync.glue.skip_table_archive](#hoodiedatasourcemetasyncglueskip_table_archive) | true | Glue catalog sync based client will skip archiving the table version if this config is set to true<br />`Config Param: GLUE_SKIP_TABLE_ARCHIVE`<br />`Since Version: 0.14.0` |
+---
+
+
+### BigQuery Sync Configs {#BigQuery-Sync-Configs}
+Configurations used by the Hudi to sync metadata to Google BigQuery.
+
+
+
+[**Basic Configs**](#BigQuery-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+
+[**Advanced Configs**](#BigQuery-Sync-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
+| [hoodie.gcp.bigquery.sync.big_lake_connection_id](#hoodiegcpbigquerysyncbig_lake_connection_id) | (N/A) | The Big Lake connection ID to use<br />`Config Param: BIGQUERY_SYNC_BIG_LAKE_CONNECTION_ID`<br />`Since Version: 0.14.1` [...]
+| [hoodie.gcp.bigquery.sync.dataset_location](#hoodiegcpbigquerysyncdataset_location) | (N/A) | Location of the target dataset in BigQuery<br />`Config Param: BIGQUERY_SYNC_DATASET_LOCATION` [...]
+| [hoodie.gcp.bigquery.sync.project_id](#hoodiegcpbigquerysyncproject_id) | (N/A) | Name of the target project in BigQuery<br />`Config Param: BIGQUERY_SYNC_PROJECT_ID` [...]
+| [hoodie.gcp.bigquery.sync.source_uri](#hoodiegcpbigquerysyncsource_uri) | (N/A) | Name of the source uri gcs path of the table<br />`Config Param: BIGQUERY_SYNC_SOURCE_URI` [...]
+| [hoodie.gcp.bigquery.sync.source_uri_prefix](#hoodiegcpbigquerysyncsource_uri_prefix) | (N/A) | Name of the source uri gcs path prefix of the table<br />`Config Param: BIGQUERY_SYNC_SOURCE_URI_PREFIX` [...]
+| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` [...]
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` [...]
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` [...]
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` [...]
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` [...]
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` [...]
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` [...]
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` [...]
+| [hoodie.gcp.bigquery.sync.assume_date_partitioning](#hoodiegcpbigquerysyncassume_date_partitioning) | false | Assume standard yyyy/mm/dd partitioning, this exists to support backward compatibility. If you use hoodie 0.3.x, do not set this parameter<br />`Config Param: BIGQUERY_SYNC_ASSUME_DATE_PARTITIONING` [...]
+| [hoodie.gcp.bigquery.sync.base_path](#hoodiegcpbigquerysyncbase_path) | | Base path of the hoodie table to sync<br />`Config Param: BIGQUERY_SYNC_SYNC_BASE_PATH` [...]
+| [hoodie.gcp.bigquery.sync.dataset_name](#hoodiegcpbigquerysyncdataset_name) | | Name of the target dataset in BigQuery<br />`Config Param: BIGQUERY_SYNC_DATASET_NAME` [...]
+| [hoodie.gcp.bigquery.sync.partition_fields](#hoodiegcpbigquerysyncpartition_fields) | | Comma-delimited partition fields. Default to non-partitioned.<br />`Config Param: BIGQUERY_SYNC_PARTITION_FIELDS` [...]
+| [hoodie.gcp.bigquery.sync.require_partition_filter](#hoodiegcpbigquerysyncrequire_partition_filter) | false | If true, configure table to require a partition filter to be specified when querying the table<br />`Config Param: BIGQUERY_SYNC_REQUIRE_PARTITION_FILTER`<br />`Since Version: 0.14.1` [...]
+| [hoodie.gcp.bigquery.sync.table_name](#hoodiegcpbigquerysynctable_name) | | Name of the target table in BigQuery<br />`Config Param: BIGQUERY_SYNC_TABLE_NAME` [...]
+| [hoodie.gcp.bigquery.sync.use_bq_manifest_file](#hoodiegcpbigquerysyncuse_bq_manifest_file) | false | If true, generate a manifest file with data file absolute paths and use BigQuery manifest file support to directly create one external table over the Hudi table. If false (default), generate a manifest file with data file names and create two external tables and one view in BigQuery. Query the view for the same results as [...]
+| [hoodie.gcp.bigquery.sync.use_file_listing_from_metadata](#hoodiegcpbigquerysyncuse_file_listing_from_metadata) | false | Fetch file listing from Hudi's metadata<br />`Config Param: BIGQUERY_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` [...]
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` [...]
+---
+
+
+### Hive Sync Configs {#Hive-Sync-Configs}
+Configurations used by the Hudi to sync metadata to Hive Metastore.
+
+
+
+[**Basic Configs**](#Hive-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+
+[**Advanced Configs**](#Hive-Sync-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` |
+| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` |
+| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` |
+| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` |
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` |
+| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` |
+| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` |
+| [hoodie.datasource.hive_sync.bucket_sync_spec](#hoodiedatasourcehive_syncbucket_sync_spec) | | The hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC_SPEC` |
+| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` |
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` |
+| [hoodie.datasource.hive_sync.filter_pushdown_enabled](#hoodiedatasourcehive_syncfilter_pushdown_enabled) | false | Whether to enable push down partitions by filter<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_ENABLED` |
+| [hoodie.datasource.hive_sync.filter_pushdown_max_size](#hoodiedatasourcehive_syncfilter_pushdown_max_size) | 1000 | Max size limit to push down partition filters, if the estimate push down filters exceed this size, will directly try to fetch all partitions between the min/max.In case of glue metastore, this value should be reduced because it has a filter length limit.<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_MAX_SIZE` |
+| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` |
+| [hoodie.datasource.hive_sync.omit_metadata_fields](#hoodiedatasourcehive_syncomit_metadata_fields) | false | Whether to omit the hoodie metadata fields in the target table.<br />`Config Param: HIVE_SYNC_OMIT_METADATA_FIELDS`<br />`Since Version: 0.13.0` |
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` |
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` |
+| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` |
+| [hoodie.datasource.hive_sync.schema_string_length_thresh](#hoodiedatasourcehive_syncschema_string_length_thresh) | 4000 | <br />`Config Param: HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD` |
+| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` |
+| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` |
+| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` |
+| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` |
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` |
+| [hoodie.datasource.hive_sync.table.strategy](#hoodiedatasourcehive_synctablestrategy) | ALL | Hive table synchronization strategy. Available option: RO, RT, ALL.<br />`Config Param: HIVE_SYNC_TABLE_STRATEGY`<br />`Since Version: 0.13.0` |
+| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` |
+| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMAT` |
+| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` |
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` |
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` |
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` |
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` |
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` |
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` |
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` |
+---
+
+
+### Global Hive Sync Configs {#Global-Hive-Sync-Configs}
+Global replication configurations used by the Hudi to sync metadata to Hive Metastore.
+
+
+
+[**Basic Configs**](#Global-Hive-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+
+[**Advanced Configs**](#Global-Hive-Sync-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` |
+| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` |
+| [hoodie.meta_sync.global.replicate.timestamp](#hoodiemeta_syncglobalreplicatetimestamp) | (N/A) | <br />`Config Param: META_SYNC_GLOBAL_REPLICATE_TIMESTAMP` |
+| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` |
+| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` |
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` |
+| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` |
+| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` |
+| [hoodie.datasource.hive_sync.bucket_sync_spec](#hoodiedatasourcehive_syncbucket_sync_spec) | | The hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC_SPEC` |
+| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` |
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` |
+| [hoodie.datasource.hive_sync.filter_pushdown_enabled](#hoodiedatasourcehive_syncfilter_pushdown_enabled) | false | Whether to enable push down partitions by filter<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_ENABLED` |
+| [hoodie.datasource.hive_sync.filter_pushdown_max_size](#hoodiedatasourcehive_syncfilter_pushdown_max_size) | 1000 | Max size limit to push down partition filters, if the estimate push down filters exceed this size, will directly try to fetch all partitions between the min/max.In case of glue metastore, this value should be reduced because it has a filter length limit.<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_MAX_SIZE` |
+| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` |
+| [hoodie.datasource.hive_sync.omit_metadata_fields](#hoodiedatasourcehive_syncomit_metadata_fields) | false | Whether to omit the hoodie metadata fields in the target table.<br />`Config Param: HIVE_SYNC_OMIT_METADATA_FIELDS`<br />`Since Version: 0.13.0` |
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` |
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` |
+| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` |
+| [hoodie.datasource.hive_sync.schema_string_length_thresh](#hoodiedatasourcehive_syncschema_string_length_thresh) | 4000 | <br />`Config Param: HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD` |
+| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` |
+| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` |
+| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` |
+| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` |
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` |
+| [hoodie.datasource.hive_sync.table.strategy](#hoodiedatasourcehive_synctablestrategy) | ALL | Hive table synchronization strategy. Available option: RO, RT, ALL.<br />`Config Param: HIVE_SYNC_TABLE_STRATEGY`<br />`Since Version: 0.13.0` |
+| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` |
+| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMAT` |
+| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` |
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` |
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` |
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` |
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` |
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` |
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` |
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` |
+---
+
+
+### DataHub Sync Configs {#DataHub-Sync-Configs}
+Configurations used by the Hudi to sync metadata to DataHub.
+
+
+
+[**Basic Configs**](#DataHub-Sync-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
+
+[**Advanced Configs**](#DataHub-Sync-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.meta.sync.datahub.emitter.server](#hoodiemetasyncdatahubemitterserver) | (N/A) | Server URL of the DataHub instance.<br />`Config Param: META_SYNC_DATAHUB_EMITTER_SERVER` |
+| [hoodie.meta.sync.datahub.emitter.supplier.class](#hoodiemetasyncdatahubemittersupplierclass) | (N/A) | Pluggable class to supply a DataHub REST emitter to connect to the DataHub instance. This overwrites other emitter configs.<br />`Config Param: META_SYNC_DATAHUB_EMITTER_SUPPLIER_CLASS` |
+| [hoodie.meta.sync.datahub.emitter.token](#hoodiemetasyncdatahubemittertoken) | (N/A) | Auth token to connect to the DataHub instance.<br />`Config Param: META_SYNC_DATAHUB_EMITTER_TOKEN` |
+| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` |
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` |
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` |
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` |
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` |
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` |
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` |
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` |
+| [hoodie.meta.sync.datahub.dataplatform.name](#hoodiemetasyncdatahubdataplatformname) | hudi | String used to represent Hudi when creating its corresponding DataPlatform entity within Datahub<br />`Config Param: META_SYNC_DATAHUB_DATAPLATFORM_NAME` |
+| [hoodie.meta.sync.datahub.dataset.env](#hoodiemetasyncdatahubdatasetenv) | DEV | Environment to use when pushing entities to Datahub<br />`Config Param: META_SYNC_DATAHUB_DATASET_ENV` |
+| [hoodie.meta.sync.datahub.dataset.identifier.class](#hoodiemetasyncdatahubdatasetidentifierclass) | org.apache.hudi.sync.datahub.config.HoodieDataHubDatasetIdentifier | Pluggable class to help provide info to identify a DataHub Dataset.<br />`Config Param: META_SYNC_DATAHUB_DATASET_IDENTIFIER_CLASS` |
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` |
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` |
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` |
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` |
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` |
+---
+
+## Metrics Configs {#METRICS}
+These set of configs are used to enable monitoring and reporting of key Hudi stats and metrics.
+
+
+### Metrics Configurations for Amazon CloudWatch {#Metrics-Configurations-for-Amazon-CloudWatch}
+Enables reporting on Hudi metrics using Amazon CloudWatch. Hudi publishes metrics on every commit, clean, rollback etc.
+
+
+
+[**Advanced Configs**](#Metrics-Configurations-for-Amazon-CloudWatch-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.cloudwatch.maxDatumsPerRequest](#hoodiemetricscloudwatchmaxDatumsPerRequest) | 20 | Max number of Datums per request<br />`Config Param: MAX_DATUMS_PER_REQUEST`<br />`Since Version: 0.10.0` |
+| [hoodie.metrics.cloudwatch.metric.prefix](#hoodiemetricscloudwatchmetricprefix) | | Metric prefix of reporter<br />`Config Param: METRIC_PREFIX`<br />`Since Version: 0.10.0` |
+| [hoodie.metrics.cloudwatch.namespace](#hoodiemetricscloudwatchnamespace) | Hudi | Namespace of reporter<br />`Config Param: METRIC_NAMESPACE`<br />`Since Version: 0.10.0` |
+| [hoodie.metrics.cloudwatch.report.period.seconds](#hoodiemetricscloudwatchreportperiodseconds) | 60 | Reporting interval in seconds<br />`Config Param: REPORT_PERIOD_SECONDS`<br />`Since Version: 0.10.0` |
+---
+
+
+### Metrics Configurations {#Metrics-Configurations}
+Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean, rollback etc. The following sections list the supported reporters.
+
+
+
+[**Basic Configs**](#Metrics-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.on](#hoodiemetricson) | false | Turn on/off metrics reporting. off by default.<br />`Config Param: TURN_METRICS_ON`<br />`Since Version: 0.5.0` |
+| [hoodie.metrics.reporter.type](#hoodiemetricsreportertype) | GRAPHITE | Type of metrics reporter.<br />`Config Param: METRICS_REPORTER_TYPE_VALUE`<br />`Since Version: 0.5.0` |
+| [hoodie.metricscompaction.log.blocks.on](#hoodiemetricscompactionlogblockson) | false | Turn on/off metrics reporting for log blocks with compaction commit. off by default.<br />`Config Param: TURN_METRICS_COMPACTION_LOG_BLOCKS_ON`<br />`Since Version: 0.14.0` |
+
+[**Advanced Configs**](#Metrics-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.executor.enable](#hoodiemetricsexecutorenable) | (N/A) | <br />`Config Param: EXECUTOR_METRICS_ENABLE`<br />`Since Version: 0.7.0` |
+| [hoodie.metrics.configs.properties](#hoodiemetricsconfigsproperties) | | Comma separated list of config file paths for metric exporter configs<br />`Config Param: METRICS_REPORTER_FILE_BASED_CONFIGS_PATH`<br />`Since Version: 0.14.0` |
+| [hoodie.metrics.lock.enable](#hoodiemetricslockenable) | false | Enable metrics for locking infra. Useful when operating in multiwriter mode<br />`Config Param: LOCK_METRICS_ENABLE`<br />`Since Version: 0.13.0` |
+| [hoodie.metrics.reporter.class](#hoodiemetricsreporterclass) | | <br />`Config Param: METRICS_REPORTER_CLASS_NAME`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.reporter.metricsname.prefix](#hoodiemetricsreportermetricsnameprefix) | | The prefix given to the metrics names.<br />`Config Param: METRICS_REPORTER_PREFIX`<br />`Since Version: 0.11.0` |
+---
+
+
+### Metrics Configurations for Datadog reporter {#Metrics-Configurations-for-Datadog-reporter}
+Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishes metrics on every commit, clean, rollback etc.
+
+
+
+[**Advanced Configs**](#Metrics-Configurations-for-Datadog-reporter-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.datadog.api.key](#hoodiemetricsdatadogapikey) | (N/A) | Datadog API key<br />`Config Param: API_KEY`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.datadog.api.key.supplier](#hoodiemetricsdatadogapikeysupplier) | (N/A) | Datadog API key supplier to supply the API key at runtime. This will take effect if hoodie.metrics.datadog.api.key is not set.<br />`Config Param: API_KEY_SUPPLIER`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.datadog.api.site](#hoodiemetricsdatadogapisite) | (N/A) | Datadog API site: EU or US<br />`Config Param: API_SITE_VALUE`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.datadog.metric.host](#hoodiemetricsdatadogmetrichost) | (N/A) | Datadog metric host to be sent along with metrics data.<br />`Config Param: METRIC_HOST_NAME`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.datadog.metric.prefix](#hoodiemetricsdatadogmetricprefix) | (N/A) | Datadog metric prefix to be prepended to each metric name with a dot as delimiter. For example, if it is set to foo, foo. will be prepended.<br />`Config Param: METRIC_PREFIX_VALUE`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.datadog.metric.tags](#hoodiemetricsdatadogmetrictags) | (N/A) | Datadog metric tags (comma-delimited) to be sent along with metrics data.<br />`Config Param: METRIC_TAG_VALUES`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.datadog.api.key.skip.validation](#hoodiemetricsdatadogapikeyskipvalidation) | false | Before sending metrics via Datadog API, whether to skip validating Datadog API key or not. Default to false.<br />`Config Param: API_KEY_SKIP_VALIDATION`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.datadog.api.timeout.seconds](#hoodiemetricsdatadogapitimeoutseconds) | 3 | Datadog API timeout in seconds. Default to 3.<br />`Config Param: API_TIMEOUT_IN_SECONDS`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.datadog.report.period.seconds](#hoodiemetricsdatadogreportperiodseconds) | 30 | Datadog reporting period in seconds. Default to 30.<br />`Config Param: REPORT_PERIOD_IN_SECONDS`<br />`Since Version: 0.6.0` |
+---
+
+
+### Metrics Configurations for Graphite {#Metrics-Configurations-for-Graphite}
+Enables reporting on Hudi metrics using Graphite. Hudi publishes metrics on every commit, clean, rollback etc.
+
+
+
+[**Advanced Configs**](#Metrics-Configurations-for-Graphite-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------ | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.graphite.metric.prefix](#hoodiemetricsgraphitemetricprefix) | (N/A) | Standard prefix applied to all metrics. This helps to add datacenter, environment information for e.g<br />`Config Param: GRAPHITE_METRIC_PREFIX_VALUE`<br />`Since Version: 0.5.1` |
+| [hoodie.metrics.graphite.host](#hoodiemetricsgraphitehost) | localhost | Graphite host to connect to.<br />`Config Param: GRAPHITE_SERVER_HOST_NAME`<br />`Since Version: 0.5.0` |
+| [hoodie.metrics.graphite.port](#hoodiemetricsgraphiteport) | 4756 | Graphite port to connect to.<br />`Config Param: GRAPHITE_SERVER_PORT_NUM`<br />`Since Version: 0.5.0` |
+| [hoodie.metrics.graphite.report.period.seconds](#hoodiemetricsgraphitereportperiodseconds) | 30 | Graphite reporting period in seconds. Default to 30.<br />`Config Param: GRAPHITE_REPORT_PERIOD_IN_SECONDS`<br />`Since Version: 0.10.0` |
+---
+
+
+### Metrics Configurations for Jmx {#Metrics-Configurations-for-Jmx}
+Enables reporting on Hudi metrics using Jmx. Hudi publishes metrics on every commit, clean, rollback etc.
+
+
+
+[**Advanced Configs**](#Metrics-Configurations-for-Jmx-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------ | ---------- | ------------------------------------------------------------------------------------- |
+| [hoodie.metrics.jmx.host](#hoodiemetricsjmxhost) | localhost | Jmx host to connect to<br />`Config Param: JMX_HOST_NAME`<br />`Since Version: 0.5.1` |
+| [hoodie.metrics.jmx.port](#hoodiemetricsjmxport) | 9889 | Jmx port to connect to<br />`Config Param: JMX_PORT_NUM`<br />`Since Version: 0.5.1` |
+---
+
+
+### Metrics Configurations for M3 {#Metrics-Configurations-for-M3}
+Enables reporting on Hudi metrics using M3. Hudi publishes metrics on every commit, clean, rollback etc.
+
+
+
+[**Basic Configs**](#Metrics-Configurations-for-M3-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.m3.env](#hoodiemetricsm3env) | production | M3 tag to label the environment (defaults to 'production'), applied to all metrics.<br />`Config Param: M3_ENV`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.host](#hoodiemetricsm3host) | localhost | M3 host to connect to.<br />`Config Param: M3_SERVER_HOST_NAME`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.port](#hoodiemetricsm3port) | 9052 | M3 port to connect to.<br />`Config Param: M3_SERVER_PORT_NUM`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.service](#hoodiemetricsm3service) | hoodie | M3 tag to label the service name (defaults to 'hoodie'), applied to all metrics.<br />`Config Param: M3_SERVICE`<br />`Since Version: 0.15.0` |
+| [hoodie.metrics.m3.tags](#hoodiemetricsm3tags) | | Optional M3 tags applied to all metrics.<br />`Config Param: M3_TAGS`<br />`Since Version: 0.15.0` |
+---
+
+
+### Metrics Configurations for Prometheus {#Metrics-Configurations-for-Prometheus}
+Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on every commit, clean, rollback etc.
+
+
+
+[**Advanced Configs**](#Metrics-Configurations-for-Prometheus-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.metrics.prometheus.port](#hoodiemetricsprometheusport) | 9090 | Port for prometheus server.<br />`Config Param: PROMETHEUS_PORT_NUM`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.pushgateway.delete.on.shutdown](#hoodiemetricspushgatewaydeleteonshutdown) | true | Delete the pushgateway info or not when job shutdown, true by default.<br />`Config Param: PUSHGATEWAY_DELETE_ON_SHUTDOWN_ENABLE`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.pushgateway.host](#hoodiemetricspushgatewayhost) | localhost | Hostname of the prometheus push gateway.<br />`Config Param: PUSHGATEWAY_HOST_NAME`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.pushgateway.job.name](#hoodiemetricspushgatewayjobname) | | Name of the push gateway job.<br />`Config Param: PUSHGATEWAY_JOBNAME`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.pushgateway.port](#hoodiemetricspushgatewayport) | 9091 | Port for the push gateway.<br />`Config Param: PUSHGATEWAY_PORT_NUM`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.pushgateway.random.job.name.suffix](#hoodiemetricspushgatewayrandomjobnamesuffix) | true | Whether the pushgateway name need a random suffix , default true.<br />`Config Param: PUSHGATEWAY_RANDOM_JOBNAME_SUFFIX`<br />`Since Version: 0.6.0` |
+| [hoodie.metrics.pushgateway.report.labels](#hoodiemetricspushgatewayreportlabels) | | Label for the metrics emitted to the Pushgateway. Labels can be specified with key:value pairs separated by commas<br />`Config Param: PUSHGATEWAY_LABELS`<br />`Since Version: 0.14.0` |
+| [hoodie.metrics.pushgateway.report.period.seconds](#hoodiemetricspushgatewayreportperiodseconds) | 30 | Reporting interval in seconds.<br />`Config Param: PUSHGATEWAY_REPORT_PERIOD_IN_SECONDS`<br />`Since Version: 0.6.0` |
+---
+
+## Record Payload Config {#RECORD_PAYLOAD}
+This is the lowest level of customization offered by Hudi. Record payloads define how to produce new values to upsert based on incoming new record and stored old record. Hudi provides default implementations such as OverwriteWithLatestAvroPayload which simply update table with the latest/last-written record. This can be overridden to a custom class extending HoodieRecordPayload class, on both datasource and WriteClient levels.
+
+
+### Payload Configurations {#Payload-Configurations}
+Payload related configs, that can be leveraged to control merges based on specific business fields in the data.
+
+
+
+[**Advanced Configs**](#Payload-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | org.apache.hudi.common.model.OverwriteWithLatestAvroPayload | This needs to be same as class used during insert/upserts. Just like writing, compaction also uses the record payload class to merge records in the log against each other, merge again with the base file and produce the final record to be written after compaction.<br />`Config Param: PAYLOAD_CLASS_NAME` |
+| [hoodie.payload.event.time.field](#hoodiepayloadeventtimefield) | ts | Table column/field name to derive timestamp associated with the records. This canbe useful for e.g, determining the freshness of the table.<br />`Config Param: EVENT_TIME_FIELD` |
+| [hoodie.payload.ordering.field](#hoodiepayloadorderingfield) | ts | Table column/field name to order records that have the same key, before merging and writing to storage.<br />`Config Param: ORDERING_FIELD` |
+---
+
+## Kafka Connect Configs {#KAFKA_CONNECT}
+These set of configs are used for Kafka Connect Sink Connector for writing Hudi Tables
+
+
+### Kafka Sink Connect Configurations {#Kafka-Sink-Connect-Configurations}
+Configurations for Kafka Connect Sink Connector for Hudi.
+
+
+
+[**Basic Configs**](#Kafka-Sink-Connect-Configurations-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------- | --------------- | ----------------------------------------------------------------------------------------- |
+| [bootstrap.servers](#bootstrapservers) | localhost:9092 | The bootstrap servers for the Kafka Cluster.<br />`Config Param: KAFKA_BOOTSTRAP_SERVERS` |
+
+[**Advanced Configs**](#Kafka-Sink-Connect-Configurations-advanced-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hadoop.conf.dir](#hadoopconfdir) | (N/A) | The Hadoop configuration directory.<br />`Config Param: HADOOP_CONF_DIR` |
+| [hadoop.home](#hadoophome) | (N/A) | The Hadoop home directory.<br />`Config Param: HADOOP_HOME` |
+| [hoodie.kafka.allow.commit.on.errors](#hoodiekafkaallowcommitonerrors) | true | Commit even when some records failed to be written<br />`Config Param: ALLOW_COMMIT_ON_ERRORS` |
+| [hoodie.kafka.commit.interval.secs](#hoodiekafkacommitintervalsecs) | 60 | The interval at which Hudi will commit the records written to the files, making them consumable on the read-side.<br />`Config Param: COMMIT_INTERVAL_SECS` |
+| [hoodie.kafka.compaction.async.enable](#hoodiekafkacompactionasyncenable) | true | Controls whether async compaction should be turned on for MOR table writing.<br />`Config Param: ASYNC_COMPACT_ENABLE` |
+| [hoodie.kafka.control.topic](#hoodiekafkacontroltopic) | hudi-control-topic | Kafka topic name used by the Hudi Sink Connector for sending and receiving control messages. Not used for data records.<br />`Config Param: CONTROL_TOPIC_NAME` |
+| [hoodie.kafka.coordinator.write.timeout.secs](#hoodiekafkacoordinatorwritetimeoutsecs) | 300 | The timeout after sending an END_COMMIT until when the coordinator will wait for the write statuses from all the partitionsto ignore the current commit and start a new commit.<br />`Config Param: COORDINATOR_WRITE_TIMEOUT_SECS` |
+| [hoodie.meta.sync.classes](#hoodiemetasyncclasses) | org.apache.hudi.hive.HiveSyncTool | Meta sync client tool, using comma to separate multi tools<br />`Config Param: META_SYNC_CLASSES` |
+| [hoodie.meta.sync.enable](#hoodiemetasyncenable) | false | Enable Meta Sync such as Hive<br />`Config Param: META_SYNC_ENABLE` |
+| [hoodie.schemaprovider.class](#hoodieschemaproviderclass) | org.apache.hudi.schema.FilebasedSchemaProvider | subclass of org.apache.hudi.schema.SchemaProvider to attach schemas to input & target table data, built in options: org.apache.hudi.schema.FilebasedSchemaProvider.<br />`Config Param: SCHEMA_PROVIDER_CLASS` |
+---
+
+## Amazon Web Services Configs {#AWS}
+Configurations specific to Amazon Web Services.
+
+
+### Amazon Web Services Configs {#Amazon-Web-Services-Configs}
+Amazon Web Services configurations to access resources like Amazon DynamoDB (for locks), Amazon CloudWatch (metrics).
+
+
+
+[**Advanced Configs**](#Amazon-Web-Services-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.aws.access.key](#hoodieawsaccesskey) | (N/A) | AWS access key id<br />`Config Param: AWS_ACCESS_KEY`<br />`Since Version: 0.10.0` |
+| [hoodie.aws.glue.endpoint](#hoodieawsglueendpoint) | (N/A) | Aws glue endpoint<br />`Config Param: AWS_GLUE_ENDPOINT`<br />`Since Version: 0.15.0` |
+| [hoodie.aws.glue.region](#hoodieawsglueregion) | (N/A) | Aws glue endpoint<br />`Config Param: AWS_GLUE_REGION`<br />`Since Version: 0.15.0` |
+| [hoodie.aws.role.arn](#hoodieawsrolearn) | (N/A) | AWS Role ARN to assume<br />`Config Param: AWS_ASSUME_ROLE_ARN`<br />`Since Version: 0.15.0` |
+| [hoodie.aws.role.external.id](#hoodieawsroleexternalid) | (N/A) | External ID use when assuming the AWS Role<br />`Config Param: AWS_ASSUME_ROLE_EXTERNAL_ID`<br />`Since Version: 0.15.0` |
+| [hoodie.aws.secret.key](#hoodieawssecretkey) | (N/A) | AWS secret key<br />`Config Param: AWS_SECRET_KEY`<br />`Since Version: 0.10.0` |
+| [hoodie.aws.session.token](#hoodieawssessiontoken) | (N/A) | AWS session token<br />`Config Param: AWS_SESSION_TOKEN`<br />`Since Version: 0.10.0` |
+| [hoodie.aws.role.session.name](#hoodieawsrolesessionname) | hoodie | Session name to use when assuming the AWS Role<br />`Config Param: AWS_ASSUME_ROLE_SESSION_NAME`<br />`Since Version: 0.15.0` |
+---
+
+## Hudi Streamer Configs {#HUDI_STREAMER}
+These set of configs are used for Hudi Streamer utility which provides the way to ingest from different sources such as DFS or Kafka.
+
+
+### Hudi Streamer Configs {#Hudi-Streamer-Configs}
+
+
+
+
+[**Basic Configs**](#Hudi-Streamer-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.kafka.topic](#hoodiestreamersourcekafkatopic) | (N/A) | Kafka topic name. The config is specific to HoodieMultiTableStreamer<br />`Config Param: KAFKA_TOPIC` |
+
+[**Advanced Configs**](#Hudi-Streamer-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.streamer.checkpoint.provider.path](#hoodiestreamercheckpointproviderpath) | (N/A) | The path for providing the checkpoints.<br />`Config Param: CHECKPOINT_PROVIDER_PATH` [...]
+| [hoodie.streamer.ingestion.tablesToBeIngested](#hoodiestreameringestiontablesToBeIngested) | (N/A) | Comma separated names of tables to be ingested in the format <database>.<table>, for example db1.table1,db1.table2<br />`Config Param: TABLES_TO_BE_INGESTED` [...]
+| [hoodie.streamer.transformer.class](#hoodiestreamertransformerclass) | (N/A) | Names of transformer classes to apply. The config is specific to HoodieMultiTableStreamer.<br />`Config Param: TRANSFORMER_CLASS`<br />`Since Version: 0.14.0` [...]
+| [hoodie.streamer.checkpoint.force.skip](#hoodiestreamercheckpointforceskip) | false | Config to force to skip saving checkpoint in the commit metadata.It is typically used in one-time backfill scenarios, where checkpoints are not to be persisted.<br />`Config Param: CHECKPOINT_FORCE_SKIP` [...]
+| [hoodie.streamer.ingestion.targetBasePath](#hoodiestreameringestiontargetBasePath) | | The path to which a particular table is ingested. The config is specific to HoodieMultiTableStreamer and overrides path determined using option `--base-path-prefix` for a table. This config is ignored for a single table streamer<br />`Config Param: TARGET_BASE_PATH` [...]
+| [hoodie.streamer.row.throw.explicit.exceptions](#hoodiestreamerrowthrowexplicitexceptions) | false | When enabled, the dataframe generated from reading source data is wrapped with an exception handler to explicitly surface exceptions.<br />`Config Param: ROW_THROW_EXPLICIT_EXCEPTIONS`<br />`Since Version: 0.15.0` [...]
+| [hoodie.streamer.sample.writes.enabled](#hoodiestreamersamplewritesenabled) | false | Set this to true to sample from the first batch of records and write to the auxiliary path, before writing to the table.The sampled records are used to calculate the average record size. The relevant write client will have `hoodie.copyonwrite.record.size.estimate` being overwritten by the calculated result.<br />`Config Param: SAMPLE_WRITES_ENABLED`<br />`Since [...]
+| [hoodie.streamer.sample.writes.size](#hoodiestreamersamplewritessize) | 5000 | Number of records to sample from the first write. To improve the estimation's accuracy, for smaller or more compressable record size, set the sample size bigger. For bigger or less compressable record size, set smaller.<br />`Config Param: SAMPLE_WRITES_SIZE`<br />`Since Version: 0.14.0` [...]
+| [hoodie.streamer.source.kafka.append.offsets](#hoodiestreamersourcekafkaappendoffsets) | false | When enabled, appends kafka offset info like source offset(_hoodie_kafka_source_offset), partition (_hoodie_kafka_source_partition) and timestamp (_hoodie_kafka_source_timestamp) to the records. By default its disabled and no kafka offsets are added<br />`Config Param: KAFKA_APPEND_OFFSETS` [...]
+| [hoodie.streamer.source.sanitize.invalid.char.mask](#hoodiestreamersourcesanitizeinvalidcharmask) | __ | Defines the character sequence that replaces invalid characters in schema field names if hoodie.streamer.source.sanitize.invalid.schema.field.names is enabled.<br />`Config Param: SCHEMA_FIELD_NAME_INVALID_CHAR_MASK` [...]
+| [hoodie.streamer.source.sanitize.invalid.schema.field.names](#hoodiestreamersourcesanitizeinvalidschemafieldnames) | false | Sanitizes names of invalid schema fields both in the data read from source and also in the schema Replaces invalid characters with hoodie.streamer.source.sanitize.invalid.char.mask. Invalid characters are by goes by avro naming convention (https://avro.apache.org/docs/++version++/specification/#names).<br />`Config Param: SANITIZE_SCHEMA_FIELD_NAMES` [...]
+---
+
+
+### Hudi Streamer SQL Transformer Configs {#Hudi-Streamer-SQL-Transformer-Configs}
+Configurations controlling the behavior of SQL transformer in Hudi Streamer.
+
+
+
+[**Basic Configs**](#Hudi-Streamer-SQL-Transformer-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.transformer.sql](#hoodiestreamertransformersql) | (N/A) | SQL Query to be executed during write<br />`Config Param: TRANSFORMER_SQL` |
+| [hoodie.streamer.transformer.sql.file](#hoodiestreamertransformersqlfile) | (N/A) | File with a SQL script to be executed during write<br />`Config Param: TRANSFORMER_SQL_FILE` |
+---
+
+
+### Hudi Streamer Source Configs {#DELTA_STREAMER_SOURCE}
+Configurations controlling the behavior of reading source data.
+
+
+#### Cloud Source Configs {#Cloud-Source-Configs}
+Configs that are common during ingestion across different cloud stores
+
+
+
+[**Advanced Configs**](#Cloud-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.streamer.source.cloud.data.datasource.options](#hoodiestreamersourceclouddatadatasourceoptions) | (N/A) | A JSON string passed to the Spark DataFrameReader while loading the dataset. Example: hoodie.streamer.gcp.spark.datasource.options=\{"header":"true","encoding":"UTF-8"} <br />`Config Param: SPARK_DATASOURCE_OPTIONS` [...]
+| [hoodie.streamer.source.cloud.data.ignore.relpath.prefix](#hoodiestreamersourceclouddataignorerelpathprefix) | (N/A) | Ignore objects in the bucket whose relative path starts this prefix<br />`Config Param: IGNORE_RELATIVE_PATH_PREFIX` [...]
+| [hoodie.streamer.source.cloud.data.ignore.relpath.substring](#hoodiestreamersourceclouddataignorerelpathsubstring) | (N/A) | Ignore objects in the bucket whose relative path contains this substring<br />`Config Param: IGNORE_RELATIVE_PATH_SUBSTR` [...]
+| [hoodie.streamer.source.cloud.data.partition.fields.from.path](#hoodiestreamersourceclouddatapartitionfieldsfrompath) | (N/A) | A comma delimited list of path-based partition fields in the source file structure.<br />`Config Param: PATH_BASED_PARTITION_FIELDS`<br />`Since Version: 0.14.0` [...]
+| [hoodie.streamer.source.cloud.data.partition.max.size](#hoodiestreamersourceclouddatapartitionmaxsize) | (N/A) | specify this value in bytes, to coalesce partitions of source dataset not greater than specified limit<br />`Config Param: SOURCE_MAX_BYTES_PER_PARTITION`<br />`Since Version: 0.14.1` [...]
+| [hoodie.streamer.source.cloud.data.select.file.extension](#hoodiestreamersourceclouddataselectfileextension) | (N/A) | Only match files with this extension. By default, this is the same as hoodie.streamer.source.hoodieincr.file.format<br />`Config Param: CLOUD_DATAFILE_EXTENSION` [...]
+| [hoodie.streamer.source.cloud.data.select.relpath.prefix](#hoodiestreamersourceclouddataselectrelpathprefix) | (N/A) | Only selects objects in the bucket whose relative path starts with this prefix<br />`Config Param: SELECT_RELATIVE_PATH_PREFIX` [...]
+| [hoodie.streamer.source.cloud.data.check.file.exists](#hoodiestreamersourceclouddatacheckfileexists) | false | If true, checks whether file exists before attempting to pull it<br />`Config Param: ENABLE_EXISTS_CHECK` [...]
+| [hoodie.streamer.source.cloud.data.datafile.format](#hoodiestreamersourceclouddatadatafileformat) | parquet | Format of the data file. By default, this will be the same as hoodie.streamer.source.hoodieincr.file.format<br />`Config Param: DATAFILE_FORMAT` [...]
+| [hoodie.streamer.source.cloud.data.reader.comma.separated.path.format](#hoodiestreamersourceclouddatareadercommaseparatedpathformat) | false | Boolean value for specifying path format in load args of spark.read.format("..").load("a.xml,b.xml,c.xml"), * set true if path format needs to be comma separated string value, if false it's passed as array of strings like * spark.read.format("..").load(new String[]\{a.xml,b.xml,c.xml})<br />`Config Param: SPARK_DATASOURCE_READER_COMMA_S [...]
+| [hoodie.streamer.source.cloud.meta.ack](#hoodiestreamersourcecloudmetaack) | true | Whether to acknowledge Metadata messages during Cloud Ingestion or not. This is useful during dev and testing. In Prod this should always be true. In case of Cloud Pubsub, not acknowledging means Pubsub will keep redelivering the same messages.<br />`Config Param: ACK_MESSAGES` [...]
+| [hoodie.streamer.source.cloud.meta.batch.size](#hoodiestreamersourcecloudmetabatchsize) | 10 | Number of metadata messages to pull in one API call to the cloud events queue. Multiple API calls with this batch size are sent to cloud events queue, until we consume hoodie.streamer.source.cloud.meta.max.num.messages.per.syncfrom the queue or hoodie.streamer.source.cloud.meta.max.fetch.time.per.sync.ms amount of time has passed or queue is [...]
+| [hoodie.streamer.source.cloud.meta.max.fetch.time.per.sync.secs](#hoodiestreamersourcecloudmetamaxfetchtimepersyncsecs) | 60 | Max time in secs to consume hoodie.streamer.source.cloud.meta.max.num.messages.per.sync messages from cloud queue. Cloud event queues like SQS, PubSub can return empty responses even when messages are available the queue, this config ensures we don't wait forever to consume MAX_MESSAGES_CONF messages, but time out and move on further.<br />`C [...]
+| [hoodie.streamer.source.cloud.meta.max.num.messages.per.sync](#hoodiestreamersourcecloudmetamaxnummessagespersync) | 1000 | Maximum number of messages to consume per sync round. Multiple rounds of hoodie.streamer.source.cloud.meta.batch.size could be invoked to reach max messages as configured by this config<br />`Config Param: MAX_NUM_MESSAGES_PER_SYNC`<br />`Since Version: 0.14.1` [...]
+---
+
+
+#### DFS Path Selector Configs {#DFS-Path-Selector-Configs}
+Configurations controlling the behavior of path selector for DFS source in Hudi Streamer.
+
+
+
+[**Basic Configs**](#DFS-Path-Selector-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------- | ------- | ------------------------------------------------------------------- |
+| [hoodie.streamer.source.dfs.root](#hoodiestreamersourcedfsroot) | (N/A) | Root path of the source on DFS<br />`Config Param: ROOT_INPUT_PATH` |
+
+[**Advanced Configs**](#DFS-Path-Selector-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------- |
+| [hoodie.streamer.source.input.selector](#hoodiestreamersourceinputselector) | (N/A) | Source input selector<br />`Config Param: SOURCE_INPUT_SELECTOR` |
+---
+
+
+#### Date Partition Path Selector Configs {#Date-Partition-Path-Selector-Configs}
+Configurations controlling the behavior of date partition path selector for DFS source in Hudi Streamer.
+
+
+
+[**Advanced Configs**](#Date-Partition-Path-Selector-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.dfs.datepartitioned.selector.currentdate](#hoodiestreamersourcedfsdatepartitionedselectorcurrentdate) | (N/A) | Current date.<br />`Config Param: CURRENT_DATE` |
+| [hoodie.streamer.source.dfs.datepartitioned.date.format](#hoodiestreamersourcedfsdatepartitioneddateformat) | yyyy-MM-dd | Date format.<br />`Config Param: DATE_FORMAT` |
+| [hoodie.streamer.source.dfs.datepartitioned.selector.depth](#hoodiestreamersourcedfsdatepartitionedselectordepth) | 0 | Depth of the files to scan. 0 implies no (date) partition.<br />`Config Param: DATE_PARTITION_DEPTH` |
+| [hoodie.streamer.source.dfs.datepartitioned.selector.lookback.days](#hoodiestreamersourcedfsdatepartitionedselectorlookbackdays) | 2 | The maximum look-back days for scanning.<br />`Config Param: LOOKBACK_DAYS` |
+| [hoodie.streamer.source.dfs.datepartitioned.selector.parallelism](#hoodiestreamersourcedfsdatepartitionedselectorparallelism) | 20 | Parallelism for listing partitions.<br />`Config Param: PARTITIONS_LIST_PARALLELISM` |
+---
+
+
+#### GCS Events Source Configs {#GCS-Events-Source-Configs}
+Configurations controlling the behavior of GCS Events Source in Hudi Streamer.
+
+
+
+[**Advanced Configs**](#GCS-Events-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.streamer.source.gcs.project.id](#hoodiestreamersourcegcsprojectid) | (N/A) | The GCP Project Id where the Pubsub Subscription to ingest from resides. Needed to connect to the Pubsub subscription<br />`Config Param: GOOGLE_PROJECT_ID` |
+| [hoodie.streamer.source.gcs.subscription.id](#hoodiestreamersourcegcssubscriptionid) | (N/A) | The GCP Pubsub subscription id for the GCS Notifications. Needed to connect to the Pubsub subscription<br />`Config Param: PUBSUB_SUBSCRIPTION_ID` |
+---
+
+
+#### Hive Incremental Pulling Source Configs {#Hive-Incremental-Pulling-Source-Configs}
+Configurations controlling the behavior of incremental pulling from a Hive table as a source in Hudi Streamer.
+
+
+
+[**Advanced Configs**](#Hive-Incremental-Pulling-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.incrpull.root](#hoodiestreamersourceincrpullroot) | (N/A) | The root path of Hive incremental pulling source.<br />`Config Param: ROOT_INPUT_PATH` |
+---
+
+
+#### Hudi Incremental Source Configs {#Hudi-Incremental-Source-Configs}
+Configurations controlling the behavior of incremental pulling from a Hudi table as a source in Hudi Streamer.
+
+
+
+[**Basic Configs**](#Hudi-Incremental-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------- |
+| [hoodie.streamer.source.hoodieincr.path](#hoodiestreamersourcehoodieincrpath) | (N/A) | Base-path for the source Hudi table<br />`Config Param: HOODIE_SRC_BASE_PATH` |
+
+[**Advanced Configs**](#Hudi-Incremental-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.streamer.source.hoodieincr.data.datasource.options](#hoodiestreamersourcehoodieincrdatadatasourceoptions) | (N/A) | A comma-separated list of Hudi options that can be passed to the spark dataframe reader of a hudi table, eg: `hoodie.metadata.enable=true,hoodie.enable.data.skipping=true`. Used only for incremental source.<br />`Config Param: HOODIE_INCREMENTAL_SPARK_DATASOURCE_OPTIONS`<br />`Since Version: 0.15.0` |
+| [hoodie.streamer.source.hoodieincr.missing.checkpoint.strategy](#hoodiestreamersourcehoodieincrmissingcheckpointstrategy) | (N/A) | Allows Hudi Streamer to decide the instant to consume from when checkpoint is not set. Possible values: [READ_LATEST (Read from latest commit in hoodie source table), READ_UPTO_LATEST_COMMIT (Read everything upto latest commit)]<br />`Config Param: MISSING_CHECKPOINT_STRATEGY` |
+| [hoodie.streamer.source.hoodieincr.partition.extractor.class](#hoodiestreamersourcehoodieincrpartitionextractorclass) | (N/A) | PartitionValueExtractor class to extract partition fields from _hoodie_partition_path<br />`Config Param: HOODIE_SRC_PARTITION_EXTRACTORCLASS` |
+| [hoodie.streamer.source.hoodieincr.partition.fields](#hoodiestreamersourcehoodieincrpartitionfields) | (N/A) | Specifies partition fields that needs to be added to source table after parsing _hoodie_partition_path.<br />`Config Param: HOODIE_SRC_PARTITION_FIELDS` |
+| [hoodie.streamer.source.hoodieincr.drop.all.meta.fields.from.source](#hoodiestreamersourcehoodieincrdropallmetafieldsfromsource) | false | Drops all meta fields from the source hudi table while ingesting into sink hudi table.<br />`Config Param: HOODIE_DROP_ALL_META_FIELDS_FROM_SOURCE` |
+| [hoodie.streamer.source.hoodieincr.file.format](#hoodiestreamersourcehoodieincrfileformat) | parquet | This config is passed to the reader while loading dataset. Default value is parquet.<br />`Config Param: SOURCE_FILE_FORMAT` |
+| [hoodie.streamer.source.hoodieincr.num_instants](#hoodiestreamersourcehoodieincrnum_instants) | 5 | Max number of instants whose changes can be incrementally fetched<br />`Config Param: NUM_INSTANTS_PER_FETCH` |
+| [hoodie.streamer.source.hoodieincr.read_latest_on_missing_ckpt](#hoodiestreamersourcehoodieincrread_latest_on_missing_ckpt) | false | If true, allows Hudi Streamer to incrementally fetch from latest committed instant when checkpoint is not provided. This config is deprecated. Please refer to hoodie.streamer.source.hoodieincr.missing.checkpoint.strategy<br />`Config Param: READ_LATEST_INSTANT_ON_MISSING_CKPT` |
+---
+
+
+#### JDBC Source Configs {#JDBC-Source-Configs}
+Configurations controlling the behavior of JDBC source in Hudi Streamer.
+
+
+
+[**Advanced Configs**](#JDBC-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.jdbc.driver.class](#hoodiestreamerjdbcdriverclass) | (N/A) | Driver class used for JDBC connection<br />`Config Param: DRIVER_CLASS` |
+| [hoodie.streamer.jdbc.extra.options.](#hoodiestreamerjdbcextraoptions) | (N/A) | Used to set any extra options the user specifies for jdbc<br />`Config Param: EXTRA_OPTIONS` |
+| [hoodie.streamer.jdbc.incr.fallback.to.full.fetch](#hoodiestreamerjdbcincrfallbacktofullfetch) | (N/A) | If set true, makes incremental fetch to fallback to full fetch in case of any error<br />`Config Param: FALLBACK_TO_FULL_FETCH` |
+| [hoodie.streamer.jdbc.incr.pull](#hoodiestreamerjdbcincrpull) | (N/A) | Will the JDBC source do an incremental pull?<br />`Config Param: IS_INCREMENTAL` |
+| [hoodie.streamer.jdbc.password](#hoodiestreamerjdbcpassword) | (N/A) | Password used for JDBC connection<br />`Config Param: PASSWORD` |
+| [hoodie.streamer.jdbc.password.file](#hoodiestreamerjdbcpasswordfile) | (N/A) | Base-path for the JDBC password file.<br />`Config Param: PASSWORD_FILE` |
+| [hoodie.streamer.jdbc.storage.level](#hoodiestreamerjdbcstoragelevel) | (N/A) | Used to control the persistence level. Default value: MEMORY_AND_DISK_SER<br />`Config Param: STORAGE_LEVEL` |
+| [hoodie.streamer.jdbc.table.incr.column.name](#hoodiestreamerjdbctableincrcolumnname) | (N/A) | If run in incremental mode, this field is to pull new data incrementally<br />`Config Param: INCREMENTAL_COLUMN` |
+| [hoodie.streamer.jdbc.table.name](#hoodiestreamerjdbctablename) | (N/A) | RDBMS table to pull<br />`Config Param: RDBMS_TABLE_NAME` |
+| [hoodie.streamer.jdbc.url](#hoodiestreamerjdbcurl) | (N/A) | JDBC url for the Hoodie datasource.<br />`Config Param: URL` |
+| [hoodie.streamer.jdbc.user](#hoodiestreamerjdbcuser) | (N/A) | Username used for JDBC connection<br />`Config Param: USER` |
+---
+
+
+#### Json Kafka Post Processor Configs {#Json-Kafka-Post-Processor-Configs}
+Configurations controlling the post processor of Json Kafka Source in Hudi Streamer.
+
+
+
+[**Advanced Configs**](#Json-Kafka-Post-Processor-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.json.kafka.post.processor.maxwell.database.regex](#hoodiestreamersourcejsonkafkapostprocessormaxwelldatabaseregex) | (N/A) | Database name regex<br />`Config Param: DATABASE_NAME_REGEX` |
+| [hoodie.streamer.source.json.kafka.post.processor.maxwell.table.regex](#hoodiestreamersourcejsonkafkapostprocessormaxwelltableregex) | (N/A) | Table name regex<br />`Config Param: TABLE_NAME_REGEX` |
+| [hoodie.streamer.source.json.kafka.processor.class](#hoodiestreamersourcejsonkafkaprocessorclass) | (N/A) | Json kafka source post processor class name, post process data after consuming fromsource and before giving it to Hudi Streamer.<br />`Config Param: JSON_KAFKA_PROCESSOR_CLASS` |
+| [hoodie.streamer.source.json.kafka.post.processor.maxwell.precombine.field.format](#hoodiestreamersourcejsonkafkapostprocessormaxwellprecombinefieldformat) | yyyy-MM-dd HH:mm:ss | When the preCombine filed is in DATE_STRING format, use should tell hoodiewhat format it is. 'yyyy-MM-dd HH:mm:ss' by default<br />`Config Param: PRECOMBINE_FIELD_FORMAT` |
+| [hoodie.streamer.source.json.kafka.post.processor.maxwell.precombine.field.type](#hoodiestreamersourcejsonkafkapostprocessormaxwellprecombinefieldtype) | DATE_STRING | Data type of the preCombine field. could be NON_TIMESTAMP, DATE_STRING,UNIX_TIMESTAMP or EPOCHMILLISECONDS. DATE_STRING by default<br />`Config Param: PRECOMBINE_FIELD_TYPE` |
+---
+
+
+#### Kafka Source Configs {#Kafka-Source-Configs}
+Configurations controlling the behavior of Kafka source in Hudi Streamer.
+
+
+
+[**Basic Configs**](#Kafka-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.kafka.topic](#hoodiestreamersourcekafkatopic) | (N/A) | Kafka topic name.<br />`Config Param: KAFKA_TOPIC_NAME` |
+| [hoodie.streamer.source.kafka.proto.value.deserializer.class](#hoodiestreamersourcekafkaprotovaluedeserializerclass) | org.apache.kafka.common.serialization.ByteArrayDeserializer | Kafka Proto Payload Deserializer Class<br />`Config Param: KAFKA_PROTO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.15.0` |
+
+[**Advanced Configs**](#Kafka-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
+| [hoodie.streamer.source.kafka.value.deserializer.schema](#hoodiestreamersourcekafkavaluedeserializerschema) | (N/A) | Schema to deserialize the records.<br />`Config Param: KAFKA_VALUE_DESERIALIZER_SCHEMA` [...]
+| [auto.offset.reset](#autooffsetreset) | LATEST | Kafka consumer strategy for reading data.<br />`Config Param: KAFKA_AUTO_OFFSET_RESET` [...]
+| [hoodie.streamer.kafka.source.maxEvents](#hoodiestreamerkafkasourcemaxEvents) | 5000000 | Maximum number of records obtained in each batch.<br />`Config Param: MAX_EVENTS_FROM_KAFKA_SOURCE` [...]
+| [hoodie.streamer.source.kafka.checkpoint.type](#hoodiestreamersourcekafkacheckpointtype) | string | Kafka checkpoint type.<br />`Config Param: KAFKA_CHECKPOINT_TYPE` [...]
+| [hoodie.streamer.source.kafka.enable.commit.offset](#hoodiestreamersourcekafkaenablecommitoffset) | false | Automatically submits offset to kafka.<br />`Config Param: ENABLE_KAFKA_COMMIT_OFFSET` [...]
+| [hoodie.streamer.source.kafka.enable.failOnDataLoss](#hoodiestreamersourcekafkaenablefailOnDataLoss) | false | Fail when checkpoint goes out of bounds instead of seeking to earliest offsets.<br />`Config Param: ENABLE_FAIL_ON_DATA_LOSS` [...]
+| [hoodie.streamer.source.kafka.fetch_partition.time.out](#hoodiestreamersourcekafkafetch_partitiontimeout) | 300000 | Time out for fetching partitions. 5min by default<br />`Config Param: KAFKA_FETCH_PARTITION_TIME_OUT` [...]
+| [hoodie.streamer.source.kafka.minPartitions](#hoodiestreamersourcekafkaminPartitions) | 0 | Desired minimum number of partitions to read from Kafka. By default, Hudi has a 1-1 mapping of topicPartitions to Hudi partitions consuming from Kafka. If set this option to a value greater than topicPartitions, Hudi will divvy up large Kafka partitions to smaller pieces. Please note that this configuration is like a hint: [...]
+| [hoodie.streamer.source.kafka.value.deserializer.class](#hoodiestreamersourcekafkavaluedeserializerclass) | io.confluent.kafka.serializers.KafkaAvroDeserializer | This class is used by kafka client to deserialize the records.<br />`Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.9.0` [...]
+---
+
+
+#### Parquet DFS Source Configs {#Parquet-DFS-Source-Configs}
+Configurations controlling the behavior of Parquet DFS source in Hudi Streamer.
+
+
+
+[**Advanced Configs**](#Parquet-DFS-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.parquet.dfs.merge_schema.enable](#hoodiestreamersourceparquetdfsmerge_schemaenable) | false | Merge schema across parquet files within a single write<br />`Config Param: PARQUET_DFS_MERGE_SCHEMA`<br />`Since Version: 0.15.0` |
+---
+
+
+#### Pulsar Source Configs {#Pulsar-Source-Configs}
+Configurations controlling the behavior of Pulsar source in Hudi Streamer.
+
+
+
+[**Basic Configs**](#Pulsar-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.pulsar.topic](#hoodiestreamersourcepulsartopic) | (N/A) | Name of the target Pulsar topic to source data from<br />`Config Param: PULSAR_SOURCE_TOPIC_NAME` |
+| [hoodie.streamer.source.pulsar.endpoint.admin.url](#hoodiestreamersourcepulsarendpointadminurl) | http://localhost:8080 | URL of the target Pulsar endpoint (of the form 'pulsar://host:port'<br />`Config Param: PULSAR_SOURCE_ADMIN_ENDPOINT_URL` |
+| [hoodie.streamer.source.pulsar.endpoint.service.url](#hoodiestreamersourcepulsarendpointserviceurl) | pulsar://localhost:6650 | URL of the target Pulsar endpoint (of the form 'pulsar://host:port'<br />`Config Param: PULSAR_SOURCE_SERVICE_ENDPOINT_URL` |
+
+[**Advanced Configs**](#Pulsar-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.pulsar.maxRecords](#hoodiestreamersourcepulsarmaxRecords) | 5000000 | Max number of records obtained in a single each batch<br />`Config Param: PULSAR_SOURCE_MAX_RECORDS_PER_BATCH_THRESHOLD` |
+| [hoodie.streamer.source.pulsar.offset.autoResetStrategy](#hoodiestreamersourcepulsaroffsetautoResetStrategy) | LATEST | Policy determining how offsets shall be automatically reset in case there's no checkpoint information present<br />`Config Param: PULSAR_SOURCE_OFFSET_AUTO_RESET_STRATEGY` |
+---
+
+
+#### S3 Event-based Hudi Incremental Source Configs {#S3-Event-based-Hudi-Incremental-Source-Configs}
+Configurations controlling the behavior of incremental pulling from S3 events meta information from Hudi table as a source in Hudi Streamer.
+
+
+
+[**Advanced Configs**](#S3-Event-based-Hudi-Incremental-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.s3incr.ignore.key.prefix](#hoodiestreamersources3incrignorekeyprefix) | (N/A) | Control whether to ignore the s3 objects starting with this prefix<br />`Config Param: S3_IGNORE_KEY_PREFIX` |
+| [hoodie.streamer.source.s3incr.ignore.key.substring](#hoodiestreamersources3incrignorekeysubstring) | (N/A) | Control whether to ignore the s3 objects with this substring<br />`Config Param: S3_IGNORE_KEY_SUBSTRING` |
+| [hoodie.streamer.source.s3incr.key.prefix](#hoodiestreamersources3incrkeyprefix) | (N/A) | Control whether to filter the s3 objects starting with this prefix<br />`Config Param: S3_KEY_PREFIX` |
+| [hoodie.streamer.source.s3incr.spark.datasource.options](#hoodiestreamersources3incrsparkdatasourceoptions) | (N/A) | Json string, passed to the reader while loading dataset. Example Hudi Streamer conf --hoodie-conf hoodie.streamer.source.s3incr.spark.datasource.options=\{"header":"true","encoding":"UTF-8"}<br />`Config Param: SPARK_DATASOURCE_OPTIONS` |
+| [hoodie.streamer.source.s3incr.check.file.exists](#hoodiestreamersources3incrcheckfileexists) | false | Control whether we do existence check for files before consuming them<br />`Config Param: S3_INCR_ENABLE_EXISTS_CHECK` |
+| [hoodie.streamer.source.s3incr.fs.prefix](#hoodiestreamersources3incrfsprefix) | s3 | The file system prefix.<br />`Config Param: S3_FS_PREFIX` |
+---
+
+
+#### S3 Source Configs {#S3-Source-Configs}
+Configurations controlling the behavior of S3 source in Hudi Streamer.
+
+
+
+[**Basic Configs**](#S3-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------- |
+| [hoodie.streamer.s3.source.queue.url](#hoodiestreamers3sourcequeueurl) | (N/A) | Queue url for cloud object events<br />`Config Param: S3_SOURCE_QUEUE_URL` |
+
+[**Advanced Configs**](#S3-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.s3.source.queue.region](#hoodiestreamers3sourcequeueregion) | (N/A) | Case-sensitive region name of the cloud provider for the queue. For example, "us-east-1".<br />`Config Param: S3_SOURCE_QUEUE_REGION` |
+| [hoodie.streamer.s3.source.queue.fs](#hoodiestreamers3sourcequeuefs) | s3 | File system corresponding to queue. For example, for AWS SQS it is s3/s3a.<br />`Config Param: S3_SOURCE_QUEUE_FS` |
+| [hoodie.streamer.s3.source.queue.long.poll.wait](#hoodiestreamers3sourcequeuelongpollwait) | 20 | Long poll wait time in seconds, If set as 0 then client will fetch on short poll basis.<br />`Config Param: S3_QUEUE_LONG_POLL_WAIT` |
+| [hoodie.streamer.s3.source.queue.max.messages.per.batch](#hoodiestreamers3sourcequeuemaxmessagesperbatch) | 5 | Max messages for each batch of Hudi Streamer run. Source will process these maximum number of message at a time.<br />`Config Param: S3_SOURCE_QUEUE_MAX_MESSAGES_PER_BATCH` |
+| [hoodie.streamer.s3.source.queue.max.messages.per.request](#hoodiestreamers3sourcequeuemaxmessagesperrequest) | 10 | Max messages for each request<br />`Config Param: S3_SOURCE_QUEUE_MAX_MESSAGES_PER_REQUEST` |
+| [hoodie.streamer.s3.source.queue.visibility.timeout](#hoodiestreamers3sourcequeuevisibilitytimeout) | 30 | Visibility timeout for messages in queue. After we consume the message, queue will move the consumed messages to in-flight state, these messages can't be consumed again by source for this timeout period.<br />`Config Param: S3_SOURCE_QUEUE_VISIBILITY_TIMEOUT` |
+---
+
+
+#### File-based SQL Source Configs {#File-based-SQL-Source-Configs}
+Configurations controlling the behavior of File-based SQL Source in Hudi Streamer.
+
+
+
+[**Basic Configs**](#File-based-SQL-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.sql.file](#hoodiestreamersourcesqlfile) | (N/A) | SQL file path containing the SQL query to read source data.<br />`Config Param: SOURCE_SQL_FILE`<br />`Since Version: 0.14.0` |
+
+[**Advanced Configs**](#File-based-SQL-Source-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.sql.checkpoint.emit](#hoodiestreamersourcesqlcheckpointemit) | false | Whether to emit the current epoch as the streamer checkpoint.<br />`Config Param: EMIT_EPOCH_CHECKPOINT`<br />`Since Version: 0.14.0` |
+---
+
+
+#### SQL Source Configs {#SQL-Source-Configs}
+Configurations controlling the behavior of SQL source in Hudi Streamer.
+
+
+
+[**Basic Configs**](#SQL-Source-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------ | ------- | ------------------------------------------------------------------- |
+| [hoodie.streamer.source.sql.sql.query](#hoodiestreamersourcesqlsqlquery) | (N/A) | SQL query for fetching source data.<br />`Config Param: SOURCE_SQL` |
+---
+
+
+### Hudi Streamer Schema Provider Configs {#SCHEMA_PROVIDER}
+Configurations that control the schema provider for Hudi Streamer.
+
+
+#### Hudi Streamer Schema Provider Configs {#Hudi-Streamer-Schema-Provider-Configs}
+
+
+
+
+[**Basic Configs**](#Hudi-Streamer-Schema-Provider-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.registry.targetUrl](#hoodiestreamerschemaproviderregistrytargetUrl) | (N/A) | The schema of the target you are writing to e.g. https://foo:bar@schemaregistry.org<br />`Config Param: TARGET_SCHEMA_REGISTRY_URL` |
+| [hoodie.streamer.schemaprovider.registry.url](#hoodiestreamerschemaproviderregistryurl) | (N/A) | The schema of the source you are reading from e.g. https://foo:bar@schemaregistry.org<br />`Config Param: SRC_SCHEMA_REGISTRY_URL` |
+
+[**Advanced Configs**](#Hudi-Streamer-Schema-Provider-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.registry.baseUrl](#hoodiestreamerschemaproviderregistrybaseUrl) | (N/A) | The base URL of the schema registry.<br />`Config Param: SCHEMA_REGISTRY_BASE_URL` |
+| [hoodie.streamer.schemaprovider.registry.schemaconverter](#hoodiestreamerschemaproviderregistryschemaconverter) | (N/A) | The class name of the custom schema converter to use.<br />`Config Param: SCHEMA_CONVERTER` |
+| [hoodie.streamer.schemaprovider.registry.sourceUrlSuffix](#hoodiestreamerschemaproviderregistrysourceUrlSuffix) | (N/A) | The source URL suffix.<br />`Config Param: SCHEMA_REGISTRY_SOURCE_URL_SUFFIX` |
+| [hoodie.streamer.schemaprovider.registry.targetUrlSuffix](#hoodiestreamerschemaproviderregistrytargetUrlSuffix) | (N/A) | The target URL suffix.<br />`Config Param: SCHEMA_REGISTRY_TARGET_URL_SUFFIX` |
+| [hoodie.streamer.schemaprovider.registry.urlSuffix](#hoodiestreamerschemaproviderregistryurlSuffix) | (N/A) | The suffix of the URL for the schema registry.<br />`Config Param: SCHEMA_REGISTRY_URL_SUFFIX` |
+| [hoodie.streamer.schemaprovider.spark_avro_post_processor.enable](#hoodiestreamerschemaproviderspark_avro_post_processorenable) | true | Whether to enable Spark Avro post processor.<br />`Config Param: SPARK_AVRO_POST_PROCESSOR_ENABLE` |
+---
+
+
+#### File-based Schema Provider Configs {#File-based-Schema-Provider-Configs}
+Configurations for file-based schema provider.
+
+
+
+[**Basic Configs**](#File-based-Schema-Provider-Configs-basic-configs)
+
+
+| Config Name | Default | Description |
+| -------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.source.schema.file](#hoodiestreamerschemaprovidersourceschemafile) | (N/A) | The schema of the source you are reading from<br />`Config Param: SOURCE_SCHEMA_FILE` |
+| [hoodie.streamer.schemaprovider.target.schema.file](#hoodiestreamerschemaprovidertargetschemafile) | (N/A) | The schema of the target you are writing to<br />`Config Param: TARGET_SCHEMA_FILE` |
+---
+
+
+#### Hive Schema Provider Configs {#Hive-Schema-Provider-Configs}
+Configurations for Hive schema provider.
+
+
+
+[**Advanced Configs**](#Hive-Schema-Provider-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.source.schema.hive.table](#hoodiestreamerschemaprovidersourceschemahivetable) | (N/A) | Hive table from where source schema can be fetched<br />`Config Param: SOURCE_SCHEMA_TABLE` |
+| [hoodie.streamer.schemaprovider.target.schema.hive.table](#hoodiestreamerschemaprovidertargetschemahivetable) | (N/A) | Hive table from where target schema can be fetched<br />`Config Param: TARGET_SCHEMA_TABLE` |
+| [hoodie.streamer.schemaprovider.source.schema.hive.database](#hoodiestreamerschemaprovidersourceschemahivedatabase) | default | Hive database from where source schema can be fetched<br />`Config Param: SOURCE_SCHEMA_DATABASE` |
+| [hoodie.streamer.schemaprovider.target.schema.hive.database](#hoodiestreamerschemaprovidertargetschemahivedatabase) | default | Hive database from where target schema can be fetched<br />`Config Param: TARGET_SCHEMA_DATABASE` |
+---
+
+
+#### JDBC-based Schema Provider Configs {#JDBC-based-Schema-Provider-Configs}
+Configurations for JDBC-based schema provider.
+
+
+
+[**Advanced Configs**](#JDBC-based-Schema-Provider-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.streamer.schemaprovider.source.schema.jdbc.connection.url](#hoodiestreamerschemaprovidersourceschemajdbcconnectionurl) | (N/A) | The JDBC URL to connect to. The source-specific connection properties may be specified in the URL. e.g., jdbc:postgresql://localhost/test?user=fred&password=secret<br />`Config Param: SOURCE_SCHEMA_JDBC_CONNECTION_URL` [...]
+| [hoodie.streamer.schemaprovider.source.schema.jdbc.dbtable](#hoodiestreamerschemaprovidersourceschemajdbcdbtable) | (N/A) | The table with the schema to reference e.g. test_database.test1_table or test1_table<br />`Config Param: SOURCE_SCHEMA_JDBC_DBTABLE` [...]
+| [hoodie.streamer.schemaprovider.source.schema.jdbc.driver.type](#hoodiestreamerschemaprovidersourceschemajdbcdrivertype) | (N/A) | The class name of the JDBC driver to use to connect to this URL. e.g. org.h2.Driver<br />`Config Param: SOURCE_SCHEMA_JDBC_DRIVER_TYPE` [...]
+| [hoodie.streamer.schemaprovider.source.schema.jdbc.nullable](#hoodiestreamerschemaprovidersourceschemajdbcnullable) | (N/A) | If true, all the columns are nullable.<br />`Config Param: SOURCE_SCHEMA_JDBC_NULLABLE` [...]
+| [hoodie.streamer.schemaprovider.source.schema.jdbc.password](#hoodiestreamerschemaprovidersourceschemajdbcpassword) | (N/A) | Password for the connection e.g. secret<br />`Config Param: SOURCE_SCHEMA_JDBC_PASSWORD` [...]
+| [hoodie.streamer.schemaprovider.source.schema.jdbc.timeout](#hoodiestreamerschemaprovidersourceschemajdbctimeout) | (N/A) | The number of seconds the driver will wait for a Statement object to execute. Zero means there is no limit. In the write path, this option depends on how JDBC drivers implement the API setQueryTimeout, e.g., the h2 JDBC driver checks the timeout of each query instead of an entire JDBC batch. It defaults to 0.<br />`Config Param: SOURCE_SCHEMA_JDBC_T [...]
+| [hoodie.streamer.schemaprovider.source.schema.jdbc.username](#hoodiestreamerschemaprovidersourceschemajdbcusername) | (N/A) | Username for the connection e.g. fred<br />`Config Param: SOURCE_SCHEMA_JDBC_USERNAME` [...]
+---
+
+
+#### JDBC-based Schema Provider Configs {#JDBC-based-Schema-Provider-Configs}
+Configurations for Proto schema provider.
+
+
+
+[**Advanced Configs**](#JDBC-based-Schema-Provider-Configs-advanced-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.streamer.schemaprovider.proto.class.name](#hoodiestreamerschemaproviderprotoclassname) | (N/A) | The Protobuf Message class used as the source for the schema.<br />`Config Param: PROTO_SCHEMA_CLASS_NAME`<br />`Since Version: 0.13.0` [...]
+| [hoodie.streamer.schemaprovider.proto.flatten.wrappers](#hoodiestreamerschemaproviderprotoflattenwrappers) | false | When set to true wrapped primitives like Int64Value are translated to a record with a single 'value' field. By default, the value is false and the wrapped primitives are treated as a nullable value<br />`Config Param: PROTO_SCHEMA_WRAPPED_PRIMITIVES_AS_RECORDS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.streamer.schemaprovider.proto.max.recursion.depth](#hoodiestreamerschemaproviderprotomaxrecursiondepth) | 5 | The max depth to unravel the Proto schema when translating into an Avro schema. Setting this depth allows the user to convert a schema that is recursive in proto into something that can be represented in their lake format like Parquet. After a given class has been seen N times within a single branch, the schema provider will create a record with a byte array t [...]
+| [hoodie.streamer.schemaprovider.proto.timestamps.as.records](#hoodiestreamerschemaproviderprototimestampsasrecords) | false | When set to true Timestamp fields are translated to a record with a seconds and nanos field. By default, the value is false and the timestamp is converted to a long with the timestamp-micros logical type<br />`Config Param: PROTO_SCHEMA_TIMESTAMPS_AS_RECORDS`<br />`Since Version: 0.13.0` [...]
+---
+
+
+#### Schema Post Processor Config Configs {#Schema-Post-Processor-Config-Configs}
+Configurations for Schema Post Processor
+
+
+
+[**Advanced Configs**](#Schema-Post-Processor-Config-Configs-advanced-configs)
+
+
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.schema_post_processor](#hoodiestreamerschemaproviderschema_post_processor) | (N/A) | The class name of the schema post processor.<br />`Config Param: SCHEMA_POST_PROCESSOR` |
+| [hoodie.streamer.schemaprovider.schema_post_processor.add.column.default](#hoodiestreamerschemaproviderschema_post_processoraddcolumndefault) | (N/A) | New column's default value<br />`Config Param: SCHEMA_POST_PROCESSOR_ADD_COLUMN_DEFAULT_PROP` |
+| [hoodie.streamer.schemaprovider.schema_post_processor.add.column.doc](#hoodiestreamerschemaproviderschema_post_processoraddcolumndoc) | (N/A) | Docs about new column<br />`Config Param: SCHEMA_POST_PROCESSOR_ADD_COLUMN_DOC_PROP` |
+| [hoodie.streamer.schemaprovider.schema_post_processor.add.column.name](#hoodiestreamerschemaproviderschema_post_processoraddcolumnname) | (N/A) | New column's name<br />`Config Param: SCHEMA_POST_PROCESSOR_ADD_COLUMN_NAME_PROP` |
+| [hoodie.streamer.schemaprovider.schema_post_processor.add.column.type](#hoodiestreamerschemaproviderschema_post_processoraddcolumntype) | (N/A) | New column's type<br />`Config Param: SCHEMA_POST_PROCESSOR_ADD_COLUMN_TYPE_PROP` |
+| [hoodie.streamer.schemaprovider.schema_post_processor.delete.columns](#hoodiestreamerschemaproviderschema_post_processordeletecolumns) | (N/A) | Columns to delete in the schema post processor.<br />`Config Param: DELETE_COLUMN_POST_PROCESSOR_COLUMN` |
+| [hoodie.streamer.schemaprovider.schema_post_processor.add.column.nullable](#hoodiestreamerschemaproviderschema_post_processoraddcolumnnullable) | true | New column's nullable<br />`Config Param: SCHEMA_POST_PROCESSOR_ADD_COLUMN_NULLABLE_PROP` |
+---
+
diff --git a/website/versioned_docs/version-0.15.1/cos_hoodie.md b/website/versioned_docs/version-0.15.1/cos_hoodie.md
new file mode 100644
index 000000000000..dfde6e8cff0a
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/cos_hoodie.md
@@ -0,0 +1,71 @@
+---
+title: Tencent Cloud
+keywords: [ hudi, hive, tencent, cos, spark, presto]
+summary: In this page, we go over how to configure Hudi with COS filesystem.
+last_modified_at: 2020-04-21T11:38:24-10:00
+---
+In this page, we explain how to get your Hudi spark job to store into Tencent Cloud COS.
+
+## Tencent Cloud COS configs
+
+There are two configurations required for Hudi-COS compatibility:
+
+- Adding Tencent Cloud COS Credentials for Hudi
+- Adding required Jars to classpath
+
+### Tencent Cloud COS Credentials
+
+Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the `fs.defaultFS` with your COS bucket name, replace `fs.cosn.userinfo.secretId` with your COS secret Id, replace `fs.cosn.userinfo.secretKey` with your COS key. Hudi should be able to read/write from the bucket.
+
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>cosn://bucketname</value>
+ <description>COS bucket name</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.userinfo.secretId</name>
+ <value>cos-secretId</value>
+ <description>Tencent Cloud Secret Id</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.userinfo.secretKey</name>
+ <value>cos-secretkey</value>
+ <description>Tencent Cloud Secret Key</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.bucket.region</name>
+ <value>ap-region</value>
+ <description>The region where the bucket is located.</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.bucket.endpoint_suffix</name>
+ <value>cos.endpoint.suffix</value>
+ <description>
+ COS endpoint to connect to.
+ For public cloud users, it is recommended not to set this option, and only the correct area field is required.
+ </description>
+ </property>
+
+ <property>
+ <name>fs.cosn.impl</name>
+ <value>org.apache.hadoop.fs.CosFileSystem</value>
+ <description>The implementation class of the CosN Filesystem.</description>
+ </property>
+
+ <property>
+ <name>fs.AbstractFileSystem.cosn.impl</name>
+ <value>org.apache.hadoop.fs.CosN</value>
+ <description>The implementation class of the CosN AbstractFileSystem.</description>
+ </property>
+
+```
+
+### Tencent Cloud COS Libs
+COS hadoop libraries to add to our classpath
+
+- org.apache.hadoop:hadoop-cos:2.8.5
diff --git a/website/versioned_docs/version-0.15.1/deployment.md b/website/versioned_docs/version-0.15.1/deployment.md
new file mode 100644
index 000000000000..bbff6c584415
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/deployment.md
@@ -0,0 +1,261 @@
+---
+title: Deployment
+keywords: [ hudi, administration, operation, devops, deployment]
+summary: This section offers an overview of tools available to operate an ecosystem of Hudi
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+
+This section provides all the help you need to deploy and operate Hudi tables at scale.
+Specifically, we will cover the following aspects.
+
+ - [Deployment Model](#deploying) : How various Hudi components are deployed and managed.
+ - [Upgrading Versions](#upgrading) : Picking up new releases of Hudi, guidelines and general best-practices.
+ - [Downgrading Versions](#downgrading) : Reverting back to an older version of Hudi
+ - [Migrating to Hudi](#migrating) : How to migrate your existing tables to Apache Hudi.
+
+## Deploying
+
+All in all, Hudi deploys with no long running servers or additional infrastructure cost to your data lake. In fact, Hudi pioneered this model of building a transactional distributed storage layer
+using existing infrastructure and its heartening to see other systems adopting similar approaches as well. Hudi writing is done via Spark jobs (Hudi Streamer or custom Spark datasource jobs), deployed per standard Apache Spark [recommendations](https://spark.apache.org/docs/latest/cluster-overview).
+Querying Hudi tables happens via libraries installed into Apache Hive, Apache Spark or PrestoDB and hence no additional infrastructure is necessary.
+
+A typical Hudi data ingestion can be achieved in 2 modes. In a single run mode, Hudi ingestion reads next batch of data, ingest them to Hudi table and exits. In continuous mode, Hudi ingestion runs as a long-running service executing ingestion in a loop.
+
+With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting delta files. Again, compaction can be performed in an asynchronous-mode by letting compaction run concurrently with ingestion or in a serial fashion with one after another.
+
+### Hudi Streamer
+
+[Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) is the standalone utility to incrementally pull upstream changes
+from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in two modes.
+
+To use Hudi Streamer in Spark, the `hudi-utilities-bundle` is required, by adding
+`--packages org.apache.hudi:hudi-utilities-bundle_2.11:0.13.0` to the `spark-submit` command. From 0.11.0 release, we start
+to provide a new `hudi-utilities-slim-bundle` which aims to exclude dependencies that can cause conflicts and compatibility
+issues with different versions of Spark. The `hudi-utilities-slim-bundle` should be used along with a Hudi Spark bundle
+corresponding to the Spark version used, e.g.,
+`--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:0.13.0,org.apache.hudi:hudi-spark3.1-bundle_2.12:0.13.0`,
+if using `hudi-utilities-bundle` solely in Spark encounters compatibility issues.
+
+ - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+
+Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
+
+```java
+[hoodie]$ spark-submit --packages org.apache.hudi:hudi-utilities-bundle_2.11:0.13.0 \
+ --master yarn \
+ --deploy-mode cluster \
+ --num-executors 10 \
+ --executor-memory 3g \
+ --driver-memory 6g \
+ --conf spark.driver.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_driver.hprof" \
+ --conf spark.executor.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_executor.hprof" \
+ --queue hadoop-platform-queue \
+ --conf spark.scheduler.mode=FAIR \
+ --conf spark.yarn.executor.memoryOverhead=1072 \
+ --conf spark.yarn.driver.memoryOverhead=2048 \
+ --conf spark.task.cpus=1 \
+ --conf spark.executor.cores=1 \
+ --conf spark.task.maxFailures=10 \
+ --conf spark.memory.fraction=0.4 \
+ --conf spark.rdd.compress=true \
+ --conf spark.kryoserializer.buffer.max=200m \
+ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+ --conf spark.memory.storageFraction=0.1 \
+ --conf spark.shuffle.service.enabled=true \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --conf spark.ui.port=5555 \
+ --conf spark.driver.maxResultSize=3g \
+ --conf spark.executor.heartbeatInterval=120s \
+ --conf spark.network.timeout=600s \
+ --conf spark.eventLog.overwrite=true \
+ --conf spark.eventLog.enabled=true \
+ --conf spark.eventLog.dir=hdfs:///user/spark/applicationHistory \
+ --conf spark.yarn.max.executor.failures=10 \
+ --conf spark.sql.catalogImplementation=hive \
+ --conf spark.sql.shuffle.partitions=100 \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+```
+
+ - **Continuous Mode** : Here, Hudi Streamer runs an infinite loop with each round performing one ingestion round as described in **Run Once Mode**. The frequency of data ingestion can be controlled by the configuration "--min-sync-interval-seconds". For Merge-On-Read tables, Compaction is run in asynchronous fashion concurrently with ingestion unless disabled by passing the flag "--disable-compaction". Every ingestion run triggers a compaction request asynchronously and this frequency [...]
+
+Here is an example invocation for reading from kafka topic in a continuous mode and writing to Merge On Read table type in a yarn cluster.
+
+```java
+[hoodie]$ spark-submit --packages org.apache.hudi:hudi-utilities-bundle_2.11:0.13.0 \
+ --master yarn \
+ --deploy-mode cluster \
+ --num-executors 10 \
+ --executor-memory 3g \
+ --driver-memory 6g \
+ --conf spark.driver.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_driver.hprof" \
+ --conf spark.executor.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_executor.hprof" \
+ --queue hadoop-platform-queue \
+ --conf spark.scheduler.mode=FAIR \
+ --conf spark.yarn.executor.memoryOverhead=1072 \
+ --conf spark.yarn.driver.memoryOverhead=2048 \
+ --conf spark.task.cpus=1 \
+ --conf spark.executor.cores=1 \
+ --conf spark.task.maxFailures=10 \
+ --conf spark.memory.fraction=0.4 \
+ --conf spark.rdd.compress=true \
+ --conf spark.kryoserializer.buffer.max=200m \
+ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+ --conf spark.memory.storageFraction=0.1 \
+ --conf spark.shuffle.service.enabled=true \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --conf spark.ui.port=5555 \
+ --conf spark.driver.maxResultSize=3g \
+ --conf spark.executor.heartbeatInterval=120s \
+ --conf spark.network.timeout=600s \
+ --conf spark.eventLog.overwrite=true \
+ --conf spark.eventLog.enabled=true \
+ --conf spark.eventLog.dir=hdfs:///user/spark/applicationHistory \
+ --conf spark.yarn.max.executor.failures=10 \
+ --conf spark.sql.catalogImplementation=hive \
+ --conf spark.sql.shuffle.partitions=100 \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
+ --continuous
+```
+
+### Spark Datasource Writer Jobs
+
+As described in [Batch Writes](writing_data.md#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+
+Here is an example invocation using spark datasource
+
+```java
+inputDF.write()
+ .format("org.apache.hudi")
+ .options(clientOpts) // any of the Hudi client opts can be passed in as well
+ .option("hoodie.datasource.write.recordkey.field", "_row_key")
+ .option("hoodie.datasource.write.partitionpath.field", "partition")
+ .option("hoodie.datasource.write.precombine.field"(), "timestamp")
+ .option("hoodie.table.name", tableName)
+ .mode(SaveMode.Append)
+ .save(basePath);
+```
+
+## Upgrading
+
+New Hudi releases are listed on the [releases page](/releases/download), with detailed notes which list all the changes, with highlights in each release.
+At the end of the day, Hudi is a storage system and with that comes a lot of responsibilities, which we take seriously.
+
+As general guidelines,
+
+ - We strive to keep all changes backwards compatible (i.e new code can read old data/timeline files) and when we cannot, we will provide upgrade/downgrade tools via the CLI
+ - We cannot always guarantee forward compatibility (i.e old code being able to read data/timeline files written by a greater version). This is generally the norm, since no new features can be built otherwise.
+ However any large such changes, will be turned off by default, for smooth transition to newer release. After a few releases and once enough users deem the feature stable in production, we will flip the defaults in a subsequent release.
+ - Always upgrade the query bundles (mr-bundle, presto-bundle, spark-bundle) first and then upgrade the writers (Hudi Streamer, spark jobs using datasource). This often provides the best experience and it's easy to fix
+ any issues by rolling forward/back the writer code (which typically you might have more control over)
+ - With large, feature rich releases we recommend migrating slowly, by first testing in staging environments and running your own tests. Upgrading Hudi is no different than upgrading any database system.
+
+Note that release notes can override this information with specific instructions, applicable on case-by-case basis.
+
+## Downgrading
+
+Upgrade is automatic whenever a new Hudi version is used whereas downgrade is a manual step. We need to use the Hudi
+CLI to downgrade a table from a higher version to lower version. Let's consider an example where we create a table using
+0.12.0, upgrade it to 0.13.0 and then downgrade it via Hudi CLI.
+
+Launch spark shell with Hudi 0.11.0 version.
+```shell
+spark-shell \
+ --packages org.apache.hudi:hudi-spark3.2-bundle_2.12:0.11.0 \
+ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
+ --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
+ --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
+```
+
+Create a hudi table by using the scala script below.
+```scala
+import org.apache.hudi.QuickstartUtils._
+import scala.collection.JavaConversions._
+import org.apache.spark.sql.SaveMode._
+import org.apache.hudi.DataSourceReadOptions._
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.config.HoodieWriteConfig._
+import org.apache.hudi.common.model.HoodieRecord
+import org.apache.hudi.common.table.timeline.HoodieTimeline
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.HoodieDataSourceHelpers
+
+val dataGen = new DataGenerator
+val tableType = MOR_TABLE_TYPE_OPT_VAL
+val basePath = "file:///tmp/hudi_table"
+val tableName = "hudi_table"
+
+val inserts = convertToStringList(dataGen.generateInserts(100)).toList
+val insertDf = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
+insertDf.write.format("hudi").
+ options(getQuickstartWriteConfigs).
+ option("hoodie.datasource.write.precombine.field", "ts").
+ option("hoodie.datasource.write.recordkey.field", "uuid").
+ option("hoodie.datasource.write.partitionpath.field", "partitionpath").
+ option("hoodie.table.name", tableName).
+ option("hoodie.datasource.write.operation", "insert").
+ mode(Append).
+ save(basePath)
+```
+
+You will see an entry for table version in hoodie.properties which states the table version is 4.
+```shell
+bash$ cat /tmp/hudi_table/.hoodie/hoodie.properties | grep hoodie.table.version
+hoodie.table.version=4
+```
+
+Launch a new spark shell using version 0.13.0 and append to the same table using the script above. Note the upgrade
+happens automatically with the new version.
+```shell
+spark-shell \
+ --packages org.apache.hudi:hudi-spark3.2-bundle_2.12:0.13.1 \
+ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
+ --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
+ --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
+```
+
+After upgrade, the table version is updated to 5.
+```shell
+bash$ cat /tmp/hudi_table/.hoodie/hoodie.properties | grep hoodie.table.version
+hoodie.table.version=5
+```
+
+Lets try downgrading the table back to version 4. For downgrading we will need to use Hudi CLI and execute downgrade.
+For more details on downgrade, please refer documentation [here](cli.md#upgrade-and-downgrade-table).
+```shell
+connect --path /tmp/hudi_table
+downgrade table --toVersion 4
+```
+
+After downgrade, the table version is updated to 4.
+```shell
+bash$ cat /tmp/hudi_table/.hoodie/hoodie.properties | grep hoodie.table.version
+hoodie.table.version=4
+```
+
+## Migrating
+
+Currently migrating to Hudi can be done using two approaches
+
+- **Convert newer partitions to Hudi** : This model is suitable for large event tables (e.g: click streams, ad impressions), which also typically receive writes for the last few days alone. You can convert the last
+ N partitions to Hudi and proceed writing as if it were a Hudi table to begin with. The Hudi query side code is able to correctly handle both hudi and non-hudi data partitions.
+- **Full conversion to Hudi** : This model is suitable if you are currently bulk/full loading the table few times a day (e.g database ingestion). The full conversion of Hudi is simply a one-time step (akin to 1 run of your existing job),
+ which moves all of the data into the Hudi format and provides the ability to incrementally update for future writes.
+
+For more details, refer to the detailed [migration guide](migration_guide.md). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
diff --git a/website/versioned_docs/version-0.15.1/disaster_recovery.md b/website/versioned_docs/version-0.15.1/disaster_recovery.md
new file mode 100644
index 000000000000..a264b7d36155
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/disaster_recovery.md
@@ -0,0 +1,309 @@
+---
+title: Disaster Recovery
+toc: true
+---
+
+Disaster Recovery is very much mission-critical for any software. Especially when it comes to data systems, the impact could be very serious
+leading to delay in business decisions or even wrong business decisions at times. Apache Hudi has two operations to assist you in recovering
+data from a previous state: `savepoint` and `restore`.
+
+## Savepoint
+
+As the name suggest, `savepoint` saves the table as of the commit time, so that it lets you restore the table to this
+savepoint at a later point in time if need be. Care is taken to ensure cleaner will not clean up any files that are savepointed.
+On similar lines, savepoint cannot be triggered on a commit that is already cleaned up. In simpler terms, this is synonymous
+to taking a backup, just that we don't make a new copy of the table, but just save the state of the table elegantly so that
+we can restore it later when in need.
+
+## Restore
+
+This operation lets you restore your table to one of the savepoint commit. This operation cannot be undone (or reversed) and so care
+should be taken before doing a restore. Hudi will delete all data files and commit files (timeline files) greater than the
+savepoint commit to which the table is being restored. You should pause all writes to the table when performing
+a restore since they are likely to fail while the restore is in progress. Also, reads could also fail since snapshot queries
+will be hitting latest files which has high possibility of getting deleted with restore.
+
+## Runbook
+
+Savepoint and restore can only be triggered from `hudi-cli`. Let's walk through an example of how one can take savepoint
+and later restore the state of the table.
+
+Let's create a hudi table via `spark-shell` and trigger a batch of inserts.
+
+```scala
+import org.apache.hudi.QuickstartUtils._
+import scala.collection.JavaConversions._
+import org.apache.spark.sql.SaveMode._
+import org.apache.hudi.DataSourceReadOptions._
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.config.HoodieWriteConfig._
+
+val tableName = "hudi_trips_cow"
+val basePath = "file:///tmp/hudi_trips_cow"
+val dataGen = new DataGenerator
+
+val inserts = convertToStringList(dataGen.generateInserts(10))
+val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
+df.write.format("hudi").
+ options(getQuickstartWriteConfigs).
+ option("hoodie.datasource.write.precombine.field", "ts").
+ option("hoodie.datasource.write.recordkey.field", "uuid").
+ option("hoodie.datasource.write.partitionpath.field", "partitionpath").
+ option("hoodie.table.name", tableName).
+ mode(Overwrite).
+ save(basePath)
+```
+
+Let's add four more batches of inserts.
+```scala
+for (_ <- 1 to 4) {
+ val inserts = convertToStringList(dataGen.generateInserts(10))
+ val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
+ df.write.format("hudi").
+ options(getQuickstartWriteConfigs).
+ option("hoodie.datasource.write.precombine.field", "ts").
+ option("hoodie.datasource.write.recordkey.field", "uuid").
+ option("hoodie.datasource.write.partitionpath.field", "partitionpath").
+ option("hoodie.table.name", tableName).
+ mode(Append).
+ save(basePath)
+}
+```
+
+Total record count should be 50.
+```scala
+val tripsSnapshotDF = spark.
+ read.
+ format("hudi").
+ load(basePath)
+tripsSnapshotDF.createOrReplaceTempView("hudi_trips_snapshot")
+
+spark.sql("select count(partitionpath, uuid) from hudi_trips_snapshot").show()
+
++--------------------------+
+|count(partitionpath, uuid)|
++--------------------------+
+| 50|
++--------------------------+
+```
+Let's take a look at the timeline after 5 batches of inserts.
+```shell
+ls -ltr /tmp/hudi_trips_cow/.hoodie
+total 128
+drwxr-xr-x 2 nsb wheel 64 Jan 28 16:00 archived
+-rw-r--r-- 1 nsb wheel 546 Jan 28 16:00 hoodie.properties
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:00 20220128160040171.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:00 20220128160040171.inflight
+-rw-r--r-- 1 nsb wheel 4374 Jan 28 16:00 20220128160040171.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:01 20220128160124637.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:01 20220128160124637.inflight
+-rw-r--r-- 1 nsb wheel 4414 Jan 28 16:01 20220128160124637.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160226172.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160226172.inflight
+-rw-r--r-- 1 nsb wheel 4427 Jan 28 16:02 20220128160226172.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160229636.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160229636.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:02 20220128160229636.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160245447.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160245447.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:02 20220128160245447.commit
+```
+
+Let's trigger a savepoint as of the latest commit. Savepoint can only be done via `hudi-cli`.
+
+```sh
+./hudi-cli.sh
+
+connect --path /tmp/hudi_trips_cow/
+commits show
+set --conf SPARK_HOME=<SPARK_HOME>
+savepoint create --commit 20220128160245447 --sparkMaster local[2]
+```
+
+:::note NOTE:
+Make sure you replace 20220128160245447 with the latest commit in your table.
+:::
+
+Let's check the timeline after savepoint.
+```shell
+ls -ltr /tmp/hudi_trips_cow/.hoodie
+total 136
+drwxr-xr-x 2 nsb wheel 64 Jan 28 16:00 archived
+-rw-r--r-- 1 nsb wheel 546 Jan 28 16:00 hoodie.properties
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:00 20220128160040171.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:00 20220128160040171.inflight
+-rw-r--r-- 1 nsb wheel 4374 Jan 28 16:00 20220128160040171.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:01 20220128160124637.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:01 20220128160124637.inflight
+-rw-r--r-- 1 nsb wheel 4414 Jan 28 16:01 20220128160124637.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160226172.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160226172.inflight
+-rw-r--r-- 1 nsb wheel 4427 Jan 28 16:02 20220128160226172.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160229636.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160229636.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:02 20220128160229636.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160245447.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160245447.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:02 20220128160245447.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:05 20220128160245447.savepoint.inflight
+-rw-r--r-- 1 nsb wheel 1168 Jan 28 16:05 20220128160245447.savepoint
+```
+
+You could notice that savepoint meta files are added which keeps track of the files that are part of the latest table snapshot.
+
+Now, let's continue adding three more batches of inserts.
+
+```scala
+for (_ <- 1 to 3) {
+ val inserts = convertToStringList(dataGen.generateInserts(10))
+ val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
+ df.write.format("hudi").
+ options(getQuickstartWriteConfigs).
+ option("hoodie.datasource.write.precombine.field", "ts").
+ option("hoodie.datasource.write.recordkey.field", "uuid").
+ option("hoodie.datasource.write.partitionpath.field", "partitionpath").
+ option("hoodie.table.name", tableName).
+ mode(Append).
+ save(basePath)
+}
+```
+
+Total record count will be 80 since we have done 8 batches in total. (5 until savepoint and 3 after savepoint)
+```scala
+val tripsSnapshotDF = spark.
+ read.
+ format("hudi").
+ load(basePath)
+tripsSnapshotDF.createOrReplaceTempView("hudi_trips_snapshot")
+
+spark.sql("select count(partitionpath, uuid) from hudi_trips_snapshot").show()
++--------------------------+
+|count(partitionpath, uuid)|
++--------------------------+
+| 80|
++--------------------------+
+```
+
+Let's say something bad happened, and you want to restore your table to an older snapshot. As we called out earlier, we can
+trigger restore only from `hudi-cli`. And do remember to bring down all of your writer processes while doing a restore.
+
+Let's checkout timeline once, before we trigger the restore.
+```shell
+ls -ltr /tmp/hudi_trips_cow/.hoodie
+total 208
+drwxr-xr-x 2 nsb wheel 64 Jan 28 16:00 archived
+-rw-r--r-- 1 nsb wheel 546 Jan 28 16:00 hoodie.properties
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:00 20220128160040171.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:00 20220128160040171.inflight
+-rw-r--r-- 1 nsb wheel 4374 Jan 28 16:00 20220128160040171.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:01 20220128160124637.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:01 20220128160124637.inflight
+-rw-r--r-- 1 nsb wheel 4414 Jan 28 16:01 20220128160124637.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160226172.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160226172.inflight
+-rw-r--r-- 1 nsb wheel 4427 Jan 28 16:02 20220128160226172.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160229636.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160229636.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:02 20220128160229636.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160245447.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160245447.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:02 20220128160245447.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:05 20220128160245447.savepoint.inflight
+-rw-r--r-- 1 nsb wheel 1168 Jan 28 16:05 20220128160245447.savepoint
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:06 20220128160620557.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:06 20220128160620557.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:06 20220128160620557.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:06 20220128160627501.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:06 20220128160627501.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:06 20220128160627501.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:06 20220128160630785.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:06 20220128160630785.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:06 20220128160630785.commit
+```
+
+If you are continuing in the same `hudi-cli` session, you can just execute `refresh` so that table state gets refreshed to
+its latest state. If not, connect to the table again.
+
+```shell
+./hudi-cli.sh
+
+connect --path /tmp/hudi_trips_cow/
+commits show
+set --conf SPARK_HOME=<SPARK_HOME>
+savepoints show
+╔═══════════════════╗
+║ SavepointTime ║
+╠═══════════════════╣
+║ 20220128160245447 ║
+╚═══════════════════╝
+savepoint rollback --savepoint 20220128160245447 --sparkMaster local[2]
+```
+
+:::note NOTE:
+Make sure you replace 20220128160245447 with the latest savepoint in your table.
+:::
+
+Hudi table should have been restored to the savepointed commit 20220128160245447. Both data files and timeline files should have
+been deleted.
+```shell
+ls -ltr /tmp/hudi_trips_cow/.hoodie
+total 152
+drwxr-xr-x 2 nsb wheel 64 Jan 28 16:00 archived
+-rw-r--r-- 1 nsb wheel 546 Jan 28 16:00 hoodie.properties
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:00 20220128160040171.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:00 20220128160040171.inflight
+-rw-r--r-- 1 nsb wheel 4374 Jan 28 16:00 20220128160040171.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:01 20220128160124637.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:01 20220128160124637.inflight
+-rw-r--r-- 1 nsb wheel 4414 Jan 28 16:01 20220128160124637.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160226172.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160226172.inflight
+-rw-r--r-- 1 nsb wheel 4427 Jan 28 16:02 20220128160226172.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160229636.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160229636.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:02 20220128160229636.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:02 20220128160245447.commit.requested
+-rw-r--r-- 1 nsb wheel 2594 Jan 28 16:02 20220128160245447.inflight
+-rw-r--r-- 1 nsb wheel 4428 Jan 28 16:02 20220128160245447.commit
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:05 20220128160245447.savepoint.inflight
+-rw-r--r-- 1 nsb wheel 1168 Jan 28 16:05 20220128160245447.savepoint
+-rw-r--r-- 1 nsb wheel 0 Jan 28 16:07 20220128160732437.restore.inflight
+-rw-r--r-- 1 nsb wheel 4152 Jan 28 16:07 20220128160732437.restore
+```
+
+Let's check the total record count in the table. Should match the records we had, just before we triggered the savepoint.
+```scala
+val tripsSnapshotDF = spark.
+ read.
+ format("hudi").
+ load(basePath)
+tripsSnapshotDF.createOrReplaceTempView("hudi_trips_snapshot")
+
+spark.sql("select count(partitionpath, uuid) from hudi_trips_snapshot").show()
++--------------------------+
+|count(partitionpath, uuid)|
++--------------------------+
+| 50|
++--------------------------+
+```
+
+As you could see, entire table state is restored back to the commit which was savepointed. Users can choose to trigger savepoint
+at regular cadence and keep deleting older savepoints when new ones are created. `hudi-cli` has a command `savepoint delete`
+to assist in deleting a savepoint. Please do remember that cleaner may not clean the files that are savepointed. And so users
+should ensure they delete the savepoints from time to time. If not, the storage reclamation may not happen.
+
+Note: Savepoint and restore for MOR table is available only from 0.11.
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Use Glue 4.0 to take regular save points for your Hudi tables for backup or disaster Recovery](https://www.youtube.com/watch?v=VgIMPSK7rFAa)
+* [How to Rollback to Previous Checkpoint during Disaster in Apache Hudi using Glue 4.0 Demo](https://www.youtube.com/watch?v=Vi25q4vzogs)
+
+
+
+
+
+
+
+
diff --git a/website/versioned_docs/version-0.15.1/docker_demo.md b/website/versioned_docs/version-0.15.1/docker_demo.md
new file mode 100644
index 000000000000..6e76d1cfb37b
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/docker_demo.md
@@ -0,0 +1,1526 @@
+---
+title: Docker Demo
+keywords: [ hudi, docker, demo]
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## A Demo using Docker containers
+
+Let's use a real world example to see how Hudi works end to end. For this purpose, a self contained
+data infrastructure is brought up in a local Docker cluster within your computer. It requires the
+Hudi repo to have been cloned locally.
+
+The steps have been tested on a Mac laptop
+
+### Prerequisites
+
+ * Clone the [Hudi repository](https://github.com/apache/hudi) to your local machine.
+ * Docker Setup : For Mac, Please follow the steps as defined in [Install Docker Desktop on Mac](https://docs.docker.com/desktop/install/mac-install/). For running Spark-SQL queries, please ensure atleast 6 GB and 4 CPUs are allocated to Docker (See Docker -> Preferences -> Advanced). Otherwise, spark-SQL queries could be killed because of memory issues.
+ * kcat : A command-line utility to publish/consume from kafka topics. Use `brew install kcat` to install kcat.
+ * /etc/hosts : The demo references many services running in container by the hostname. Add the following settings to /etc/hosts
+
+ ```java
+ 127.0.0.1 adhoc-1
+ 127.0.0.1 adhoc-2
+ 127.0.0.1 namenode
+ 127.0.0.1 datanode1
+ 127.0.0.1 hiveserver
+ 127.0.0.1 hivemetastore
+ 127.0.0.1 kafkabroker
+ 127.0.0.1 sparkmaster
+ 127.0.0.1 zookeeper
+ ```
+ * Java : Java SE Development Kit 8.
+ * Maven : A build automation tool for Java projects.
+ * jq : A lightweight and flexible command-line JSON processor. Use `brew install jq` to install jq.
+
+Also, this has not been tested on some environments like Docker on Windows.
+
+
+## Setting up Docker Cluster
+
+
+### Build Hudi
+
+The first step is to build Hudi. **Note** This step builds Hudi on default supported scala version - 2.11.
+
+NOTE: Make sure you've cloned the [Hudi repository](https://github.com/apache/hudi) first.
+
+```java
+cd <HUDI_WORKSPACE>
+mvn clean package -Pintegration-tests -DskipTests -Dscala-2.11
+```
+
+### Bringing up Demo Cluster
+
+The next step is to run the Docker compose script and setup configs for bringing up the cluster. These files are in the [Hudi repository](https://github.com/apache/hudi) which you should already have locally on your machine from the previous steps.
+
+This should pull the Docker images from Docker hub and setup the Docker cluster.
+
+<Tabs
+defaultValue="default"
+values={[
+{ label: 'Default', value: 'default', },
+{ label: 'Mac AArch64', value: 'm1', },
+]}
+>
+<TabItem value="default">
+
+```java
+cd docker
+./setup_demo.sh
+....
+....
+....
+[+] Running 10/13
+⠿ Container zookeeper Removed 8.6s
+⠿ Container datanode1 Removed 18.3s
+⠿ Container trino-worker-1 Removed 50.7s
+⠿ Container spark-worker-1 Removed 16.7s
+⠿ Container adhoc-2 Removed 16.9s
+⠿ Container graphite Removed 16.9s
+⠿ Container kafkabroker Removed 14.1s
+⠿ Container adhoc-1 Removed 14.1s
+⠿ Container presto-worker-1 Removed 11.9s
+⠿ Container presto-coordinator-1 Removed 34.6s
+.......
+......
+[+] Running 17/17
+⠿ adhoc-1 Pulled 2.9s
+⠿ graphite Pulled 2.8s
+⠿ spark-worker-1 Pulled 3.0s
+⠿ kafka Pulled 2.9s
+⠿ datanode1 Pulled 2.9s
+⠿ hivemetastore Pulled 2.9s
+⠿ hiveserver Pulled 3.0s
+⠿ hive-metastore-postgresql Pulled 2.8s
+⠿ presto-coordinator-1 Pulled 2.9s
+⠿ namenode Pulled 2.9s
+⠿ trino-worker-1 Pulled 2.9s
+⠿ sparkmaster Pulled 2.9s
+⠿ presto-worker-1 Pulled 2.9s
+⠿ zookeeper Pulled 2.8s
+⠿ adhoc-2 Pulled 2.9s
+⠿ historyserver Pulled 2.9s
+⠿ trino-coordinator-1 Pulled 2.9s
+[+] Running 17/17
+⠿ Container zookeeper Started 41.0s
+⠿ Container kafkabroker Started 41.7s
+⠿ Container graphite Started 41.5s
+⠿ Container hive-metastore-postgresql Running 0.0s
+⠿ Container namenode Running 0.0s
+⠿ Container hivemetastore Running 0.0s
+⠿ Container trino-coordinator-1 Runni... 0.0s
+⠿ Container presto-coordinator-1 Star... 42.1s
+⠿ Container historyserver Started 41.0s
+⠿ Container datanode1 Started 49.9s
+⠿ Container hiveserver Running 0.0s
+⠿ Container trino-worker-1 Started 42.1s
+⠿ Container sparkmaster Started 41.9s
+⠿ Container spark-worker-1 Started 50.2s
+⠿ Container adhoc-2 Started 38.5s
+⠿ Container adhoc-1 Started 38.5s
+⠿ Container presto-worker-1 Started 38.4s
+Copying spark default config and setting up configs
+Copying spark default config and setting up configs
+$ docker ps
+```
+
+</TabItem>
+<TabItem value="m1">
+
+:::note Please note the following for Mac AArch64 users
+<ul>
+ <li> The demo must be built and run using the master branch. We currently plan to include support starting with the
+ 0.13.0 release. </li>
+ <li> Presto and Trino are not currently supported in the demo. </li>
+</ul>
+:::
+
+```java
+cd docker
+./setup_demo.sh --mac-aarch64
+.......
+......
+[+] Running 12/12
+⠿ adhoc-1 Pulled 2.9s
+⠿ spark-worker-1 Pulled 3.0s
+⠿ kafka Pulled 2.9s
+⠿ datanode1 Pulled 2.9s
+⠿ hivemetastore Pulled 2.9s
+⠿ hiveserver Pulled 3.0s
+⠿ hive-metastore-postgresql Pulled 2.8s
+⠿ namenode Pulled 2.9s
+⠿ sparkmaster Pulled 2.9s
+⠿ zookeeper Pulled 2.8s
+⠿ adhoc-2 Pulled 2.9s
+⠿ historyserver Pulled 2.9s
+[+] Running 12/12
+⠿ Container zookeeper Started 41.0s
+⠿ Container kafkabroker Started 41.7s
+⠿ Container hive-metastore-postgresql Running 0.0s
+⠿ Container namenode Running 0.0s
+⠿ Container hivemetastore Running 0.0s
+⠿ Container historyserver Started 41.0s
+⠿ Container datanode1 Started 49.9s
+⠿ Container hiveserver Running 0.0s
+⠿ Container sparkmaster Started 41.9s
+⠿ Container spark-worker-1 Started 50.2s
+⠿ Container adhoc-2 Started 38.5s
+⠿ Container adhoc-1 Started 38.5s
+Copying spark default config and setting up configs
+Copying spark default config and setting up configs
+$ docker ps
+```
+</TabItem>
+
+</Tabs
+>
+
+At this point, the Docker cluster will be up and running. The demo cluster brings up the following services
+
+ * HDFS Services (NameNode, DataNode)
+ * Spark Master and Worker
+ * Hive Services (Metastore, HiveServer2 along with PostgresDB)
+ * Kafka Broker and a Zookeeper Node (Kafka will be used as upstream source for the demo)
+ * Containers for Presto setup (Presto coordinator and worker)
+ * Containers for Trino setup (Trino coordinator and worker)
+ * Adhoc containers to run Hudi/Hive CLI commands
+
+## Demo
+
+Stock Tracker data will be used to showcase different Hudi query types and the effects of Compaction.
+
+Take a look at the directory `docker/demo/data`. There are 2 batches of stock data - each at 1 minute granularity.
+The first batch contains stocker tracker data for some stock symbols during the first hour of trading window
+(9:30 a.m to 10:30 a.m). The second batch contains tracker data for next 30 mins (10:30 - 11 a.m). Hudi will
+be used to ingest these batches to a table which will contain the latest stock tracker data at hour level granularity.
+The batches are windowed intentionally so that the second batch contains updates to some of the rows in the first batch.
+
+### Step 1 : Publish the first batch to Kafka
+
+Upload the first batch to Kafka topic 'stock ticks'
+
+`cat docker/demo/data/batch_1.json | kcat -b kafkabroker -t stock_ticks -P`
+
+To check if the new topic shows up, use
+```java
+kcat -b kafkabroker -L -J | jq .
+{
+ "originating_broker": {
+ "id": 1001,
+ "name": "kafkabroker:9092/1001"
+ },
+ "query": {
+ "topic": "*"
+ },
+ "brokers": [
+ {
+ "id": 1001,
+ "name": "kafkabroker:9092"
+ }
+ ],
+ "topics": [
+ {
+ "topic": "stock_ticks",
+ "partitions": [
+ {
+ "partition": 0,
+ "leader": 1001,
+ "replicas": [
+ {
+ "id": 1001
+ }
+ ],
+ "isrs": [
+ {
+ "id": 1001
+ }
+ ]
+ }
+ ]
+ }
+ ]
+}
+```
+
+### Step 2: Incrementally ingest data from Kafka topic
+
+Hudi comes with a tool named Hudi Streamer. This tool can connect to variety of data sources (including Kafka) to
+pull changes and apply to Hudi table using upsert/insert primitives. Here, we will use the tool to download
+json data from kafka topic and ingest to both COW and MOR tables we initialized in the previous step. This tool
+automatically initializes the tables in the file-system if they do not exist yet.
+
+```java
+docker exec -it adhoc-2 /bin/bash
+
+# Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_cow table in HDFS
+spark-submit \
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
+ --table-type COPY_ON_WRITE \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_cow \
+ --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+
+# Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_mor table in HDFS
+spark-submit \
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
+ --disable-compaction
+
+# As part of the setup (Look at setup_demo.sh), the configs needed for Hudi Streamer is uploaded to HDFS. The configs
+# contain mostly Kafa connectivity settings, the avro-schema to be used for ingesting along with key and partitioning fields.
+
+exit
+```
+
+You can use HDFS web-browser to look at the tables
+`http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow`.
+
+You can explore the new partition folder created in the table along with a "commit" / "deltacommit"
+file under .hoodie which signals a successful commit.
+
+There will be a similar setup when you browse the MOR table
+`http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_mor`
+
+
+### Step 3: Sync with Hive
+
+At this step, the tables are available in HDFS. We need to sync with Hive to create new Hive tables and add partitions
+inorder to run Hive queries against those tables.
+
+```java
+docker exec -it adhoc-2 /bin/bash
+
+# This command takes in HiveServer URL and COW Hudi table location in HDFS and sync the HDFS state to Hive
+/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
+ --jdbc-url jdbc:hive2://hiveserver:10000 \
+ --user hive \
+ --pass hive \
+ --partitioned-by dt \
+ --base-path /user/hive/warehouse/stock_ticks_cow \
+ --database default \
+ --table stock_ticks_cow \
+ --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
+.....
+2020-01-25 19:51:28,953 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_cow
+.....
+
+# Now run hive-sync for the second data-set in HDFS using Merge-On-Read (MOR table type)
+/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
+ --jdbc-url jdbc:hive2://hiveserver:10000 \
+ --user hive \
+ --pass hive \
+ --partitioned-by dt \
+ --base-path /user/hive/warehouse/stock_ticks_mor \
+ --database default \
+ --table stock_ticks_mor \
+ --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
+...
+2020-01-25 19:51:51,066 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_mor_ro
+...
+2020-01-25 19:51:51,569 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_mor_rt
+....
+
+exit
+```
+After executing the above command, you will notice
+
+1. A hive table named `stock_ticks_cow` created which supports Snapshot and Incremental queries on Copy On Write table.
+2. Two new tables `stock_ticks_mor_rt` and `stock_ticks_mor_ro` created for the Merge On Read table. The former
+supports Snapshot and Incremental queries (providing near-real time data) while the later supports ReadOptimized queries.
+
+
+### Step 4 (a): Run Hive Queries
+
+Run a hive query to find the latest timestamp ingested for stock symbol 'GOOG'. You will notice that both snapshot
+(for both COW and MOR _rt table) and read-optimized queries (for MOR _ro table) give the same value "10:29 a.m" as Hudi create a
+parquet file for the first batch of data.
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 \
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
+ --hiveconf hive.stats.autogather=false
+
+# List Tables
+0: jdbc:hive2://hiveserver:10000> show tables;
++---------------------+--+
+| tab_name |
++---------------------+--+
+| stock_ticks_cow |
+| stock_ticks_mor_ro |
+| stock_ticks_mor_rt |
++---------------------+--+
+3 rows selected (1.199 seconds)
+0: jdbc:hive2://hiveserver:10000>
+
+
+# Look at partitions that were added
+0: jdbc:hive2://hiveserver:10000> show partitions stock_ticks_mor_rt;
++----------------+--+
+| partition |
++----------------+--+
+| dt=2018-08-31 |
++----------------+--+
+1 row selected (0.24 seconds)
+
+
+# COPY-ON-WRITE Queries:
+=========================
+
+
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+
+Now, run a projection query:
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924221953 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924221953 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against M-O-R table. Lets look at both
+ReadOptimized and Snapshot(realtime data) queries supported by M-O-R table
+
+# Run ReadOptimized Query. Notice that the latest timestamp is 10:29
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (6.326 seconds)
+
+
+# Run Snapshot Query. Notice that the latest timestamp is again 10:29
+
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (1.606 seconds)
+
+
+# Run Read Optimized and Snapshot project queries
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+exit
+```
+
+### Step 4 (b): Run Spark-SQL Queries
+Hudi support Spark as query processor just like Hive. Here are the same hive queries
+running in spark-sql
+
+```java
+docker exec -it adhoc-1 /bin/bash
+$SPARK_INSTALL/bin/spark-shell \
+ --jars $HUDI_SPARK_BUNDLE \
+ --master local[2] \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --deploy-mode client \
+ --driver-memory 1G \
+ --executor-memory 3G \
+ --num-executors 1
+...
+
+Welcome to
+ ____ __
+ / __/__ ___ _____/ /__
+ _\ \/ _ \/ _ `/ __/ '_/
+ /___/ .__/\_,_/_/ /_/\_\ version 2.4.4
+ /_/
+
+Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
+Type in expressions to have them evaluated.
+Type :help for more information.
+
+scala> spark.sql("show tables").show(100, false)
++--------+------------------+-----------+
+|database|tableName |isTemporary|
++--------+------------------+-----------+
+|default |stock_ticks_cow |false |
+|default |stock_ticks_mor_ro|false |
+|default |stock_ticks_mor_rt|false |
++--------+------------------+-----------+
+
+# Copy-On-Write Table
+
+## Run max timestamp query against COW table
+
+scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
+[Stage 0:> (0 + 1) / 1]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
+SLF4J: Defaulting to no-operation (NOP) logger implementation
+SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:29:00|
++------+-------------------+
+
+## Projection Query
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG'").show(100, false)
++-------------------+------+-------------------+------+---------+--------+
+|_hoodie_commit_time|symbol|ts |volume|open |close |
++-------------------+------+-------------------+------+---------+--------+
+|20180924221953 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
+|20180924221953 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
++-------------------+------+-------------------+------+---------+--------+
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against M-O-R table. Lets look at both
+ReadOptimized and Snapshot queries supported by M-O-R table
+
+# Run ReadOptimized Query. Notice that the latest timestamp is 10:29
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:29:00|
++------+-------------------+
+
+
+# Run Snapshot Query. Notice that the latest timestamp is again 10:29
+
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:29:00|
++------+-------------------+
+
+# Run Read Optimized and Snapshot project queries
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG'").show(100, false)
++-------------------+------+-------------------+------+---------+--------+
+|_hoodie_commit_time|symbol|ts |volume|open |close |
++-------------------+------+-------------------+------+---------+--------+
+|20180924222155 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
+|20180924222155 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
++-------------------+------+-------------------+------+---------+--------+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
++-------------------+------+-------------------+------+---------+--------+
+|_hoodie_commit_time|symbol|ts |volume|open |close |
++-------------------+------+-------------------+------+---------+--------+
+|20180924222155 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
+|20180924222155 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
++-------------------+------+-------------------+------+---------+--------+
+```
+
+### Step 4 (c): Run Presto Queries
+
+Here are the Presto queries for similar Hive and Spark queries.
+
+:::note
+<ul>
+ <li> Currently, Presto does not support snapshot or incremental queries on Hudi tables. </li>
+ <li> This section of the demo is not supported for Mac AArch64 users at this time. </li>
+</ul>
+:::
+
+```java
+docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
+presto> show catalogs;
+ Catalog
+-----------
+ hive
+ jmx
+ localfile
+ system
+(4 rows)
+
+Query 20190817_134851_00000_j8rcz, FINISHED, 1 node
+Splits: 19 total, 19 done (100.00%)
+0:04 [0 rows, 0B] [0 rows/s, 0B/s]
+
+presto> use hive.default;
+USE
+presto:default> show tables;
+ Table
+--------------------
+ stock_ticks_cow
+ stock_ticks_mor_ro
+ stock_ticks_mor_rt
+(3 rows)
+
+Query 20190822_181000_00001_segyw, FINISHED, 2 nodes
+Splits: 19 total, 19 done (100.00%)
+0:05 [3 rows, 99B] [0 rows/s, 18B/s]
+
+
+# COPY-ON-WRITE Queries:
+=========================
+
+
+presto:default> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20190822_181011_00002_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:12 [197 rows, 613B] [16 rows/s, 50B/s]
+
+presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180221 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822180221 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20190822_181141_00003_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [109 rows/s, 341B/s]
+
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against M-O-R table.
+
+# Run ReadOptimized Query. Notice that the latest timestamp is 10:29
+ presto:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20190822_181158_00004_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:02 [197 rows, 613B] [110 rows/s, 343B/s]
+
+
+presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180250 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822180250 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20190822_181256_00006_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [92 rows/s, 286B/s]
+
+presto:default> exit
+```
+
+### Step 4 (d): Run Trino Queries
+
+Here are the similar queries with Trino.
+:::note
+<ul>
+ <li> Currently, Trino does not support snapshot or incremental queries on Hudi tables. </li>
+ <li> This section of the demo is not supported for Mac AArch64 users at this time. </li>
+</ul>
+:::
+
+```java
+docker exec -it adhoc-2 trino --server trino-coordinator-1:8091
+trino> show catalogs;
+ Catalog
+---------
+ hive
+ system
+(2 rows)
+
+Query 20220112_055038_00000_sac73, FINISHED, 1 node
+Splits: 19 total, 19 done (100.00%)
+3.74 [0 rows, 0B] [0 rows/s, 0B/s]
+
+trino> use hive.default;
+USE
+trino:default> show tables;
+ Table
+--------------------
+ stock_ticks_cow
+ stock_ticks_mor_ro
+ stock_ticks_mor_rt
+(3 rows)
+
+Query 20220112_055050_00003_sac73, FINISHED, 2 nodes
+Splits: 19 total, 19 done (100.00%)
+1.84 [3 rows, 102B] [1 rows/s, 55B/s]
+
+# COPY-ON-WRITE Queries:
+=========================
+
+trino:default> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20220112_055101_00005_sac73, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+4.08 [197 rows, 442KB] [48 rows/s, 108KB/s]
+
+trino:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20220112054822108 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20220112054822108 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20220112_055113_00006_sac73, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0.40 [197 rows, 450KB] [487 rows/s, 1.09MB/s]
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against MOR table.
+
+# Run ReadOptimized Query. Notice that the latest timestamp is 10:29
+
+trino:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20220112_055125_00007_sac73, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0.50 [197 rows, 442KB] [395 rows/s, 888KB/s]
+
+trino:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20220112054844841 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20220112054844841 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20220112_055136_00008_sac73, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0.49 [197 rows, 450KB] [404 rows/s, 924KB/s]
+
+trino:default> exit
+```
+
+### Step 5: Upload second batch to Kafka and run Hudi Streamer to ingest
+
+Upload the second batch of data and ingest this batch using Hudi Streamer. As this batch does not bring in any new
+partitions, there is no need to run hive-sync
+
+```java
+cat docker/demo/data/batch_2.json | kcat -b kafkabroker -t stock_ticks -P
+
+# Within Docker container, run the ingestion command
+docker exec -it adhoc-2 /bin/bash
+
+# Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_cow table in HDFS
+spark-submit \
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
+ --table-type COPY_ON_WRITE \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_cow \
+ --target-table stock_ticks_cow \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+
+# Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_mor table in HDFS
+spark-submit \
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
+ --disable-compaction
+
+exit
+```
+
+With Copy-On-Write table, the second ingestion by Hudi Streamer resulted in a new version of Parquet file getting created.
+See `http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow/2018/08/31`
+
+With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file.
+Take a look at the HDFS filesystem to get an idea: `http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_mor/2018/08/31`
+
+### Step 6 (a): Run Hive Queries
+
+With Copy-On-Write table, the Snapshot query immediately sees the changes as part of second batch once the batch
+got committed as each ingestion creates newer versions of parquet files.
+
+With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file.
+This is the time, when ReadOptimized and Snapshot queries will provide different results. ReadOptimized query will still
+return "10:29 am" as it will only read from the Parquet file. Snapshot query will do on-the-fly merge and return
+latest committed data which is "10:59 a.m".
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 \
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
+ --hiveconf hive.stats.autogather=false
+
+# Copy On Write Table:
+
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+1 row selected (1.932 seconds)
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924221953 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224524 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+
+# Merge On Read Table:
+
+# Read Optimized Query
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (1.6 seconds)
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+# Snapshot Query
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224537 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+exit
+```
+
+### Step 6 (b): Run Spark SQL Queries
+
+Running the same queries in Spark-SQL:
+
+```java
+docker exec -it adhoc-1 /bin/bash
+$SPARK_INSTALL/bin/spark-shell \
+ --jars $HUDI_SPARK_BUNDLE \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --deploy-mode client \
+ --driver-memory 1G \
+ --master local[2] \
+ --executor-memory 3G \
+ --num-executors 1
+
+# Copy On Write Table:
+
+scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:59:00|
++------+-------------------+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG'").show(100, false)
+
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924221953 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224524 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+
+# Merge On Read Table:
+
+# Read Optimized Query
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+
+| symbol | _c1 |
++---------+----------------------+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+
+1 row selected (1.6 seconds)
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+
+
+# Snapshot Query
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+
+| symbol | _c1 |
++---------+----------------------+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224537 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+
+
+exit
+```
+
+### Step 6 (c): Run Presto Queries
+
+Running the same queries on Presto for ReadOptimized queries.
+
+:::note
+This section of the demo is not supported for Mac AArch64 users at this time.
+:::
+
+```java
+docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
+presto> use hive.default;
+USE
+
+# Copy On Write Table:
+
+presto:default>select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:59:00
+(1 row)
+
+Query 20190822_181530_00007_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:02 [197 rows, 613B] [125 rows/s, 389B/s]
+
+presto:default>select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180221 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822181433 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215
+(2 rows)
+
+Query 20190822_181545_00008_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [106 rows/s, 332B/s]
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+
+# Merge On Read Table:
+
+# Read Optimized Query
+presto:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20190822_181602_00009_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:01 [197 rows, 613B] [139 rows/s, 435B/s]
+
+presto:default>select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180250 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822180250 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20190822_181615_00010_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:01 [197 rows, 613B] [154 rows/s, 480B/s]
+
+presto:default> exit
+```
+
+### Step 6 (d): Run Trino Queries
+
+Running the same queries on Trino for Read-Optimized queries.
+
+:::note
+This section of the demo is not supported for Mac AArch64 users at this time.
+:::
+
+```java
+docker exec -it adhoc-2 trino --server trino-coordinator-1:8091
+trino> use hive.default;
+USE
+
+# Copy On Write Table:
+
+trino:default> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:59:00
+(1 row)
+
+Query 20220112_055443_00012_sac73, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0.63 [197 rows, 442KB] [310 rows/s, 697KB/s]
+
+trino:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20220112054822108 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20220112055352654 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215
+(2 rows)
+
+Query 20220112_055450_00013_sac73, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0.65 [197 rows, 450KB] [303 rows/s, 692KB/s]
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+# Merge On Read Table:
+# Read Optimized Query
+
+trino:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20220112_055500_00014_sac73, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0.59 [197 rows, 442KB] [336 rows/s, 756KB/s]
+
+trino:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20220112054844841 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20220112054844841 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20220112_055506_00015_sac73, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0.35 [197 rows, 450KB] [556 rows/s, 1.24MB/s]
+
+trino:default> exit
+```
+
+### Step 7 (a): Incremental Query for COPY-ON-WRITE Table
+
+With 2 batches of data ingested, lets showcase the support for incremental queries in Hudi Copy-On-Write tables
+
+Lets take the same projection query example
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 \
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
+ --hiveconf hive.stats.autogather=false
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924064621 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924065039 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+```
+
+As you notice from the above queries, there are 2 commits - 20180924064621 and 20180924065039 in timeline order.
+When you follow the steps, you will be getting different timestamps for commits. Substitute them
+in place of the above timestamps.
+
+To show the effects of incremental-query, let us assume that a reader has already seen the changes as part of
+ingesting first batch. Now, for the reader to see effect of the second batch, he/she has to keep the start timestamp to
+the commit time of the first batch (20180924064621) and run incremental query
+
+Hudi incremental mode provides efficient scanning for incremental queries by filtering out files that do not have any
+candidate rows using hudi-managed metadata.
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 \
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
+ --hiveconf hive.stats.autogather=false
+
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.mode=INCREMENTAL;
+No rows affected (0.009 seconds)
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.max.commits=3;
+No rows affected (0.009 seconds)
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.start.timestamp=20180924064621;
+```
+
+With the above setting, file-ids that do not have any updates from the commit 20180924065039 is filtered out without scanning.
+Here is the incremental query :
+
+```java
+0: jdbc:hive2://hiveserver:10000>
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG' and `_hoodie_commit_time` > '20180924064621';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924065039 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+1 row selected (0.83 seconds)
+0: jdbc:hive2://hiveserver:10000>
+```
+
+### Step 7 (b): Incremental Query with Spark SQL:
+
+```java
+docker exec -it adhoc-1 /bin/bash
+$SPARK_INSTALL/bin/spark-shell \
+ --jars $HUDI_SPARK_BUNDLE \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --deploy-mode client \
+ --driver-memory 1G \
+ --master local[2] \
+ --executor-memory 3G \
+ --num-executors 1
+
+Welcome to
+ ____ __
+ / __/__ ___ _____/ /__
+ _\ \/ _ \/ _ `/ __/ '_/
+ /___/ .__/\_,_/_/ /_/\_\ version 2.4.4
+ /_/
+
+Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
+Type in expressions to have them evaluated.
+Type :help for more information.
+
+scala> import org.apache.hudi.DataSourceReadOptions
+import org.apache.hudi.DataSourceReadOptions
+
+# In the below query, 20180925045257 is the first commit's timestamp
+scala> val hoodieIncViewDF = spark.read.format("org.apache.hudi").option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL).option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY, "20180924064621").load("/user/hive/warehouse/stock_ticks_cow")
+SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
+SLF4J: Defaulting to no-operation (NOP) logger implementation
+SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
+hoodieIncViewDF: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 15 more fields]
+
+scala> hoodieIncViewDF.registerTempTable("stock_ticks_cow_incr_tmp1")
+warning: there was one deprecation warning; re-run with -deprecation for details
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow_incr_tmp1 where symbol = 'GOOG'").show(100, false);
++----------------------+---------+----------------------+---------+------------+-----------+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+
+| 20180924065039 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+
+```
+
+### Step 8: Schedule and Run Compaction for Merge-On-Read table
+
+Lets schedule and run a compaction to create a new version of columnar file so that read-optimized readers will see fresher data.
+Again, You can use Hudi CLI to manually schedule and run compaction
+
+```java
+docker exec -it adhoc-1 /bin/bash
+root@adhoc-1:/opt# /var/hoodie/ws/hudi-cli/hudi-cli.sh
+...
+Table command getting loaded
+HoodieSplashScreen loaded
+===================================================================
+* ___ ___ *
+* /\__\ ___ /\ \ ___ *
+* / / / /\__\ / \ \ /\ \ *
+* / /__/ / / / / /\ \ \ \ \ \ *
+* / \ \ ___ / / / / / \ \__\ / \__\ *
+* / /\ \ /\__\ / /__/ ___ / /__/ \ |__| / /\/__/ *
+* \/ \ \/ / / \ \ \ /\__\ \ \ \ / / / /\/ / / *
+* \ / / \ \ / / / \ \ / / / \ /__/ *
+* / / / \ \/ / / \ \/ / / \ \__\ *
+* / / / \ / / \ / / \/__/ *
+* \/__/ \/__/ \/__/ Apache Hudi CLI *
+* *
+===================================================================
+
+Welcome to Apache Hudi CLI. Please type help if you are looking for help.
+hudi->connect --path /user/hive/warehouse/stock_ticks_mor
+18/09/24 06:59:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+18/09/24 06:59:35 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
+18/09/24 06:59:35 INFO util.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://namenode:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1261652683_11, ugi=root (auth:SIMPLE)]]]
+18/09/24 06:59:35 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
+18/09/24 06:59:36 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1) from /user/hive/warehouse/stock_ticks_mor
+Metadata for table stock_ticks_mor loaded
+hoodie:stock_ticks_mor->compactions show all
+20/02/10 03:41:32 INFO timeline.HoodieActiveTimeline: Loaded instants [[20200210015059__clean__COMPLETED], [20200210015059__deltacommit__COMPLETED], [20200210022758__clean__COMPLETED], [20200210022758__deltacommit__COMPLETED], [==>20200210023843__compaction__REQUESTED]]
+___________________________________________________________________
+| Compaction Instant Time| State | Total FileIds to be Compacted|
+|==================================================================|
+
+# Schedule a compaction. This will use Spark Launcher to schedule compaction
+hoodie:stock_ticks_mor->compaction schedule --hoodieConfigs hoodie.compact.inline.max.delta.commits=1
+....
+Compaction successfully completed for 20180924070031
+
+# Now refresh and check again. You will see that there is a new compaction requested
+
+hoodie:stock_ticks_mor->refresh
+18/09/24 07:01:16 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:01:16 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
+18/09/24 07:01:16 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1) from /user/hive/warehouse/stock_ticks_mor
+Metadata for table stock_ticks_mor loaded
+
+hoodie:stock_ticks_mor->compactions show all
+18/09/24 06:34:12 INFO timeline.HoodieActiveTimeline: Loaded instants [[20180924041125__clean__COMPLETED], [20180924041125__deltacommit__COMPLETED], [20180924042735__clean__COMPLETED], [20180924042735__deltacommit__COMPLETED], [==>20180924063245__compaction__REQUESTED]]
+___________________________________________________________________
+| Compaction Instant Time| State | Total FileIds to be Compacted|
+|==================================================================|
+| 20180924070031 | REQUESTED| 1 |
+
+# Execute the compaction. The compaction instant value passed below must be the one displayed in the above "compactions show all" query
+hoodie:stock_ticks_mor->compaction run --compactionInstant 20180924070031 --parallelism 2 --sparkMemory 1G --schemaFilePath /var/demo/config/schema.avsc --retry 1
+....
+Compaction successfully completed for 20180924070031
+
+## Now check if compaction is completed
+
+hoodie:stock_ticks_mor->refresh
+18/09/24 07:03:00 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:03:00 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
+18/09/24 07:03:00 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1) from /user/hive/warehouse/stock_ticks_mor
+Metadata for table stock_ticks_mor loaded
+
+hoodie:stock_ticks_mor->compactions show all
+18/09/24 07:03:15 INFO timeline.HoodieActiveTimeline: Loaded instants [[20180924064636__clean__COMPLETED], [20180924064636__deltacommit__COMPLETED], [20180924065057__clean__COMPLETED], [20180924065057__deltacommit__COMPLETED], [20180924070031__commit__COMPLETED]]
+___________________________________________________________________
+| Compaction Instant Time| State | Total FileIds to be Compacted|
+|==================================================================|
+| 20180924070031 | COMPLETED| 1 |
+
+```
+
+### Step 9: Run Hive Queries including incremental queries
+
+You will see that both ReadOptimized and Snapshot queries will show the latest committed data.
+Lets also run the incremental query for MOR table.
+From looking at the below query output, it will be clear that the fist commit time for the MOR table is 20180924064636
+and the second commit time is 20180924070031
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 \
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
+ --hiveconf hive.stats.autogather=false
+
+# Read Optimized Query
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+1 row selected (1.6 seconds)
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924064636 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+# Snapshot Query
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924064636 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+# Incremental Query:
+
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.mode=INCREMENTAL;
+No rows affected (0.008 seconds)
+# Max-Commits covers both second batch and compaction commit
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.max.commits=3;
+No rows affected (0.007 seconds)
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.start.timestamp=20180924064636;
+No rows affected (0.013 seconds)
+# Query:
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG' and `_hoodie_commit_time` > '20180924064636';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+exit
+```
+
+### Step 10: Read Optimized and Snapshot queries for MOR with Spark-SQL after compaction
+
+```java
+docker exec -it adhoc-1 /bin/bash
+$SPARK_INSTALL/bin/spark-shell \
+ --jars $HUDI_SPARK_BUNDLE \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --deploy-mode client \
+ --driver-memory 1G \
+ --master local[2] \
+ --executor-memory 3G \
+ --num-executors 1
+
+# Read Optimized Query
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+
+| symbol | max(ts) |
++---------+----------------------+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+
+1 row selected (1.6 seconds)
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+
+| 20180924064636 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+
+
+# Snapshot Query
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+
+| symbol | max(ts) |
++---------+----------------------+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+
+| 20180924064636 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924070031 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+
+```
+
+### Step 11: Presto Read Optimized queries on MOR table after compaction
+:::note
+This section of the demo is not supported for Mac AArch64 users at this time.
+:::
+
+```java
+docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
+presto> use hive.default;
+USE
+
+# Read Optimized Query
+resto:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:59:00
+(1 row)
+
+Query 20190822_182319_00011_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:01 [197 rows, 613B] [133 rows/s, 414B/s]
+
+presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180250 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822181944 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215
+(2 rows)
+
+Query 20190822_182333_00012_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [98 rows/s, 307B/s]
+
+presto:default>
+```
+
+
+This brings the demo to an end.
+
+## Testing Hudi in Local Docker environment
+
+You can bring up a Hadoop Docker environment containing Hadoop, Hive and Spark services with support for Hudi.
+```java
+$ mvn pre-integration-test -DskipTests
+```
+The above command builds Docker images for all the services with
+current Hudi source installed at /var/hoodie/ws and also brings up the services using a compose file. We
+currently use Hadoop (v2.8.4), Hive (v2.3.3) and Spark (v2.4.4) in Docker images.
+
+To bring down the containers
+```java
+$ cd hudi-integ-test
+$ mvn docker-compose:down
+```
+
+If you want to bring up the Docker containers, use
+```java
+$ cd hudi-integ-test
+$ mvn docker-compose:up -DdetachedMode=true
+```
+
+Hudi is a library that is operated in a broader data analytics/ingestion environment
+involving Hadoop, Hive and Spark. Interoperability with all these systems is a key objective for us. We are
+actively adding integration-tests under __hudi-integ-test/src/test/java__ that makes use of this
+docker environment (See __hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieSanity.java__ )
+
+
+### Building Local Docker Containers:
+
+The Docker images required for demo and running integration test are already in docker-hub. The Docker images
+and compose scripts are carefully implemented so that they serve dual-purpose
+
+1. The Docker images have inbuilt Hudi jar files with environment variable pointing to those jars (HUDI_HADOOP_BUNDLE, ...)
+2. For running integration-tests, we need the jars generated locally to be used for running services within docker. The
+ docker-compose scripts (see `docker/compose/docker-compose_hadoop284_hive233_spark244.yml`) ensures local jars override
+ inbuilt jars by mounting local Hudi workspace over the Docker location
+3. As these Docker containers have mounted local Hudi workspace, any changes that happen in the workspace would automatically
+ reflect in the containers. This is a convenient way for developing and verifying Hudi for
+ developers who do not own a distributed environment. Note that this is how integration tests are run.
+
+This helps avoid maintaining separate Docker images and avoids the costly step of building Hudi Docker images locally.
+But if users want to test Hudi from locations with lower network bandwidth, they can still build local images
+run the script
+`docker/build_local_docker_images.sh` to build local Docker images before running `docker/setup_demo.sh`
+
+Here are the commands:
+
+```java
+cd docker
+./build_local_docker_images.sh
+.....
+
+[INFO] Reactor Summary:
+[INFO]
+[INFO] Hudi ............................................... SUCCESS [ 2.507 s]
+[INFO] hudi-common ........................................ SUCCESS [ 15.181 s]
+[INFO] hudi-aws ........................................... SUCCESS [ 2.621 s]
+[INFO] hudi-timeline-service .............................. SUCCESS [ 1.811 s]
+[INFO] hudi-client ........................................ SUCCESS [ 0.065 s]
+[INFO] hudi-client-common ................................. SUCCESS [ 8.308 s]
+[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 3.733 s]
+[INFO] hudi-spark-client .................................. SUCCESS [ 18.567 s]
+[INFO] hudi-sync-common ................................... SUCCESS [ 0.794 s]
+[INFO] hudi-hive-sync ..................................... SUCCESS [ 3.691 s]
+[INFO] hudi-spark-datasource .............................. SUCCESS [ 0.121 s]
+[INFO] hudi-spark-common_2.11 ............................. SUCCESS [ 12.979 s]
+[INFO] hudi-spark2_2.11 ................................... SUCCESS [ 12.516 s]
+[INFO] hudi-spark_2.11 .................................... SUCCESS [ 35.649 s]
+[INFO] hudi-utilities_2.11 ................................ SUCCESS [ 5.881 s]
+[INFO] hudi-utilities-bundle_2.11 ......................... SUCCESS [ 12.661 s]
+[INFO] hudi-cli ........................................... SUCCESS [ 19.858 s]
+[INFO] hudi-java-client ................................... SUCCESS [ 3.221 s]
+[INFO] hudi-flink-client .................................. SUCCESS [ 5.731 s]
+[INFO] hudi-spark3_2.12 ................................... SUCCESS [ 8.627 s]
+[INFO] hudi-dla-sync ...................................... SUCCESS [ 1.459 s]
+[INFO] hudi-sync .......................................... SUCCESS [ 0.053 s]
+[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [ 5.652 s]
+[INFO] hudi-hive-sync-bundle .............................. SUCCESS [ 1.623 s]
+[INFO] hudi-spark-bundle_2.11 ............................. SUCCESS [ 10.930 s]
+[INFO] hudi-presto-bundle ................................. SUCCESS [ 3.652 s]
+[INFO] hudi-timeline-server-bundle ........................ SUCCESS [ 4.804 s]
+[INFO] hudi-trino-bundle .................................. SUCCESS [ 5.991 s]
+[INFO] hudi-hadoop-docker ................................. SUCCESS [ 2.061 s]
+[INFO] hudi-hadoop-base-docker ............................ SUCCESS [ 53.372 s]
+[INFO] hudi-hadoop-base-java11-docker ..................... SUCCESS [ 48.545 s]
+[INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [ 6.098 s]
+[INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [ 4.825 s]
+[INFO] hudi-hadoop-history-docker ......................... SUCCESS [ 3.829 s]
+[INFO] hudi-hadoop-hive-docker ............................ SUCCESS [ 52.660 s]
+[INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [01:02 min]
+[INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [ 12.661 s]
+[INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [ 4.350 s]
+[INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [ 59.083 s]
+[INFO] hudi-hadoop-presto-docker .......................... SUCCESS [01:31 min]
+[INFO] hudi-hadoop-trinobase-docker ....................... SUCCESS [02:40 min]
+[INFO] hudi-hadoop-trinocoordinator-docker ................ SUCCESS [ 14.003 s]
+[INFO] hudi-hadoop-trinoworker-docker ..................... SUCCESS [ 12.100 s]
+[INFO] hudi-integ-test .................................... SUCCESS [ 13.581 s]
+[INFO] hudi-integ-test-bundle ............................. SUCCESS [ 27.212 s]
+[INFO] hudi-examples ...................................... SUCCESS [ 8.090 s]
+[INFO] hudi-flink_2.11 .................................... SUCCESS [ 4.217 s]
+[INFO] hudi-kafka-connect ................................. SUCCESS [ 2.966 s]
+[INFO] hudi-flink-bundle_2.11 ............................. SUCCESS [ 11.155 s]
+[INFO] hudi-kafka-connect-bundle .......................... SUCCESS [ 12.369 s]
+[INFO] ------------------------------------------------------------------------
+[INFO] BUILD SUCCESS
+[INFO] ------------------------------------------------------------------------
+[INFO] Total time: 14:35 min
+[INFO] Finished at: 2022-01-12T18:41:27-08:00
+[INFO] ------------------------------------------------------------------------
+```
diff --git a/website/versioned_docs/version-0.15.1/encryption.md b/website/versioned_docs/version-0.15.1/encryption.md
new file mode 100644
index 000000000000..d34cef749351
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/encryption.md
@@ -0,0 +1,73 @@
+---
+title: Encryption
+keywords: [ hudi, security ]
+summary: This section offers an overview of encryption feature in Hudi
+toc: true
+last_modified_at: 2022-02-14T15:59:57-04:00
+---
+
+Since Hudi 0.11.0, Spark 3.2 support has been added and accompanying that, Parquet 1.12 has been included, which brings encryption feature to Hudi. In this section, we will show a guide on how to enable encryption in Hudi tables.
+
+## Encrypt Copy-on-Write tables
+
+First, make sure Hudi Spark 3.2 bundle jar is used.
+
+Then, set the following Parquet configurations to make data written to Hudi COW tables encrypted.
+
+```java
+// Activate Parquet encryption, driven by Hadoop properties
+jsc.hadoopConfiguration().set("parquet.crypto.factory.class", "org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory")
+// Explicit master keys (base64 encoded) - required only for mock InMemoryKMS
+jsc.hadoopConfiguration().set("parquet.encryption.kms.client.class" , "org.apache.parquet.crypto.keytools.mocks.InMemoryKMS")
+jsc.hadoopConfiguration().set("parquet.encryption.key.list", "k1:AAECAwQFBgcICQoLDA0ODw==, k2:AAECAAECAAECAAECAAECAA==")
+// Write encrypted dataframe files.
+// Column "rider" will be protected with master key "key2".
+// Parquet file footers will be protected with master key "key1"
+jsc.hadoopConfiguration().set("parquet.encryption.footer.key", "k1")
+jsc.hadoopConfiguration().set("parquet.encryption.column.keys", "k2:rider")
+
+spark.read().format("org.apache.hudi").load("path").show();
+```
+
+Here is an example.
+
+```java
+JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());
+jsc.hadoopConfiguration().set("parquet.crypto.factory.class", "org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory");
+jsc.hadoopConfiguration().set("parquet.encryption.kms.client.class" , "org.apache.parquet.crypto.keytools.mocks.InMemoryKMS");
+jsc.hadoopConfiguration().set("parquet.encryption.footer.key", "k1");
+jsc.hadoopConfiguration().set("parquet.encryption.column.keys", "k2:rider");
+jsc.hadoopConfiguration().set("parquet.encryption.key.list", "k1:AAECAwQFBgcICQoLDA0ODw==, k2:AAECAAECAAECAAECAAECAA==");
+
+QuickstartUtils.DataGenerator dataGen = new QuickstartUtils.DataGenerator();
+List<String> inserts = convertToStringList(dataGen.generateInserts(3));
+Dataset<Row> inputDF1 = spark.read().json(jsc.parallelize(inserts, 1));
+inputDF1.write().format("org.apache.hudi")
+ .option("hoodie.table.name", "encryption_table")
+ .option("hoodie.upsert.shuffle.parallelism","2")
+ .option("hoodie.insert.shuffle.parallelism","2")
+ .option("hoodie.delete.shuffle.parallelism","2")
+ .option("hoodie.bulkinsert.shuffle.parallelism","2")
+ .mode(SaveMode.Overwrite)
+ .save("path");
+
+spark.read().format("org.apache.hudi").load("path").select("rider").show();
+```
+
+Reading the table works if configured correctly
+
+```
++---------+
+|rider |
++---------+
+|rider-213|
+|rider-213|
+|rider-213|
++---------+
+```
+
+Read more from [Spark docs](https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#columnar-encryption) and [Parquet docs](https://github.com/apache/parquet-format/blob/master/Encryption.md).
+
+### Note
+
+This feature is currently only available for COW tables due to only Parquet base files present there.
diff --git a/website/versioned_docs/version-0.15.1/faq.md b/website/versioned_docs/version-0.15.1/faq.md
new file mode 100644
index 000000000000..c53c8ebbc68c
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/faq.md
@@ -0,0 +1,15 @@
+---
+title: Overview
+keywords: [hudi, writing, reading]
+---
+# Overview
+
+The FAQs are split into following pages. Please refer to the specific pages for more info.
+
+- [General](faq_general.md)
+- [Design & Concepts](faq_design_and_concepts.md)
+- [Writing Tables](faq_writing_tables.md)
+- [Reading Tables](faq_reading_tables.md)
+- [Table Services](faq_table_services.md)
+- [Storage](faq_storage.md)
+- [Integrations](faq_integrations.md)
diff --git a/website/versioned_docs/version-0.15.1/faq_design_and_concepts.md b/website/versioned_docs/version-0.15.1/faq_design_and_concepts.md
new file mode 100644
index 000000000000..9cac3c0f9e9e
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/faq_design_and_concepts.md
@@ -0,0 +1,61 @@
+---
+title: Design & Concepts
+keywords: [hudi, writing, reading]
+---
+# Design & Concepts FAQ
+
+### How does Hudi ensure atomicity?
+
+Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline.md) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudi’s timeline.
+
+### Does Hudi extend the Hive table layout?
+
+Hudi is very different from Hive in important aspects described below. However, based on practical considerations, it chooses to be compatible with Hive table layout by adopting partitioning, schema evolution and being queryable through Hive query engine. Here are the key aspect where Hudi differs:
+
+* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases.md). Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
+* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially. Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering.md) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which “actually” help users ex [...]
+* Hudi considers partition evolution as an anti-pattern and avoids such schemes due to the inconsistent performance of queries that goes to depend on which part of the table is being queried. Hudi’s design favors consistent performance and is aware of the need to redesign to partitioning/tables to achieve the same.
+
+### What concurrency control approaches does Hudi adopt?
+
+Hudi provides snapshot isolation between all three types of processes - writers, readers, and table services, meaning they all operate on a consistent snapshot of the table. Hudi provides optimistic concurrency control (OCC) between writers, while providing lock-free, non-blocking MVCC-based concurrency control between writers and table-services and between different table services. Widely accepted database literature like “[Architecture of a database system, pg 81](https://dsf.berkeley. [...]
+
+### Hudi’s commits are based on transaction start time instead of completed time. Does this cause data loss or inconsistency in case of incremental and time travel queries?
+
+Let’s take a closer look at the scenario here: two commits C1 and C2 (with C2 starting later than C1) start with a later commit (C2) finishing first leaving the inflight transaction of the earlier commit (C1)
+before the completed write of the later transaction (C2) in Hudi’s timeline. This is not an uncommon scenario, especially with various ingestions needs such as backfilling, deleting, bootstrapping, etc
+alongside regular writes. When/Whether the first job would commit will depend on factors such as conflicts between concurrent commits, inflight compactions, other actions on the table’s timeline etc.
+If the first job fails for some reason, Hudi will abort the earlier commit inflight (c1) and the writer has to retry next time with a new instant time > c2 much similar to other OCC implementations.
+Firstly, for snapshot queries the order of commits should not matter at all, since any incomplete writes on the active timeline is ignored by queries and cause no side-effects.
+
+In these scenarios, it might be tempting to think of data inconsistencies/data loss when using Hudi’s incremental queries. However, Hudi takes special handling
+(examples [1](https://github.com/apache/hudi/blob/aea5bb6f0ab824247f5e3498762ad94f643a2cb6/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L76),
+[2](https://github.com/apache/hudi/blame/7a6543958368540d221ddc18e0c12b8d526b6859/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java#L173)) in incremental queries to ensure that no data
+is served beyond the point there is an inflight instant in its timeline, so no data loss or drop happens. This detection is made possible because Hudi writes first request a transaction on the timeline, before planning/executing
+the write, as explained in the [timeline](timeline.md#states) section.
+
+In this case, on seeing C1’s inflight commit (publish to timeline is atomic), C2 data (which is > C1 in the timeline) is not served until C1 inflight transitions to a terminal state such as completed or marked as failed.
+This [test](https://github.com/apache/hudi/blob/master/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java#L137) demonstrates how Hudi incremental source stops proceeding until C1 completes.
+Hudi favors [safety and sacrifices liveness](https://en.wikipedia.org/wiki/Safety_and_liveness_properties), in such a case. For a single writer, the start times of the transactions are the same as the order of completion of transactions, and both incremental and time-travel queries work as expected.
+In the case of multi-writer, incremental queries still work as expected but time travel queries don't. Since most time travel queries are on historical snapshots with a stable continuous timeline, this has not been implemented upto Hudi 0.13.
+However, a similar approach like above can be easily applied to failing time travel queries as well in this window.
+
+### How does Hudi plan to address the liveness issue above for incremental queries?
+
+Hudi 0.14 improves the liveness aspects by enabling change streams, incremental query and time-travel based on the file/object's timestamp (similar to [Delta Lake](https://docs.delta.io/latest/delta-batch.html#query-an-older-snapshot-of-a-table-time-travel)).
+
+To expand more on the long term approach, Hudi has had a proposal to streamline/improve this experience by adding a transition-time to our timeline, which will remove the [liveness sacrifice](https://en.wikipedia.org/wiki/Safety_and_liveness_properties) and makes it easier to understand.
+This has been delayed for a few reasons
+
+- Large hosted query engines and users not upgrading fast enough.
+- The issues brought up - \[[1](faq_design_and_concepts.md#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts.md#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
+relevant to this are not practically very important to users beyond good pedantic discussions,
+- Wanting to do it alongside [non-blocking concurrency control](https://github.com/apache/hudi/pull/7907) in Hudi version 1.x.
+
+It's planned to be addressed in the first 1.x release.
+
+### Does Hudi’s use of wall clock timestamp for instants pose any clock skew issues?
+
+Theoretically speaking, a clock skew between two writers can result in different notions of time, and order the timeline differently. But, the current NTP implementations and regions standardizing on UTC make this very impractical to happen in practice. Even many popular OLTP-based systems such as DynamoDB and Cassandra use timestamps for record level conflict detection, cloud providers/OSS NTP are moving towards atomic/synchronized clocks all the time \[[1](https://aws.amazon.com/about- [...]
+
+Further - Hudi’s commit time can be a logical time and need not strictly be a timestamp. If there are still uniqueness concerns over clock skew, it is easy for Hudi to further extend the timestamp implementation with salts or employ [TrueTime](https://www.cockroachlabs.com/blog/living-without-atomic-clocks/) approaches that have been proven at planet scale. In short, this is not a design issue, but more of a pragmatic implementation choice, that allows us to implement unique features lik [...]
diff --git a/website/versioned_docs/version-0.15.1/faq_general.md b/website/versioned_docs/version-0.15.1/faq_general.md
new file mode 100644
index 000000000000..5c44f8d6ebe7
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/faq_general.md
@@ -0,0 +1,98 @@
+---
+title: General
+keywords: [hudi, writing, reading]
+---
+# General FAQ
+
+### When is Hudi useful for me or my organization?
+
+If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](hoodie_streaming_ingestion.md). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
+
+As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker.
+
+### What are some non-goals for Hudi?
+
+Hudi is not designed for any OLTP use-cases, where typically you are using existing NoSQL/RDBMS data stores. Hudi cannot replace your in-memory analytical database (at-least not yet!). Hudi support near-real time ingestion in the order of few minutes, trading off latency for efficient batching. If you truly desirable sub-minute processing delays, then stick with your favorite stream processing solution.
+
+### What is incremental processing? Why does Hudi docs/talks keep talking about it?
+
+Incremental processing was first introduced by Vinoth Chandar, in the O'reilly [blog](https://www.oreilly.com/content/ubers-case-for-incremental-processing-on-hadoop/), that set off most of this effort. In purely technical terms, incremental processing merely refers to writing mini-batch programs in streaming processing style. Typical batch jobs consume **all input** and recompute **all output**, every few hours. Typical stream processing jobs consume some **new input** and recompute **n [...]
+
+While we can merely refer to this as stream processing, we call it _incremental processing_, to distinguish from purely stream processing pipelines built using Apache Flink or Apache Kafka Streams.
+
+### How is Hudi optimized for CDC and streaming use cases?
+
+One of the core use-cases for Apache Hudi is enabling seamless, efficient database ingestion to your lake, and change data capture is a direct application of that. Hudi’s core design primitives support fast upserts and deletes of data that are suitable for CDC and streaming use cases. Here is a glimpse of some of the challenges accompanying streaming and cdc workloads that Hudi handles efficiently out of the box.
+
+* **_Processing of deletes:_** Deletes are treated no differently than updates and are logged with the same filegroups where the corresponding keys exist. This helps process deletes faster same like regular inserts and updates and Hudi processes deletes at file group level using compaction in MOR tables. This can be very expensive in other open source systems that store deletes as separate files than data files and incur N(Data files)\*N(Delete files) merge cost to process deletes ever [...]
+* **_Operational overhead of merging deletes at scale:_** When deletes are stored as separate files without any notion of data locality, the merging of data and deletes can become a run away job that cannot complete in time due to various reasons (Spark retries, executor failure, OOM, etc.). As more data files and delete files are added, the merge becomes even more expensive and complex later on, making it hard to manage in practice causing operation overhead. Hudi removes this complex [...]
+* **_File sizing with updates:_** Other open source systems, process updates by generating new data files for inserting the new records after deletion, where both data files and delete files get introduced for every batch of updates. This yields to small file problem and requires file sizing. Whereas, Hudi embraces mutations to the data, and manages the table automatically by keeping file sizes in check without passing the burden of file sizing to users as manual maintenance.
+* **_Support for partial updates and payload ordering:_** Hudi support partial updates where already existing record can be updated for specific fields that are non null from newer records (with newer timestamps). Similarly, Hudi supports payload ordering with timestamp through specific payload implementation where late-arriving data with older timestamps will be ignored or dropped. Users can even implement custom logic and plug in to handle what they want.
+
+### How do I choose a storage type for my workload?
+
+A key goal of Hudi is to provide **upsert functionality** that is orders of magnitude faster than rewriting entire tables or partitions.
+
+Choose Copy-on-write storage if :
+
+* You are looking for a simple alternative, that replaces your existing parquet tables without any need for real-time data.
+* Your current job is rewriting entire table/partition to deal with updates, while only a few files actually change in each partition.
+* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations.md#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
+* Your workload is fairly well-understood and does not have sudden bursts of large amount of update or inserts to older partitions. COW absorbs all the merging cost on the writer side and thus these sudden changes can clog up your ingestion and interfere with meeting normal mode ingest latency targets.
+
+Choose merge-on-read storage if :
+
+* You want the data to be ingested as quickly & queryable as much as possible.
+* Your workload can have sudden spikes/changes in pattern (e.g bulk updates to older transactions in upstream database causing lots of updates to old partitions on DFS). Asynchronous compaction helps amortize the write amplification caused by such scenarios, while normal ingestion keeps up with incoming stream of changes.
+
+Immaterial of what you choose, Hudi provides
+
+* Snapshot isolation and atomic write of batch of records
+* Incremental pulls
+* Ability to de-duplicate data
+
+Find more [here](concepts.md).
+
+### Is Hudi an analytical database?
+
+A typical database has a bunch of long running storage servers always running, which takes writes and reads. Hudi's architecture is very different and for good reasons. It's highly decoupled where writes and queries/reads can be scaled independently to be able to handle the scale challenges. So, it may not always seems like a database.
+
+Nonetheless, Hudi is designed very much like a database and provides similar functionality (upserts, change capture) and semantics (transactional writes, snapshot isolated reads).
+
+### How do I model the data stored in Hudi?
+
+When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data.md) for an example.
+
+When querying/reading data, Hudi just presents itself as a json-like hierarchical table, everyone is used to querying using Hive/Spark/Presto over Parquet/Json/Avro.
+
+### Why does Hudi require a key field to be configured?
+
+Hudi was designed to support fast record level Upserts and thus requires a key to identify whether an incoming record is
+an insert or update or delete, and process accordingly. Additionally, Hudi automatically maintains indexes on this primary
+key and for many use-cases like CDC, ensuring such primary key constraints is crucial to ensure data quality. In this context,
+pre combine key helps reconcile multiple records with same key in a single batch of input records. Even for append-only data
+streams, Hudi supports key based de-duplication before inserting records. For e-g; you may have atleast once data integration
+systems like Kafka MirrorMaker that can introduce duplicates during failures. Even for plain old batch pipelines, keys
+help eliminate duplication that could be caused by backfill pipelines, where commonly it's unclear what set of records
+need to be re-written. We are actively working on making keys easier by only requiring them for Upsert and/or automatically
+generate the key internally (much like RDBMS row\_ids)
+
+### How does Hudi actually store data inside a table?
+
+At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts.md) for more details.
+
+### How Hudi handles partition evolution requirements ?
+
+Hudi recommends keeping coarse grained top level partition paths e.g date(ts) and within each such partition do clustering in a flexible way to z-order, sort data based on interested columns. This provides excellent performance by : minimzing the number of files in each partition, while still packing data that will be queried together physically closer (what partitioning aims to achieve).
+
+Let's take an example of a table, where we store log\_events with two fields `ts` (time at which event was produced) and `cust_id` (user for which event was produced) and a common option is to partition by both date(ts) and cust\_id.
+Some users may want to start granular with hour(ts) and then later evolve to new partitioning scheme say date(ts). But this means, the number of partitions in the table could be very high - 365 days x 1K customers = at-least 365K potentially small parquet files, that can significantly slow down queries, facing throttling issues on the actual S3/DFS reads.
+
+For the afore mentioned reasons, we don't recommend mixing different partitioning schemes within the same table, since it adds operational complexity, and unpredictable performance.
+Old data stays in old partitions and only new data gets into newer evolved partitions. If you want to tidy up the table, one has to rewrite all partition/data anwyay! This is where we suggest start with coarse grained partitions
+and lean on clustering techniques to optimize for query performance.
+
+We find that most datasets have at-least one high fidelity field, that can be used as a coarse partition. Clustering strategies in Hudi provide a lot of power - you can alter which partitions to cluster, and which fields to cluster each by etc.
+Unlike Hive partitioning, Hudi does not remove the partition field from the data files i.e if you write new partition paths, it does not mean old partitions need to be rewritten.
+Partitioning by itself is a relic of the Hive era; Hudi is working on replacing partitioning with database like indexing schemes/functions,
+for even more flexibility and get away from Hive-style partition evol route.
diff --git a/website/versioned_docs/version-0.15.1/faq_integrations.md b/website/versioned_docs/version-0.15.1/faq_integrations.md
new file mode 100644
index 000000000000..614b1b45b62a
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/faq_integrations.md
@@ -0,0 +1,68 @@
+---
+title: Integrations
+keywords: [hudi, writing, reading]
+---
+# Integrations FAQ
+
+### Does AWS GLUE support Hudi ?
+
+AWS Glue jobs can write, read and update Glue Data Catalog for hudi tables. In order to successfully integrate with Glue Data Catalog, you need to subscribe to one of the AWS provided Glue connectors named "AWS Glue Connector for Apache Hudi". Glue job needs to have "Use Glue data catalog as the Hive metastore" option ticked. Detailed steps with a sample scripts is available on this article provided by AWS - [https://aws.amazon.com/blogs/big-data/writing-to-apache-hudi-tables-using-aws-g [...]
+
+In case if your using either notebooks or Zeppelin through Glue dev-endpoints, your script might not be able to integrate with Glue DataCatalog when writing to hudi tables.
+
+### How to override Hudi jars in EMR?
+
+If you are looking to override Hudi jars in your EMR clusters one way to achieve this is by providing the Hudi jars through a bootstrap script.
+
+Here are the example steps for overriding Hudi version 0.7.0 in EMR 0.6.2.
+
+**Build Hudi Jars:**
+
+```bash
+# Git clone
+git clone https://github.com/apache/hudi.git && cd hudi
+
+# Get version 0.7.0
+git checkout --track origin/release-0.7.0
+
+# Build jars with spark 3.0.0 and scala 2.12 (since emr 6.2.0 uses spark 3 which requires scala 2.12):
+mvn clean package -DskipTests -Dspark3 -Dscala-2.12 -T 30
+```
+
+**Copy jars to s3:**
+
+These are the jars we are interested in after build completes. Copy them to a temp location first.
+
+```bash
+mkdir -p ~/Downloads/hudi-jars
+cp packaging/hudi-hadoop-mr-bundle/target/hudi-hadoop-mr-bundle-0.7.0.jar ~/Downloads/hudi-jars/
+cp packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-0.7.0.jar ~/Downloads/hudi-jars/
+cp packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.12-0.7.0.jar ~/Downloads/hudi-jars/
+cp packaging/hudi-timeline-server-bundle/target/hudi-timeline-server-bundle-0.7.0.jar ~/Downloads/hudi-jars/
+cp packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-0.7.0.jar ~/Downloads/hudi-jars/
+```
+
+Upload all jars from ~/Downloads/hudi-jars/ to the s3 location s3://xxx/yyy/hudi-jars
+
+**Include Hudi jars as part of the emr bootstrap script:**
+
+Below script downloads Hudi jars from above s3 location. Use this script as part `bootstrap-actions` when launching the EMR cluster to install the jars in each node.
+
+```bash
+#!/bin/bash
+sudo mkdir -p /mnt1/hudi-jars
+
+sudo aws s3 cp s3://xxx/yyy/hudi-jars /mnt1/hudi-jars --recursive
+
+# create symlinks
+cd /mnt1/hudi-jars
+sudo ln -sf hudi-hadoop-mr-bundle-0.7.0.jar hudi-hadoop-mr-bundle.jar
+sudo ln -sf hudi-hive-sync-bundle-0.7.0.jar hudi-hive-sync-bundle.jar
+sudo ln -sf hudi-spark-bundle_2.12-0.7.0.jar hudi-spark-bundle.jar
+sudo ln -sf hudi-timeline-server-bundle-0.7.0.jar hudi-timeline-server-bundle.jar
+sudo ln -sf hudi-utilities-bundle_2.12-0.7.0.jar hudi-utilities-bundle.jar
+```
+
+**Using the overriden jar in Deltastreamer:**
+
+When invoking DeltaStreamer specify the above jar location as part of spark-submit command.
diff --git a/website/versioned_docs/version-0.15.1/faq_reading_tables.md b/website/versioned_docs/version-0.15.1/faq_reading_tables.md
new file mode 100644
index 000000000000..e9db97dbbdd2
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/faq_reading_tables.md
@@ -0,0 +1,31 @@
+---
+title: Reading Tables
+keywords: [hudi, writing, reading]
+---
+# Reading Tables FAQ
+
+### Does deleted records appear in Hudi's incremental query results?
+
+Soft Deletes (unlike hard deletes) do appear in the incremental pull query results. So, if you need a mechanism to propagate deletes to downstream tables, you can use Soft deletes.
+
+### How do I pass hudi configurations to my beeline Hive queries?
+
+If Hudi's input format is not picked the returned results may be incorrect. To ensure correct inputformat is picked, please use `org.apache.hadoop.hive.ql.io.HiveInputFormat` or `org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat` for `hive.input.format` config. This can be set like shown below:
+
+```plain
+set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat
+```
+
+or
+
+```plain
+set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat
+```
+
+### Does Hudi guarantee consistent reads? How to think about read optimized queries?
+
+Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types.md#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
+
+Hudi’s read-optimized query is targeted for the MOR table only, with guidance around how compaction should be run to achieve predictable results. In the MOR table, the compaction, which runs every few commits (or “deltacommit” to be exact for the MOR table) by default, merges the base (parquet) file and corresponding change log files to a new base file within each file group, so that the snapshot query serving the latest data immediately after compaction reads the base files only. Simil [...]
+
+Users must use snapshot queries to read the latest snapshot of a MOR table. Popular engines including Spark, Presto, and Hive already support snapshot queries on MOR table and the snapshot query support in Trino is in progress (the [PR](https://github.com/trinodb/trino/pull/14786) is under review). Note that the read-optimized query does not apply to the COW table.
diff --git a/website/versioned_docs/version-0.15.1/faq_storage.md b/website/versioned_docs/version-0.15.1/faq_storage.md
new file mode 100644
index 000000000000..652bc73e23b4
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/faq_storage.md
@@ -0,0 +1,207 @@
+---
+title: Storage
+keywords: [hudi, writing, reading]
+---
+# Storage FAQ
+
+### Does Hudi support cloud storage/object stores?
+
+Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud.md) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations.md#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
+
+### What is the difference between copy-on-write (COW) vs merge-on-read (MOR) table types?
+
+**Copy On Write** - This storage type enables clients to ingest data on columnar file formats, currently parquet. Any new data that is written to the Hudi table using COW storage type, will write new parquet files. Updating an existing set of rows will result in a rewrite of the entire parquet files that collectively contain the affected rows being updated. Hence, all writes to such tables are limited by parquet writing performance, the larger the parquet file, the higher is the time tak [...]
+
+**Merge On Read** - This storage type enables clients to ingest data quickly onto row based data format such as avro. Any new data that is written to the Hudi table using MOR table type, will write new log/delta files that internally store the data as avro encoded bytes. A compaction process (configured as inline or asynchronous) will convert log file format to columnar file format (parquet). Two different InputFormats expose 2 different views of this data, Read Optimized view exposes co [...]
+
+More details can be found [here](concepts.md) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
+
+### How do I migrate my data to Hudi?
+
+Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables.md#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide.md), including ways to doing partial migrations.
+
+### How to convert an existing COW table to MOR?
+
+All you need to do is to edit the table type property in hoodie.properties(located at hudi_table_path/.hoodie/hoodie.properties).
+
+But manually changing it will result in checksum errors. So, we have to go via hudi-cli.
+
+1. Copy existing hoodie.properties to a new location.
+2. Edit table type to MERGE\_ON\_READ
+3. launch hudi-cli
+ 1. connect --path hudi\_table\_path
+ 2. repair overwrite-hoodie-props --new-props-file new\_hoodie.properties
+
+### How can I find the average record size in a commit?
+
+The `commit showpartitons` command in [HUDI CLI](cli.md) will show both "bytes written" and
+
+"records inserted." Divide the bytes written by records inserted to find the average size. Note that this answer assumes
+
+metadata overhead is negligible. For a small table (such as 5 columns, 100 records) this will not be the case.
+
+### How does the Hudi indexing work & what are its benefits?
+
+The indexing component is a key part of the Hudi writing and it maps a given recordKey to a fileGroup inside Hudi consistently. This enables faster identification of the file groups that are affected/dirtied by a given write operation.
+
+Hudi supports a few options for indexing as below
+
+* _HoodieBloomIndex_ : Uses a bloom filter and ranges information placed in the footer of parquet/base files (and soon log files as well)
+* _HoodieGlobalBloomIndex_ : The non global indexing only enforces uniqueness of a key inside a single partition i.e the user is expected to know the partition under which a given record key is stored. This helps the indexing scale very well for even [very large datasets](https://www.uber.com/en-IN/blog/uber-big-data-platform/). However, in some cases, it might be necessary instead to do the de-duping/enforce uniqueness across all partitions and the global bloom index does exactly that [...]
+* _HBaseIndex_ : Apache HBase is a key value store, typically found in close proximity to HDFS. You can also store the index inside HBase, which could be handy if you are already operating HBase.
+* _HoodieSimpleIndex (default)_ : A simple index which reads interested fields (record key and partition path) from base files and joins with incoming records to find the tagged location.
+* _HoodieGlobalSimpleIndex_ : Global version of Simple Index, where in uniqueness is on record key across entire table.
+* _HoodieBucketIndex_ : Each partition has statically defined buckets to which records are tagged with. Since locations are tagged via hashing mechanism, this index lookup will be very efficient.
+* _HoodieSparkConsistentBucketIndex_ : This is also similar to Bucket Index. Only difference is that, data skews can be tackled by dynamically changing the bucket number.
+
+You can implement your own index if you'd like, by subclassing the `HoodieIndex` class and configuring the index class name in configs.
+
+### Can I switch from one index type to another without having to rewrite the entire table?
+
+It should be okay to switch between Bloom index and Simple index as long as they are not global.
+
+Moving from global to non-global and vice versa may not work. Also switching between Hbase (gloabl index) and regular bloom might not work.
+
+### I have an existing dataset and want to evaluate Hudi using portion of that data ?
+
+You can bulk import portion of that data to a new hudi table. For example, if you want to try on a month of data -
+
+```scala
+spark.read.parquet("your_data_set/path/to/month")
+ .write.format("org.apache.hudi")
+ .option("hoodie.datasource.write.operation", "bulk_insert")
+ .option("hoodie.datasource.write.storage.type", "storage_type") // COPY_ON_WRITE or MERGE_ON_READ
+ .option("hoodie.datasource.write.recordkey.field", "<your key>").
+ .option("hoodie.datasource.write.partitionpath.field", "<your_partition>")
+ ...
+ .mode(SaveMode.Append)
+ .save(basePath);
+```
+
+Once you have the initial copy, you can simply run upsert operations on this by selecting some sample of data every round
+
+```scala
+spark.read.parquet("your_data_set/path/to/month").limit(n) // Limit n records
+ .write.format("org.apache.hudi")
+ .option("hoodie.datasource.write.operation", "upsert")
+ .option("hoodie.datasource.write.recordkey.field", "<your key>").
+ .option("hoodie.datasource.write.partitionpath.field", "<your_partition>")
+ ...
+ .mode(SaveMode.Append)
+ .save(basePath);
+```
+
+For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli.md).
+
+### Why does maintain record level commit metadata? Isn't tracking table version at file level good enough?
+
+By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering.md#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hu [...]
+
+### Why partition fields are also stored in parquet files in addition to the partition path ?
+
+Hudi supports customizable partition values which could be a derived value of another field. Also, storing the partition value only as part of the field results in losing type information when queried by various query engines.
+
+### How do I configure Bloom filter (when Bloom/Global\_Bloom index is used)?
+
+Bloom filters are used in bloom indexes to look up the location of record keys in write path. Bloom filters are used only when the index type is chosen as “BLOOM” or “GLOBAL\_BLOOM”. Hudi has few config knobs that users can use to tune their bloom filters.
+
+On a high level, hudi has two types of blooms: Simple and Dynamic.
+
+Simple, as the name suggests, is simple. Size is statically allocated based on few configs.
+
+`hoodie.bloom.index.filter.type`: SIMPLE
+
+`hoodie.index.bloom.num_entries` refers to the total number of entries per bloom filter, which refers to one file slice. Default value is 60000.
+
+`hoodie.index.bloom.fpp` refers to the false positive probability with the bloom filter. Default value: 1\*10^-9.
+
+Size of the bloom filter depends on these two values. This is statically allocated and here is the formula that determines the size of bloom. Until the total number of entries added to the bloom is within the configured `hoodie.index.bloom.num_entries` value, the fpp will be honored. i.e. with default values of 60k and 1\*10^-9, bloom filter serialized size = 430kb. But if more entries are added, then the false positive probability will not be honored. Chances that more false positives c [...]
+
+Hudi suggests to have roughly 100 to 120 mb sized files for better query performance. So, based on the record size, one could determine how many records could fit into one data file.
+
+Lets say your data file max size is 128Mb and default avg record size is 1024 bytes. Hence, roughly this translates to 130k entries per data file. For this config, you should set num\_entries to ~130k.
+
+Dynamic bloom filter:
+
+`hoodie.bloom.index.filter.type` : DYNAMIC
+
+This is an advanced version of the bloom filter which grows dynamically as the number of entries grows. So, users are expected to set two values wrt num\_entries. `hoodie.index.bloom.num_entries` will determine the starting size of the bloom. `hoodie.bloom.index.filter.dynamic.max.entries` will determine the max size to which the bloom can grow upto. And fpp needs to be set similar to “Simple” bloom filter. Bloom size will be allotted based on the first config `hoodie.index.bloom.num_ent [...]
+
+### How do I verify datasource schema reconciliation in Hudi?
+
+With Hudi you can reconcile schema, meaning you can apply target table schema on your incoming data, so if there's a missing field in your batch it'll be injected null value. You can enable schema reconciliation using [hoodie.datasource.write.reconcile.schema](configurations.md#hoodiedatasourcewritereconcileschema) config.
+
+Example how schema reconciliation works with Spark:
+
+```scala
+hudi_options = {
+ 'hoodie.table.name': "test_recon1",
+ 'hoodie.datasource.write.recordkey.field': 'uuid',
+ 'hoodie.datasource.write.table.name': "test_recon1",
+ 'hoodie.datasource.write.precombine.field': 'ts',
+ 'hoodie.upsert.shuffle.parallelism': 2,
+ 'hoodie.insert.shuffle.parallelism': 2,
+ "hoodie.datasource.write.hive_style_partitioning":"true",
+ "hoodie.datasource.write.reconcile.schema": "true",
+ "hoodie.datasource.hive_sync.jdbcurl":"thrift://localhost:9083",
+ "hoodie.datasource.hive_sync.database":"hudi",
+ "hoodie.datasource.hive_sync.table":"test_recon1",
+ "hoodie.datasource.hive_sync.enable":"true",
+ "hoodie.datasource.hive_sync.mode": "hms"
+}
+
+some_json = '{"uuid":1,"ts":1,"Url":"hudi.apache.com"}'
+df = spark.read.json(sc.parallelize([some_json]))
+
+df.write.format("hudi").mode("append").options(**hudi_options).save(base_path)
+
+spark.sql("select * from hudi.test_recon1;").show()
+
+missing_field_json = '{"uuid":2,"ts":1}'
+df = spark.read.json(sc.parallelize([missing_field_json]))
+
+df.write.format("hudi").mode("append").options(**hudi_options).save(base_path)
+
+spark.sql("select * from hudi.test_recon1;").show()
+```
+
+After first write:
+
+| \_hoodie\_commit\_time | \_hoodie\_commit\_seqno | \_hoodie\_record\_key | \_hoodie\_partition\_path | \_hoodie\_file\_name | Url | ts | uuid |
+| ---| ---| ---| ---| ---| ---| ---| --- |
+| 20220622204044318 | 20220622204044318... | 1 | | 890aafc0-d897-44d... | [hudi.apache.com](http://hudi.apache.com) | 1 | 1 |
+
+After the second write:
+
+| \_hoodie\_commit\_time | \_hoodie\_commit\_seqno | \_hoodie\_record\_key | \_hoodie\_partition\_path | \_hoodie\_file\_name | Url | ts | uuid |
+| ---| ---| ---| ---| ---| ---| ---| --- |
+| 20220622204044318 | 20220622204044318... | 1 | | 890aafc0-d897-44d... | [hudi.apache.com](http://hudi.apache.com) | 1 | 1 |
+| 20220622204208997 | 20220622204208997... | 2 | | 890aafc0-d897-44d... | null | 1 | 2 |
+
+### Can I change keygenerator for an existing table?
+
+No. There are small set of properties that cannot change once chosen. KeyGenerator is one among them. [Here](https://github.com/apache/hudi/blob/3f37d4fb08169c95930f9cc32389abf4e5cd5551/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala#L128) is a code referecne where we validate the properties.
+
+### How can I resolve the NoSuchMethodError from HBase when using Hudi with metadata table on HDFS?
+
+In 0.15.0 release, we have upgraded the HBase version to 2.4.13, which is released based on Hadoop 2.x. Hudi's metadata table uses HFile as the base file format, relying on the HBase library. When enabling metadata table in a Hudi table on HDFS using Hadoop 3.x, NoSuchMethodError can be thrown due to compatibility issues between Hadoop 2.x and 3.x. To address this, here's the workaround:
+
+(1) Download HBase source code from `https://github.com/apache/hbase`;
+
+(2) Switch to the source code of 2.4.13 release with the tag `rel/2.4.13`;
+```scala
+git checkout rel/2.4.13
+```
+(3) Package a new version of HBase 2.4.13 with Hadoop 3 version:
+```scala
+mvn clean install -Denforcer.skip -DskipTests -Dhadoop.profile=3.0 -Psite-install-step
+```
+(4) Package Hudi again.
+
+### Is Hudi JVM dependent? Does Hudi leverage Java specific serialization?
+
+Hudi was not originally designed as a database layer that would fit under the various big data query engines, that were painfully hard to integrate with (Spark did not have DataSet/DataSource APIs, Trino was still Presto, Presto SPI was still budding, Hive storage handlers were just out). Popular engines including Spark, Flink, Presto, Trino, and Athena do not have issues integrating with Hudi as they are all based on JVM, and access access to Timeline, Metadata table are well-abstracted [...]
+
+Since it was not thought of as a "format", the focus on the APIs for such lower level integrations and documenting the serialized bytes has been historically inadequate. However, with some understanding of the serialization, looking beyond the APIs used and focus on what the serialized bytes are, its possible to integrate Hudi from outside the JVM. For e.g Bloom filters are serialized as hex strings, from byte arrays/primitive types, and should be **readable cross language**. The Hudi Lo [...]
+
+**_Note_**: _In a recent release the delete block keys were unintentionally serialized as kryo, and is being fixed in the 0.14 release. Thankfully, since Hudi’s log blocks and format are versioned, when the file slice is compacted things return to normal._
diff --git a/website/versioned_docs/version-0.15.1/faq_table_services.md b/website/versioned_docs/version-0.15.1/faq_table_services.md
new file mode 100644
index 000000000000..dc4dbcf9716f
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/faq_table_services.md
@@ -0,0 +1,55 @@
+---
+title: Table Services
+keywords: [hudi, writing, reading]
+---
+# Table Services FAQ
+
+### What does the Hudi cleaner do?
+
+The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations.md#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be r [...]
+
+### How do I run compaction for a MOR table?
+
+Simplest way to run compaction on MOR table is to run the [compaction inline](configurations.md#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted mos [...]
+
+That said, for obvious reasons of not blocking ingesting for compaction, you may want to run it asynchronously as well. This can be done either via a separate [compaction job](https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java) that is scheduled by your workflow scheduler/notebook independently. If you are using Hudi Streamer, then you can run in [continuous mode](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca [...]
+
+### What options do I have for asynchronous/offline compactions on MOR table?
+
+There are a couple of options depending on how you write to Hudi. But first let us understand briefly what is involved. There are two parts to compaction
+
+* Scheduling: In this step, Hudi scans the partitions and selects file slices to be compacted. A compaction plan is finally written to Hudi timeline. Scheduling needs tighter coordination with other writers (regular ingestion is considered one of the writers). If scheduling is done inline with the ingestion job, this coordination is automatically taken care of. Else when scheduling happens asynchronously a lock provider needs to be configured for this coordination among multiple writers.
+* Execution: In this step the compaction plan is read and file slices are compacted. Execution doesnt need the same level of coordination with other writers as Scheduling step and can be decoupled from ingestion job easily.
+
+Depending on how you write to Hudi these are the possible options currently.
+
+* DeltaStreamer:
+ * In Continuous mode, asynchronous compaction is achieved by default. Here scheduling is done by the ingestion job inline and compaction execution is achieved asynchronously by a separate parallel thread.
+ * In non continuous mode, only inline compaction is possible.
+ * Please note in either mode, by passing --disable-compaction compaction is completely disabled
+* Spark datasource:
+ * Async scheduling and async execution can be achieved by periodically running an offline Hudi Compactor Utility or Hudi CLI. However this needs a lock provider to be configured.
+ * Alternately, from 0.11.0, to avoid dependency on lock providers, scheduling alone can be done inline by regular writer using the config `hoodie.compact.schedule.inline` . And compaction execution can be done offline by periodically triggering the Hudi Compactor Utility or Hudi CLI.
+* Spark structured streaming:
+ * Compactions are scheduled and executed asynchronously inside the streaming job. Async Compactions are enabled by default for structured streaming jobs on Merge-On-Read table.
+ * Please note it is not possible to disable async compaction for MOR table with spark structured streaming.
+* Flink:
+ * Async compaction is enabled by default for Merge-On-Read table.
+ * Offline compaction can be achieved by setting `compaction.async.enabled` to `false` and periodically running [Flink offline Compactor](compaction/#flink-offline-compaction). When running the offline compactor, one needs to ensure there are no active writes to the table.
+ * Third option (highly recommended over the second one) is to schedule the compactions from the regular ingestion job and executing the compaction plans from an offline job. To achieve this set `compaction.async.enabled` to `false`, `compaction.schedule.enabled` to `true` and then run the [Flink offline Compactor](compaction/#flink-offline-compaction) periodically to execute the plans.
+
+### How to disable all table services in case of multiple writers?
+
+[hoodie.table.services.enabled](configurations.md#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
+
+### Why does Hudi retain at-least one previous commit even after setting hoodie.cleaner.commits.retained': 1 ?
+
+Hudi runs cleaner to remove old file versions as part of writing data either in inline or in asynchronous mode (0.6.0 onwards). Hudi Cleaner retains at-least one previous commit when cleaning old file versions. This is to prevent the case when concurrently running queries which are reading the latest file versions suddenly see those files getting deleted by cleaner because a new file version got added . In other words, retaining at-least one previous commit is needed for ensuring snapsho [...]
+
+### Can I get notified when new commits happen in my Hudi table?
+
+Yes. Hudi provides the ability to post a callback notification about a write commit. You can use a http hook or choose to
+
+be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback.md)
+
+for details
diff --git a/website/versioned_docs/version-0.15.1/faq_writing_tables.md b/website/versioned_docs/version-0.15.1/faq_writing_tables.md
new file mode 100644
index 000000000000..da2bfa6a7b84
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/faq_writing_tables.md
@@ -0,0 +1,194 @@
+---
+title: Writing Tables
+keywords: [hudi, writing, reading]
+---
+# Writing Tables FAQ
+
+### What are some ways to write a Hudi table?
+
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations.md) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API a [...]
+
+### How is a Hudi writer job deployed?
+
+The nice thing about Hudi writing is that it just runs like any other spark job would on a YARN/Mesos or even a K8S cluster. So you could simply use the Spark UI to get visibility into write operations.
+
+### Can I implement my own logic for how input records are merged with record on storage?
+
+Here is the payload interface that is used in Hudi to represent any hudi record.
+
+```java
+public interface HoodieRecordPayload<T extends HoodieRecordPayload> extends Serializable {
+ /**
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert by taking in a property map.
+ * Implementation can leverage the property to decide their business logic to do preCombine.
+ * @param another instance of another {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return the combined value
+ */
+ default T preCombine(T another, Properties properties);
+/**
+ * This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained
+ * in this object. Implementations can leverage properties if required.
+ * <p>
+ * eg:
+ * 1) You are updating counters, you may want to add counts to currentValue and write back updated counts
+ * 2) You may be reading DB redo logs, and merge them with current image for a database row on storage
+ * </p>
+ *
+ * @param currentValue Current value in storage, to merge/combine this payload with
+ * @param schema Schema used for record
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return new combined/merged value to be written back to storage. EMPTY to skip writing this record.
+ */
+ default Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException;
+
+/**
+ * Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a new value for the given
+ * HoodieKey, wherein there is no existing record in storage to be combined against. (i.e insert) Return EMPTY to skip writing this record.
+ * Implementations can leverage properties if required.
+ * @param schema Schema used for record
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return the {@link IndexedRecord} to be inserted.
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default Option<IndexedRecord> getInsertValue(Schema schema, Properties properties) throws IOException;
+/**
+ * This method can be used to extract some metadata from HoodieRecordPayload. The metadata is passed to {@code WriteStatus.markSuccess()} and
+ * {@code WriteStatus.markFailure()} in order to compute some aggregate metrics using the metadata in the context of a write success or failure.
+ * @return the metadata in the form of Map<String, String> if any.
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default Option<Map<String, String>> getMetadata() {
+ return Option.empty();
+ }
+}
+```
+
+As you could see, ([combineAndGetUpdateValue(), getInsertValue()](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java)) that control how the record on storage is combined with the incoming update/insert to generate the final value to be written back to storage. preCombine() is used to merge records within the same incoming batch.
+
+### How do I delete records in the dataset using Hudi?
+
+GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data.md#deletes).
+
+### Should I need to worry about deleting all copies of the records in case of duplicates?
+
+No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables.md#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables.md#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https:/ [...]
+
+### How does Hudi handle duplicate record keys in an input?
+
+When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
+
+For an insert or bulk\_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations.md#hoodiecombinebeforeinsert) [...]
+
+### How can I pass hudi configurations to my spark writer job?
+
+Hudi configuration options covering the datasource and low level Hudi write client (which both deltastreamer & datasource internally call) are [here](configurations.md). Invoking _\--help_ on any tool such as DeltaStreamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
+
+* For Spark DataSource, you can use the "options" API of DataFrameWriter to pass in these configs.
+
+```scala
+inputDF.write().format("org.apache.hudi")
+ .options(clientOpts) // any of the Hudi client opts can be passed in as well
+ .option("hoodie.datasource.write.recordkey.field", "_row_key")
+ ...
+```
+
+* When using `HoodieWriteClient` directly, you can simply construct HoodieWriteConfig object with the configs in the link you mentioned.
+* When using HoodieDeltaStreamer tool to ingest, you can set the configs in properties file and pass the file as the cmdline argument "_\--props_"
+
+### How to create Hive style partition folder structure?
+
+By default Hudi creates the partition folders with just the partition values, but if would like to create partition folders similar to the way Hive will generate the structure, with paths that contain key value pairs, like country=us/… or datestr=2021-04-20. This is Hive style (or format) partitioning. The paths include both the names of the partition keys and the values that each path represents.
+
+To enable hive style partitioning, you need to add this hoodie config when you write your data:
+
+```plain
+hoodie.datasource.write.hive_style_partitioning: true
+```
+
+### Can I register my Hudi table with Apache Hive metastore?
+
+Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore.md#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations.md#hoodiedatasourcehive_syncenable).
+
+### What's Hudi's schema evolution story?
+
+Hudi uses Avro as the internal canonical representation for records, primarily due to its nice [schema compatibility & evolution](https://docs.confluent.io/platform/current/schema-registry/avro.html) properties. This is a key aspect of having reliability in your ingestion or ETL pipelines. As long as the schema passed to Hudi (either explicitly in Hudi Streamer schema provider configs or implicitly by Spark Datasource's Dataset schemas) is backwards compatible (e.g no field deletes, only [...]
+
+Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution.md) for more details on Schema-on-read for Spark..
+
+### What performance/ingest latency can I expect for Hudi writing?
+
+The speed at which you can write into Hudi depends on the [write operation](write_operations.md) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when [...]
+
+| Storage Type | Type of workload | Performance | Tips |
+| ---| ---| ---| --- |
+| copy on write | bulk\_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism) to get right number of files. use insert if you want this auto tuned . Configure [hoodie.bulkinsert.sort.mode](configurations.md#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value NONE offers the fastest performance and matches `spark.write.parqu [...]
+| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations.md#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
+| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or \<50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For e.g, if a partition has 1000 f [...]
+| merge on read | bulk insert | Currently new data only goes to parquet files and thus performance here should be similar to copy\_on\_write bulk insert. This has the nice side-effect of getting data into parquet directly for query performance. [This GitHub issue](https://github.com/apache/hudi/issues/14468) will add support for logging inserts directly and this up drastically. | |
+| merge on read | insert | Similar to above | |
+| merge on read | upsert/ de-duplicate & insert | Indexing performance would remain the same as copy-on-write, while ingest latency for updates (costliest I/O operation in copy\_on\_write) are sent to log files and thus with asynchronous compaction provides very very good ingest performance with low write amplification. | |
+
+Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide.md) for more tips on JVM and other configurations.
+
+### What performance can I expect for Hudi reading/queries?
+
+* For ReadOptimized views, you can expect the same best in-class columnar query performance as a standard parquet table in Hive/Spark/Presto
+* For incremental views, you can expect speed up relative to how much data usually changes in a given time window and how much time your entire scan takes. For e.g, if only 100 files changed in the last hour in a partition of 1000 files, then you can expect a speed of 10x using incremental pull in Hudi compared to full scanning the partition to find out new data.
+* For real time views, you can expect performance similar to the same avro backed table in Hive/Spark/Presto
+
+### How do I to avoid creating tons of small files?
+
+A key design decision in Hudi was to avoid creating small files and always write properly sized files.
+
+There are 2 ways to avoid creating tons of small files in Hudi and both of them have different trade-offs:
+
+a) **Auto Size small files during ingestion**: This solution trades ingest/writing time to keep queries always efficient. Common approaches to writing very small files and then later stitching them together only solve for system scalability issues posed by small files and also let queries slow down by exposing small files to them anyway.
+
+Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk\_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
+
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations.md#hoodieparquetmaxfilesize) and the [soft limit](configurations.md#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on p [...]
+
+For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
+
+* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations.md#hoodielogfilemaxsize) and a [factor](configurations.md#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
+* Indexes with **canIndexLogFiles = false** : Inserts of new data go only to parquet files. In this case, the same configurations as above for the COPY\_ON\_WRITE case applies.
+
+NOTE : In either case, small files will be auto sized only if there is no PENDING compaction or associated log file for that particular file slice. For example, for case 1: If you had a log file and a compaction C1 was scheduled to convert that log file to parquet, no more inserts can go into that log file. For case 2: If you had a parquet file and an update ended up creating an associated delta log file, no more inserts can go into that parquet file. Only after the compaction has been p [...]
+
+b) [**Clustering**](/blog/2021/01/27/hudi-clustering-intro) : This is a feature in Hudi to group small files into larger ones either synchronously or asynchronously. Since first solution of auto-sizing small files has a tradeoff on ingestion speed (since the small files are sized during ingestion), if your use-case is very sensitive to ingestion latency where you don't want to compromise on ingestion speed which may end up creating a lot of small files, clustering comes to the rescue. Cl [...]
+
+_Please note that Hudi always creates immutable files on disk. To be able to do auto-sizing or clustering, Hudi will always create a newer version of the smaller file, resulting in 2 versions of the same file. The cleaner service will later kick in and delte the older version small file and keep the latest one._
+
+### How do I use DeltaStreamer or Spark DataSource API to write to a Non-partitioned Hudi table ?
+
+Hudi supports writing to non-partitioned tables. For writing to a non-partitioned Hudi table and performing hive table syncing, you need to set the below configurations in the properties passed:
+
+```plain
+hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator
+hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPartitionedExtractor
+```
+
+### How can I reduce table versions created by Hudi in AWS Glue Data Catalog/ metastore?
+
+With each commit, Hudi creates a new table version in the metastore. This can be reduced by setting the option
+
+[hoodie.datasource.meta\_sync.condition.sync](configurations.md#hoodiedatasourcemeta_syncconditionsync) to true.
+
+This will ensure that hive sync is triggered on schema or partitions changes.
+
+### If there are failed writes in my timeline, do I see duplicates?
+
+No, Hudi does not expose uncommitted files/blocks to the readers. Further, Hudi strives to automatically manage the table for the user, by actively cleaning up files created from failed/aborted writes. See [marker mechanism](/blog/2021/08/18/improving-marker-mechanism/).
+
+### How are conflicts detected in Hudi between multiple writers?
+
+Hudi employs [optimistic concurrency control](concurrency_control.md) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in “[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/ [...]
+
+### Can single-writer inserts have duplicates?
+
+By default, Hudi turns off key based de-duplication for INSERT/BULK\_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations.md#hoodiecombinebeforeinsert) or in [...]
+
+### Can concurrent inserts cause duplicates?
+
+Yes. As mentioned before, the default conflict detection strategy only check for conflicting updates to the same file group IDs. In the case of concurrent inserts, inserted records end up creating new file groups and thus can go undetected. Most common workload patterns use multi-writer capability in the case of running ingestion of new data and concurrently backfilling/deleting older data, with NO overlap in the primary keys of the records. However, this can be implemented (or better ye [...]
diff --git a/website/versioned_docs/version-0.15.1/file_layouts.md b/website/versioned_docs/version-0.15.1/file_layouts.md
new file mode 100644
index 000000000000..d510f9e7d38b
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/file_layouts.md
@@ -0,0 +1,20 @@
+---
+title: File Layouts
+toc: false
+---
+
+The following describes the general file layout structure for Apache Hudi. Please refer the **[tech spec](https://hudi.apache.org/tech-specs#file-layout-hierarchy)** for a more detailed description of the file layouts.
+* Hudi organizes data tables into a directory structure under a base path on a distributed file system
+* Tables are broken up into partitions
+* Within each partition, files are organized into file groups, uniquely identified by a file ID
+* Each file group contains several file slices
+* Each slice contains a base file (*.parquet/*.orc) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) ) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
+
+Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction.md) action merges logs and base files to produce new
+file slices and [cleaning](hoodie_cleaner.md) action gets rid of unused/older file slices to reclaim space on the file system.
+
+
+
+### Configs
+
+Please refer [here](https://hudi.apache.org/docs/next/configurations/#Layout-Configs) for additional configs that control storage layout and data distribution, which defines how the files are organized within a table.
diff --git a/website/versioned_docs/version-0.15.1/file_sizing.md b/website/versioned_docs/version-0.15.1/file_sizing.md
new file mode 100644
index 000000000000..61356932afbb
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/file_sizing.md
@@ -0,0 +1,177 @@
+---
+title: "File Sizing"
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+---
+
+Solving the [small file problem](https://hudi.apache.org/blog/2021/03/01/hudi-file-sizing/) is fundamental to ensuring
+great experience on the data lake. If you don’t size the files appropriately, you can slow down the queries and the pipelines.
+Some of the issues you may encounter with small files include the following:
+
+- **Queries slow down**: You’ll have to scan through many small files to retrieve data for a query. It’s a very inefficient
+ way of accessing and utilizing the data. Also, cloud storage, like S3, enforces a rate-limit on how many requests can
+ be processed per second per prefix in a bucket. A higher number of files, i.e., at least one request per file regardless
+ of the file size, increases the chance of encountering a rate-limit, as well as additional fixed costs for opening/closing
+ them. All of these causes the queries to slow down.
+
+- **Pipelines slow down**: You can slow down your Spark, Flink or Hive jobs due to excessive scheduling overhead or memory
+ requirements; the more files you have, the more tasks you create.
+
+- **Storage inefficiencies**: When working with many small files, you can be inefficient in using your storage. For example,
+ many small files can yield a lower compression ratio, increasing storage costs. If you’re indexing the data, that also
+ takes up more storage space to store additional metadata, such as column statistics. If you’re working with a smaller
+ amount of data, you might not see a significant impact with storage. However, when dealing with petabyte and exabyte
+ data, you’ll need to be efficient in managing storage resources.
+
+A critical design decision in the Hudi architecture is to avoid small file creation. Hudi is uniquely designed to write
+appropriately sized files automatically. This page will show you how Apache Hudi overcomes the dreaded small files problem.
+There are two ways to manage small files in Hudi:
+
+- [Auto-size during writes](#auto-sizing-during-writes)
+- [Clustering after writes](#auto-sizing-with-clustering)
+
+Below, we will describe the advantages and trade-offs of each.
+
+:::note
+the bulk_insert write operation does not have auto-sizing capabilities during ingestion
+:::
+
+## Auto-sizing during writes
+
+You can manage file sizes through Hudi’s auto-sizing capability during ingestion. The default targeted file size for
+Parquet base files is 120MB, which can be configured by `hoodie.parquet.max.file.size`. Auto-sizing may add some write
+latency, but it ensures that the queries are always efficient when a write transaction is committed. It’s important to
+note that if you don’t manage file sizing as you write and, instead, try to run clustering to fix your file sizing
+periodically, your queries might be slow until the point when the clustering finishes. This is only supported for
+**append** use cases only; **mutable** are not supported at the moment. Please refer to the
+[clustering documentation](https://hudi.apache.org/docs/clustering) for more details.
+
+
+
+If you need to control the file sizing, i.e., increase the target file size or change how small files are identified,
+follow the instructions below for Copy-On-Write and Merge-On-Read tables.
+
+### File sizing for Copy-On-Write (COW) and Merge-On-Read (MOR) tables
+To tune the file sizing for both COW and MOR tables, you can set the small file limit and the maximum Parquet file size.
+Hudi will try to add enough records to a small file at write time to get it to the configured maximum limit.
+
+ - For example, if the `hoodie.parquet.small.file.limit=104857600` (100MB) and `hoodie.parquet.max.file.size=125829120` (120MB),
+ Hudi will pick all files < 100MB and try to get them up to 120MB.
+
+For creating a Hudi table initially, setting an accurate record size estimate is vital to ensure Hudi can adequately
+estimate how many records need to be bin-packed in a Parquet file for the first ingestion batch. Then, Hudi automatically
+uses the average record size for subsequent writes based on previous commits.
+
+
+### More details about file sizing for Merge-On-Read(MOR) tables
+As a MOR table aims to reduce the write amplification, compared to a COW table, when writing to a MOR table, Hudi limits
+the number of Parquet base files to one for auto file sizing during insert and upsert operation. This limits the number
+of rewritten files. This can be configured through `hoodie.merge.small.file.group.candidates.limit`.
+
+For storage systems that support append operation, in addition to file sizing Parquet base files for a MOR table, you
+can also tune the log files file-sizing with `hoodie.logfile.max.size`.
+
+MergeOnRead works differently for different INDEX choices so there are few more configs to set:
+
+- Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure
+ the [maximum log size](https://hudi.apache.org/docs/configurations#hoodielogfilemaxsize) and a
+ [factor](https://hudi.apache.org/docs/configurations#hoodielogfiletoparquetcompressionratio) that denotes reduction
+ in size when data moves from avro to parquet files.
+- Indexes with **canIndexLogFiles = false** : Inserts of new data go only to parquet files. In this case, the same configurations
+ as above for the COPY_ON_WRITE case applies.
+**NOTE** : In either case, small files will be auto sized only if there is no PENDING compaction or associated log file
+ for that particular file slice. For example, for case 1: If you had a log file and a compaction C1 was scheduled to
+ convert that log file to parquet, no more inserts can go into that log file. For case 2: If you had a parquet file and
+ an update ended up creating an associated delta log file, no more inserts can go into that parquet file. Only after the
+ compaction has been performed and there are NO log files associated with the base parquet file, can new inserts be sent
+ to auto size that parquet file.
+
+### Configs
+Here are the essential configurations for **COW tables**.
+
+**Spark based configs:**
+
+| Config Name | Default | Description [...]
+|-----------------------------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
+| hoodie.parquet.small.file.limit | 104857600 (Optional) | During an insert and upsert operation, we opportunistically expand existing small files on storage, instead of writing new files, to keep number of files to an optimum. This config sets the file size limit below which a file on storage becomes a candidate to be selected as such a `small file`. By default, treat any file <= 100MB as a small file. Also note that if this set <= 0, will not try to get small files and d [...]
+| hoodie.parquet.max.file.size | 125829120 (Optional) | Target size in bytes for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /><br />`Config Param: PARQUET_MAX_FILE_SIZE` [...]
+| hoodie.copyonwrite.record.size.estimate | 1024 (Optional) | The average record size. If not explicitly specified, hudi will compute the record size estimate compute dynamically based on commit metadata. This is critical in computing the insert parallelism and bin-packing inserts into small files.<br /><br />`Config Param: COPY_ON_WRITE_RECORD_SIZE_ESTIMATE` [...]
+
+**Flink based configs:**
+
+| Config Name | Default | Description |
+|-----------------------------------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| write.parquet.max.file.size | 120 (Optional) | Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /><br /> `Config Param: WRITE_PARQUET_MAX_FILE_SIZE` |
+
+Here are the essential configurations for **MOR tables**:
+
+**Spark based configs:**
+
+| Config Name | Default | Description [...]
+|------------------------------------------------|-----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| hoodie.parquet.small.file.limit | 104857600 (Optional) | During an insert and upsert operation, we opportunistically expand existing small files on storage, instead of writing new files, to keep number of files to an optimum. This config sets the file size limit below which a file on storage becomes a candidate to be selected as such a `small file`. By default, treat any file <= 100MB as a small file. Also note that if this set <= 0, will not try to get small fil [...]
+| hoodie.parquet.max.file.size | 125829120 (Optional) | Target size in bytes for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /><br />`Config Param: PARQUET_MAX_FILE_SIZE` [...]
+| hoodie.merge.small.file.group.candidates.limit | 1 (Optional) | Limits number of file groups, whose base file satisfies small-file limit, to consider for appending records during upsert operation. Only applicable to MOR tables<br /><br />`Config Param: MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT` [...]
+| hoodie.logfile.max.size | 1073741824 (Optional) | LogFile max size in bytes. This is the maximum size allowed for a log file before it is rolled over to the next version. This log rollover limit only works on storage systems that support append operation. Please note that on cloud storage like S3/GCS, this may not be respected<br /><br />`Config Param: LOGFILE_MAX_SIZE` [...]
+| hoodie.logfile.to.parquet.compression.ratio | 0.35 (Optional) | Expected additional compression as records move from log files to parquet. Used for merge_on_read table to send inserts into log files & control the size of compacted parquet file.<br /><br />`Config Param: LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION` [...]
+
+
+**Flink based configs:**
+
+| Config Name | Default | Description |
+|-----------------------------------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| write.parquet.max.file.size | 120 (Optional) | Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /><br /> `Config Param: WRITE_PARQUET_MAX_FILE_SIZE` |
+| write.log.max.size | 1024 (Optional) | Maximum size allowed in MB for a log file before it is rolled over to the next version, default 1GB<br /><br /> `Config Param: WRITE_LOG_MAX_SIZE` |
+
+
+## Auto-Sizing With Clustering
+Clustering is a service that allows you to combine small files into larger ones while at the same time (optionally) changing
+the data layout by sorting or applying [space-filling curves](https://hudi.apache.org/blog/2021/12/29/hudi-zorder-and-hilbert-space-filling-curves/)
+like Z-order or Hilbert curve. We won’t go into all the details about clustering here, but please refer to the
+[clustering section](https://hudi.apache.org/docs/clustering) for more details.
+
+Clustering is one way to achieve file sizing, so you can have faster queries. When you ingest data, you may still have a
+lot of small files (depending on your configurations and the data size from ingestion i.e., input batch). In this case,
+you will want to cluster all the small files to larger files to improve query performance. Clustering can be performed
+in different ways. Please check out the [clustering documentation](https://hudi.apache.org/docs/clustering) for more details.
+
+An example where clustering might be very useful is when a user has a Hudi table with many small files. For example, if
+you're using BULK_INSERT without any sort modes, or you want a different file layout, you can use the clustering service
+to fix all the file sizes without ingesting any new data.
+
+:::note
+Clustering in Hudi is not a blocking operation, and writes can continue concurrently as long as no files need to be
+updated while the clustering service is running. The writes will fail if there are updates to the data being clustered
+while the clustering service runs.
+:::
+
+:::note
+Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner.md)
+will later kick in and delete the older version small file and keep the latest one.
+:::
+
+Here are the critical file sizing configurations:
+
+### Configs
+
+**Spark based configs:**
+
+| Config Name | Default | Description |
+|-------------------------------------------------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.clustering.plan.strategy.small.file.limit | 314572800 (Optional) | Files smaller than the size in bytes specified here are candidates for clustering<br /><br />`Config Param: PLAN_STRATEGY_SMALL_FILE_LIMIT`<br />`Since Version: 0.7.0` |
+| hoodie.clustering.plan.strategy.target.file.max.bytes | 1073741824 (Optional) | Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups<br /><br />`Config Param: PLAN_STRATEGY_TARGET_FILE_MAX_BYTES`<br />`Since Version: 0.7.0` |
+
+**Flink based configs:**
+
+| Config Name | Default | Description |
+|-------------------------------------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| clustering.plan.strategy.small.file.limit | 600 (Optional) | Files smaller than the size specified here are candidates for clustering, default 600 MB<br /><br /> `Config Param: CLUSTERING_PLAN_STRATEGY_SMALL_FILE_LIMIT` |
+| clustering.plan.strategy.target.file.max.bytes | 1073741824 (Optional)| Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups, default 1 GB<br /><br /> `Config Param: CLUSTERING_PLAN_STRATEGY_TARGET_FILE_MAX_BYTES` |
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Mastering File Sizing in Hudi: Boosting Performance and Efficiency](https://www.youtube.com/watch?v=qg-2aYyvfts)
+* ["How do I Ingest Extremely Small Files into Hudi Data lake with Glue Incremental data processing](https://www.youtube.com/watch?v=BvoLVeidd-0)
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.15.1/flink-quick-start-guide.md b/website/versioned_docs/version-0.15.1/flink-quick-start-guide.md
new file mode 100644
index 000000000000..9bec4622d3d1
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/flink-quick-start-guide.md
@@ -0,0 +1,473 @@
+---
+title: "Flink Quick Start"
+toc: true
+last_modified_at: 2023-08-16T12:53:57+08:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This page introduces Flink-Hudi integration. We can feel the unique charm of how Flink brings in the power of streaming into Hudi.
+
+## Setup
+
+
+### Flink Support Matrix
+
+
+| Hudi | Supported Flink version |
+|:----------------|:----------------------------------------|
+| 0.15.x | 1.14.x, 1.15.x, 1.16.x, 1.17.x, 1.18.x |
+| 0.14.x | 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x |
+| 0.13.x | 1.13.x, 1.14.x, 1.15.x, 1.16.x |
+| 0.12.x | 1.13.x, 1.14.x, 1.15.x |
+| 0.11.x | 1.13.x, 1.14.x |
+
+
+### Download Flink and Start Flink cluster
+
+Hudi works with Flink 1.13 (up to Hudi 0.14.x release), Flink 1.14, Flink 1.15, Flink 1.16, Flink 1.17, and Flink 1.18.
+You can follow the instructions [here](https://flink.apache.org/downloads) for setting up Flink. Then, start a standalone Flink cluster
+within hadoop environment. In case we are trying on local setup, then we could download hadoop binaries and set HADOOP_HOME.
+
+```bash
+# HADOOP_HOME is your hadoop root directory after unpack the binary package.
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+
+# Start the Flink standalone cluster
+./bin/start-cluster.sh
+```
+<div className="notice--info">
+ <h4>Please note the following: </h4>
+<ul>
+ <li>We suggest hadoop 2.9.x+ version because some of the object storage has filesystem implementation only after that</li>
+ <li>The flink-parquet and flink-avro formats are already packaged into the hudi-flink-bundle jar</li>
+</ul>
+</div>
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+<TabItem value="flinksql">
+
+We use the [Flink Sql Client](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sqlclient/) because it's a good
+quick start tool for SQL users.
+
+### Start Flink SQL client
+
+Hudi supports packaged bundle jar for Flink, which should be loaded in the Flink SQL Client when it starts up.
+You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore.md#install)), or download it from the
+[Apache Official Repository](https://repo.maven.apache.org/maven2/org/apache/hudi/).
+
+Now start the SQL CLI:
+
+```bash
+# For Flink versions: 1.13 - 1.18
+export FLINK_VERSION=1.17
+export HUDI_VERSION=0.15.0
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-flink${FLINK_VERSION}-bundle/${HUDI_VERSION}/hudi-flink${FLINK_VERSION}-bundle-${HUDI_VERSION}.jar -P /tmp/
+./bin/sql-client.sh embedded -j /tmp/hudi-flink${FLINK_VERSION}-bundle-${HUDI_VERSION}.jar shell
+```
+
+Setup table name, base path and operate using SQL for this guide.
+The SQL CLI only executes the SQL line by line.
+</TabItem>
+
+<TabItem value="dataStream">
+
+Hudi works with Flink 1.13 (up to Hudi 0.14.x release), Flink 1.14, Flink 1.15, Flink 1.16, Flink 1.17, and Flink 1.18.
+Please add the desired dependency to your project:
+```xml
+<!-- For Flink versions 1.13 - 1.18-->
+<properties>
+ <flink.version>1.17.0</flink.version>
+ <flink.binary.version>1.17</flink.binary.version>
+ <hudi.version>0.15.0</hudi.version>
+</properties>
+<dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-flink${flink.binary.version}-bundle</artifactId>
+ <version>${hudi.version}</version>
+</dependency>
+```
+
+</TabItem>
+
+</Tabs
+>
+
+## Create Table
+
+First, let's create a Hudi table. Here, we use a partitioned table for illustration, but Hudi also supports non-partitioned tables.
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+Here is an example of creating a flink Hudi table.
+
+```sql
+-- sets up the result mode to tableau to show the results directly in the CLI
+set sql-client.execution.result-mode = tableau;
+DROP TABLE hudi_table;
+CREATE TABLE hudi_table(
+ ts BIGINT,
+ uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
+ rider VARCHAR(20),
+ driver VARCHAR(20),
+ fare DOUBLE,
+ city VARCHAR(20)
+)
+PARTITIONED BY (`city`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'file:///tmp/hudi_table',
+ 'table.type' = 'MERGE_ON_READ'
+);
+```
+
+</TabItem>
+
+<TabItem value="dataStream">
+
+```java
+// Java
+// First commit will auto-initialize the table, if it did not exist in the specified base path.
+```
+
+</TabItem>
+
+
+</Tabs
+>
+
+## Insert Data
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+Insert data into the Hudi table using SQL `VALUES`.
+
+```sql
+-- insert data using values
+INSERT INTO hudi_table
+VALUES
+(1695159649087,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554788,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462179,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066204,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137016,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo'),
+(1695376420876,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo'),
+(1695173887231,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai'),
+(1695115999911,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+```
+</TabItem>
+
+<TabItem value="dataStream">
+
+Add some streaming source to flink and load the data in hudi table. Since, this is the first write, it will also auto-create the table.
+
+```java
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.data.RowData;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.util.HoodiePipeline;
+
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+String targetTable = "hudi_table";
+String basePath = "file:///tmp/hudi_table";
+
+Map<String, String> options = new HashMap<>();
+options.put("path", basePath);
+options.put("table.type", HoodieTableType.MERGE_ON_READ.name());
+options.put("precombine.field", "ts");
+
+DataStream<RowData> dataStream = env.addSource(...);
+HoodiePipeline.Builder builder = HoodiePipeline.builder(targetTable)
+ .column("uuid VARCHAR(20)")
+ .column("name VARCHAR(10)")
+ .column("age INT")
+ .column("ts TIMESTAMP(3)")
+ .column("`partition` VARCHAR(20)")
+ .pk("uuid")
+ .partition("partition")
+ .options(options);
+
+builder.sink(dataStream, false); // The second parameter indicating whether the input data stream is bounded
+env.execute("Api_Sink");
+```
+Refer Full Quickstart Example [here](https://github.com/ad1happy2go/hudi-examples/blob/main/flink/src/main/java/com/hudi/flink/quickstart/HudiDataStreamWriter.java)
+
+</TabItem>
+
+</Tabs
+>
+
+## Query Data
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+```sql
+-- query from the Hudi table
+select * from hudi_table;
+```
+</TabItem>
+
+<TabItem value="dataStream">
+
+```java
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.data.RowData;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.util.HoodiePipeline;
+
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+String targetTable = "hudi_table";
+String basePath = "file:///tmp/hudi_table";
+
+Map<String, String> options = new HashMap<>();
+options.put("path", basePath);
+options.put("table.type", HoodieTableType.MERGE_ON_READ.name());
+options.put("read.streaming.enabled", "true"); // this option enable the streaming read
+options.put("read.start-commit", "20210316134557"); // specifies the start commit instant time
+
+HoodiePipeline.Builder builder = HoodiePipeline.builder(targetTable)
+ .column("uuid VARCHAR(20)")
+ .column("name VARCHAR(10)")
+ .column("age INT")
+ .column("ts TIMESTAMP(3)")
+ .column("`partition` VARCHAR(20)")
+ .pk("uuid")
+ .partition("partition")
+ .options(options);
+
+DataStream<RowData> rowDataDataStream = builder.source(env);
+rowDataDataStream.print();
+env.execute("Api_Source");
+```
+Refer Full Streaming Reader Example [here](https://github.com/ad1happy2go/hudi-examples/blob/main/flink/src/main/java/com/hudi/flink/quickstart/HudiDataStreamReader.java)
+
+</TabItem>
+
+</Tabs
+>
+
+This statement queries snapshot view of the dataset.
+Refers to [Table types and queries](concepts.md#table-types--queries) for more info on all table types and query types supported.
+
+## Update Data
+
+This is similar to inserting new data.
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+Hudi tables can be updated by either inserting reocrds with same primary key or using a standard UPDATE statement shown as below.
+
+```sql
+-- Update Queries only works with batch execution mode
+SET 'execution.runtime-mode' = 'batch';
+UPDATE hudi_table SET fare = 25.0 WHERE uuid = '334e26e9-8355-45cc-97c6-c31daf0df330';
+```
+
+:::note
+The `UPDATE` statement is supported since Flink 1.17, so only Hudi Flink bundle compiled with Flink 1.17+ supplies this functionality.
+Only **batch** queries on Hudi table with primary key work correctly.
+:::
+</TabItem>
+
+<TabItem value="dataStream">
+
+Add some streaming source to flink and load the data in hudi table using DataStream API as [above](#insert-data).
+When new rows with the same primary key arrive in stream, then it will be be updated.
+In the insert example incoming row with same record id will be updated.
+
+Refer Update Example [here](https://github.com/ad1happy2go/hudi-examples/blob/main/flink/src/main/java/com/hudi/flink/quickstart/HudiDataStreamWriter.java)
+
+</TabItem>
+
+</Tabs
+>
+
+[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts.md)
+denoted by the timestamp.
+
+
+## Delete Data {#deletes}
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+### Row-level Delete
+
+When consuming data in streaming query, Hudi Flink source can also accept the change logs from the upstream data source if the `RowKind` is set up per-row,
+it can then apply the UPDATE and DELETE in row level. You can then sync a NEAR-REAL-TIME snapshot on Hudi for all kinds
+of RDBMS.
+
+### Batch Delete
+
+```sql
+-- delete all the records with age greater than 23
+-- NOTE: only works for batch sql queries
+SET 'execution.runtime-mode' = 'batch';
+DELETE FROM t1 WHERE age > 23;
+```
+
+:::note
+The `DELETE` statement is supported since Flink 1.17, so only Hudi Flink bundle compiled with Flink 1.17+ supplies this functionality.
+Only **batch** queries on Hudi table with primary key work correctly.
+:::
+
+</TabItem>
+
+<TabItem value="dataStream">
+
+Creates a Flink Hudi table first and insert data into the Hudi table using DataStream API as below.
+When new rows with the same primary key and Row Kind as Delete arrive in stream, then it will be be deleted.
+
+Refer Delete Example [here](https://github.com/ad1happy2go/hudi-examples/blob/main/flink/src/main/java/com/hudi/flink/quickstart/HudiDataStreamWriter.java)
+
+</TabItem
+>
+
+</Tabs
+>
+
+## Streaming Query
+
+Hudi Flink also provides capability to obtain a stream of records that changed since given commit timestamp.
+This can be achieved using Hudi's streaming querying and providing a start time from which changes need to be streamed.
+We do not need to specify endTime, if we want all changes after the given commit (as is the common case).
+
+```sql
+CREATE TABLE t1(
+ uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = '${path}',
+ 'table.type' = 'MERGE_ON_READ',
+ 'read.streaming.enabled' = 'true', -- this option enable the streaming read
+ 'read.start-commit' = '20210316134557', -- specifies the start commit instant time
+ 'read.streaming.check-interval' = '4' -- specifies the check interval for finding new source commits, default 60s.
+);
+
+-- Then query the table in stream mode
+select * from t1;
+```
+
+## Change Data Capture Query
+
+Hudi Flink also provides capability to obtain a stream of records with Change Data Capture.
+CDC queries are useful for applications that need to obtain all the changes, along with before/after images of records.
+
+```sql
+set sql-client.execution.result-mode = tableau;
+
+CREATE TABLE hudi_table(
+ ts BIGINT,
+ uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
+ rider VARCHAR(20),
+ driver VARCHAR(20),
+ fare DOUBLE,
+ city VARCHAR(20)
+)
+PARTITIONED BY (`city`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'file:///tmp/hudi_table',
+ 'table.type' = 'COPY_ON_WRITE',
+ 'cdc.enabled' = 'true' -- this option enable the cdc log enabled
+);
+-- insert data using values
+INSERT INTO hudi_table
+VALUES
+(1695159649087,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554788,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462179,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066204,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137016,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo'),
+(1695376420876,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo'),
+(1695173887231,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai'),
+(1695115999911,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+SET 'execution.runtime-mode' = 'batch';
+UPDATE hudi_table SET fare = 25.0 WHERE uuid = '334e26e9-8355-45cc-97c6-c31daf0df330';
+-- Query the table in stream mode in another shell to see change logs
+SET 'execution.runtime-mode' = 'streaming';
+select * from hudi_table/*+ OPTIONS('read.streaming.enabled'='true')*/;
+```
+
+This will give all changes that happened after the `read.start-commit` commit. The unique thing about this
+feature is that it now lets you author streaming pipelines on streaming or batch data source.
+
+## Where To Go From Here?
+- **Quick Start** : Read [Quick Start](flink-quick-start-guide.md#setup) to get started quickly Flink sql client to write to(read from) Hudi.
+- **Configuration** : For [Global Configuration](flink_tuning.md#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning.md#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink.md#cdc-ingestion), [Bulk Insert](ingestion_flink.md#bulk-insert), [Index Bootstrap](ingestion_flink.md#index-bootstrap), [Changelog Mode](ingestion_flink.md#changelog-mode) and [Append Mode](ingestion_flink.md#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml.md#non-blocking-concurrency-control-experimental).
+- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries.md#streaming-query) and [Incremental Query](sql_queries.md#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning.md#memory-optimization) and [Write Rate Limit](flink_tuning.md#write-rate-limit).
+- **Optimization**: Offline compaction is supported [Offline Compaction](compaction.md#flink-offline-compaction).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore.md#flink-setup), [Presto Query](sql_queries.md#presto).
+- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](sql_ddl/#create-catalog).
+
+If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
+ - [Hudi Timeline](timeline.md) – How Hudi manages transactions and other table services
+ - [Hudi File Layout](file_layouts.md) - How the files are laid out on storage
+ - [Hudi Table Types](table_types.md) – `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types.md#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
+
+See more in the "Concepts" section of the docs.
+
+Take a look at recent [blog posts](/blog) that go in depth on certain topics or use cases.
+
+Hudi tables can be queried from query engines like Hive, Spark, Flink, Presto and much more. We have put together a
+[demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that show cases all of this on a docker based setup with all
+dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
+steps [here](docker_demo.md) to get a taste for it. Also, if you are looking for ways to migrate your existing data
+to Hudi, refer to [migration guide](migration_guide.md).
diff --git a/website/versioned_docs/version-0.15.1/flink_tuning.md b/website/versioned_docs/version-0.15.1/flink_tuning.md
new file mode 100644
index 000000000000..9c76d71f50f5
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/flink_tuning.md
@@ -0,0 +1,117 @@
+---
+title: Flink Tuning Guide
+toc: true
+---
+
+## Global Configurations
+When using Flink, you can set some global configurations in `$FLINK_HOME/conf/flink-conf.yaml`
+
+### Parallelism
+
+| Option Name | Default | Type | Description |
+| ----------- | ------- | ------- | ------- |
+| `taskmanager.numberOfTaskSlots` | `1` | `Integer` | The number of parallel operator or user function instances that a single TaskManager can run. We recommend setting this value > 4, and the actual value needs to be set according to the amount of data |
+| `parallelism.default` | `1` | `Integer` | The default parallelism used when no parallelism is specified anywhere (default: 1). For example, If the value of [`write.bucket_assign.tasks`](#parallelism-1) is not set, this value will be used |
+
+### Memory
+
+| Option Name | Default | Type | Description |
+| ----------- | ------- | ------- | ------- |
+| `jobmanager.memory.process.size` | `(none)` | `MemorySize` | Total Process Memory size for the JobManager. This includes all the memory that a JobManager JVM process consumes, consisting of Total Flink Memory, JVM Metaspace, and JVM Overhead |
+| `taskmanager.memory.task.heap.size` | `(none)` | `MemorySize` | Task Heap Memory size for TaskExecutors. This is the size of JVM heap memory reserved for write cache |
+| `taskmanager.memory.managed.size` | `(none)` | `MemorySize` | Managed Memory size for TaskExecutors. This is the size of off-heap memory managed by the memory manager, reserved for sorting and RocksDB state backend. If you choose RocksDB as the state backend, you need to set this memory |
+
+### Checkpoint
+
+| Option Name | Default | Type | Description |
+| ----------- | ------- | ------- | ------- |
+| `execution.checkpointing.interval` | `(none)` | `Duration` | Setting this value as `execution.checkpointing.interval = 150000ms`, 150000ms = 2.5min. Configuring this parameter is equivalent to enabling the checkpoint |
+| `state.backend` | `(none)` | `String` | The state backend to be used to store state. We recommend setting store state as `rocksdb` : `state.backend: rocksdb` |
+| `state.backend.rocksdb.localdir` | `(none)` | `String` | The local directory (on the TaskManager) where RocksDB puts its files |
+| `state.checkpoints.dir` | `(none)` | `String` | The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem. The storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers), like hdfs and oss path |
+| `state.backend.incremental` | `false` | `Boolean` | Option whether the state backend should create incremental checkpoints, if possible. For an incremental checkpoint, only a diff from the previous checkpoint is stored, rather than the complete checkpoint state. If store state is setting as `rocksdb`, recommending to turn on |
+
+## Table Options
+
+Flink SQL jobs can be configured through options in the `WITH` clause.
+The actual datasource level configs are listed below.
+
+### Memory
+
+:::note
+When optimizing memory, we need to pay attention to the memory configuration
+and the number of taskManagers, parallelism of write tasks (write.tasks : 4) first. After confirm each write task to be
+allocated with enough memory, we can try to set these memory options.
+:::
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.task.max.size` | Maximum memory in MB for a write task, when the threshold hits, it flushes the max size data bucket to avoid OOM. Default `1024MB` | `1024D` | The memory reserved for write buffer is `write.task.max.size` - `compaction.max_memory`. When total buffer of write tasks reach the threshold, the largest buffer in the memory will be flushed |
+| `write.batch.size` | In order to improve the efficiency of writing, Flink write task will cache data in buffer according to the write bucket until the memory reaches the threshold. When reached threshold, the data buffer would be flushed out. Default `64MB` | `64D` | Recommend to use the default settings |
+| `write.log_block.size` | The log writer of Hudi will not flush the data immediately after receiving data. The writer flush data to the disk in the unit of `LogBlock`. Before `LogBlock` reached threshold, records will be buffered in the writer in form of serialized bytes. Default `128MB` | `128` | Recommend to use the default settings |
+| `write.merge.max_memory` | If write type is `COPY_ON_WRITE`, Hudi will merge the incremental data and base file data. The incremental data will be cached and spilled to disk. this threshold controls the max heap size that can be used. Default `100MB` | `100` | Recommend to use the default settings |
+| `compaction.max_memory` | Same as `write.merge.max_memory`, but occurs during compaction. Default `100MB` | `100` | If it is online compaction, it can be turned up when resources are sufficient, such as setting as `1024MB` |
+
+### Parallelism
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.tasks` | The parallelism of writer tasks. Each write task writes 1 to `N` buckets in sequence. Default `4` | `4` | Increases the parallelism has no effect on the number of small files |
+| `write.bucket_assign.tasks` | The parallelism of bucket assigner operators. No default value, using Flink `parallelism.default` | [`parallelism.default`](#parallelism) | Increases the parallelism also increases the number of buckets, thus the number of small files (small buckets) |
+| `write.index_boostrap.tasks` | The parallelism of index bootstrap. Increasing parallelism can speed up the efficiency of the bootstrap stage. The bootstrap stage will block checkpointing. Therefore, it is necessary to set more checkpoint failure tolerance times. Default using Flink `parallelism.default` | [`parallelism.default`](#parallelism) | It only take effect when `index.bootsrap.enabled` is `true` |
+| `read.tasks` | The parallelism of read operators (batch and stream). Default `4` | `4` | |
+| `compaction.tasks` | The parallelism of online compaction. Default `4` | `4` | `Online compaction` will occupy the resources of the write task. It is recommended to use [`offline compaction`](compaction.md#flink-offline-compaction) |
+
+### Compaction
+
+:::note
+These are options only for `online compaction`.
+:::
+
+:::note
+Turn off online compaction by setting `compaction.async.enabled` = `false`, but we still recommend turning on `compaction.schedule.enable` for the writing job. You can then execute the compaction plan by [`offline compaction`](compaction.md#flink-offline-compaction).
+:::
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `compaction.schedule.enabled` | Whether to generate compaction plan periodically | `true` | Recommend to turn it on, even if `compaction.async.enabled` = `false` |
+| `compaction.async.enabled` | Async Compaction, enabled by default for MOR | `true` | Turn off `online compaction` by turning off this option |
+| `compaction.trigger.strategy` | Strategy to trigger compaction | `num_commits` | Options are `num_commits`: trigger compaction when reach N delta commits; `time_elapsed`: trigger compaction when time elapsed > N seconds since last compaction; `num_and_time`: trigger compaction when both `NUM_COMMITS` and `TIME_ELAPSED` are satisfied; `num_or_time`: trigger compaction when `NUM_COMMITS` or `TIME_ELAPSED` is satisfied. |
+| `compaction.delta_commits` | Max delta commits needed to trigger compaction, default `5` commits | `5` | -- |
+| `compaction.delta_seconds` | Max delta seconds time needed to trigger compaction, default `1` hour | `3600` | -- |
+| `compaction.max_memory` | Max memory in MB for compaction spillable map, default `100MB` | `100` | If your have sufficient resources, recommend to adjust to `1024MB` |
+| `compaction.target_io` | Target IO per compaction (both read and write), default `500GB`| `512000` | -- |
+
+## Memory Optimization
+
+### MOR
+
+1. [Setting Flink state backend to `rocksdb`](#checkpoint) (the default `in memory` state backend is very memory intensive).
+2. If there is enough memory, `compaction.max_memory` can be set larger (`100MB` by default, and can be adjust to `1024MB`).
+3. Pay attention to the memory allocated to each write task by taskManager to ensure that each write task can be allocated to the
+ desired memory size `write.task.max.size`. For example, taskManager has `4GB` of memory running two streamWriteFunction, so each write task
+ can be allocated with `2GB` memory. Please reserve some buffers because the network buffer and other types of tasks on taskManager (such as bucketAssignFunction) will also consume memory.
+4. Pay attention to the memory changes of compaction. `compaction.max_memory` controls the maximum memory that each task can be used when compaction tasks read
+ logs. `compaction.tasks` controls the parallelism of compaction tasks.
+
+### COW
+
+1. [Setting Flink state backend to `rocksdb`](#checkpoint) (the default `in memory` state backend is very memory intensive).
+2. Increase both `write.task.max.size` and `write.merge.max_memory` (`1024MB` and `100MB` by default, adjust to `2014MB` and `1024MB`).
+3. Pay attention to the memory allocated to each write task by taskManager to ensure that each write task can be allocated to the
+ desired memory size `write.task.max.size`. For example, taskManager has `4GB` of memory running two write tasks, so each write task
+ can be allocated with `2GB` memory. Please reserve some buffers because the network buffer and other types of tasks on taskManager (such as `BucketAssignFunction`) will also consume memory.
+
+
+## Write Rate Limit
+
+In the existing data synchronization, `snapshot data` and `incremental data` are send to kafka first, and then streaming write
+to Hudi by Flink. Because the direct consumption of `snapshot data` will lead to problems such as high throughput and serious
+disorder (writing partition randomly), which will lead to write performance degradation and throughput glitches. At this time,
+the `write.rate.limit` option can be turned on to ensure smooth writing.
+
+### Options
+
+| Option Name | Required | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.rate.limit` | `false` | `0` | Turn off by default |
diff --git a/website/versioned_docs/version-0.15.1/gcp_bigquery.md b/website/versioned_docs/version-0.15.1/gcp_bigquery.md
new file mode 100644
index 000000000000..cd61ec0eef09
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/gcp_bigquery.md
@@ -0,0 +1,103 @@
+---
+title: Google BigQuery
+keywords: [ hudi, gcp, bigquery ]
+summary: Introduce BigQuery integration in Hudi.
+---
+
+Hudi tables can be queried from [Google Cloud BigQuery](https://cloud.google.com/bigquery) as external tables. As of
+now, the Hudi-BigQuery integration only works for hive-style partitioned Copy-On-Write and Read-Optimized Merge-On-Read tables.
+
+## Sync Modes
+### Manifest File
+As of version 0.14.0, the `BigQuerySyncTool` supports syncing table to BigQuery using [manifests](https://cloud.google.com/blog/products/data-analytics/bigquery-manifest-file-support-for-open-table-format-queries). On the first run, the tool will create a manifest file representing the current base files in the table and a table in BigQuery based on the provided configurations. The tool produces a new manifest file on each subsequent run and will update the schema of the table in BigQuer [...]
+#### Benefits of using the new manifest approach:
+<ol>
+ <li>Only the files in the manifest can be scanned leading to less cost and better performance for your queries</li>
+ <li>The schema is now synced from the Hudi commit metadata allowing for proper schema evolution</li>
+ <li>Lists no longer have unnecessary nesting when querying in BigQuery as list inference is enabled by default</li>
+ <li>Partition column no longer needs to be dropped from the files due to new schema handling improvements</li>
+</ol>
+
+To enable this feature, set `hoodie.gcp.bigquery.sync.use_bq_manifest_file` to true.
+
+### View Over Files (Legacy)
+This is the current default behavior to preserve compatibility as users upgrade to 0.14.0 and beyond.
+After run, the sync tool will create 2 tables and 1 view in the target dataset in BigQuery. The tables and the view
+share the same name prefix, which is taken from the Hudi table name. Query the view for the same results as querying the
+Copy-on-Write Hudi table.
+**NOTE:** The view can scan all of the parquet files under your table's base path so it is recommended to upgrade to the manifest based approach for improved cost and performance.
+
+## Configurations
+
+Hudi uses `org.apache.hudi.gcp.bigquery.BigQuerySyncTool` to sync tables. It works with `HoodieStreamer` via
+setting sync tool class. A few BigQuery-specific configurations are required.
+
+| Config | Notes |
+|:---------------------------------------------|:----------------------------------------------------------------------------------------------------------------|
+| `hoodie.gcp.bigquery.sync.project_id` | The target Google Cloud project |
+| `hoodie.gcp.bigquery.sync.dataset_name` | BigQuery dataset name; create before running the sync tool |
+| `hoodie.gcp.bigquery.sync.dataset_location` | Region info of the dataset; same as the GCS bucket that stores the Hudi table |
+| `hoodie.gcp.bigquery.sync.source_uri` | A wildcard path pattern pointing to the first level partition; partition key can be specified or auto-inferred. Only required for partitioned tables |
+| `hoodie.gcp.bigquery.sync.source_uri_prefix` | The common prefix of the `source_uri`, usually it's the path to the Hudi table, trailing slash does not matter. |
+| `hoodie.gcp.bigquery.sync.base_path` | The usual basepath config for Hudi table. |
+| `hoodie.gcp.bigquery.sync.use_bq_manifest_file` | Set to true to enable the manifest based sync |
+| `hoodie.gcp.bigquery.sync.require_partition_filter` | Introduced in Hudi version 0.14.1, this configuration accepts a BOOLEAN value, with the default being false. When enabled (set to true), you must create a partition filter (a WHERE clause) for all queries, targeting the partitioning column of a partitioned table. Queries lacking such a filter will result in an error. |
+
+
+Refer to `org.apache.hudi.gcp.bigquery.BigQuerySyncConfig` for the complete configuration list.
+### Partition Handling
+In addition to the BigQuery-specific configs, you will need to use hive style partitioning for partition pruning in BigQuery. On top of that, the value in partition path will be the value returned for that field in your query. For example if you partition on a time-millis field, `time`, with an output format of `time=yyyy-MM-dd`, the query will return `time` values with day level granularity instead of the original milliseconds so keep this in mind while setting up your tables.
+
+```
+hoodie.datasource.write.hive_style_partitioning = 'true'
+```
+
+For the view based sync you must also specify the following configurations:
+```
+hoodie.datasource.write.drop.partition.columns = 'true'
+hoodie.partition.metafile.use.base.format = 'true'
+```
+
+## Example
+
+Below shows an example for running `BigQuerySyncTool` with `HoodieStreamer`.
+
+```shell
+spark-submit --master yarn \
+--packages com.google.cloud:google-cloud-bigquery:2.10.4 \
+--jars /opt/hudi-gcp-bundle-0.13.0.jar \
+--class org.apache.hudi.utilities.streamer.HoodieStreamer \
+/opt/hudi-utilities-bundle_2.12-0.13.0.jar \
+--target-base-path gs://my-hoodie-table/path \
+--target-table mytable \
+--table-type COPY_ON_WRITE \
+--base-file-format PARQUET \
+# ... other Hudi Streamer options
+--enable-sync \
+--sync-tool-classes org.apache.hudi.gcp.bigquery.BigQuerySyncTool \
+--hoodie-conf hoodie.streamer.source.dfs.root=gs://my-source-data/path \
+--hoodie-conf hoodie.gcp.bigquery.sync.project_id=hudi-bq \
+--hoodie-conf hoodie.gcp.bigquery.sync.dataset_name=rxusandbox \
+--hoodie-conf hoodie.gcp.bigquery.sync.dataset_location=asia-southeast1 \
+--hoodie-conf hoodie.gcp.bigquery.sync.table_name=mytable \
+--hoodie-conf hoodie.gcp.bigquery.sync.base_path=gs://rxusandbox/testcases/stocks/data/target/${NOW} \
+--hoodie-conf hoodie.gcp.bigquery.sync.partition_fields=year,month,day \
+--hoodie-conf hoodie.gcp.bigquery.sync.source_uri=gs://my-hoodie-table/path/year=* \
+--hoodie-conf hoodie.gcp.bigquery.sync.source_uri_prefix=gs://my-hoodie-table/path/ \
+--hoodie-conf hoodie.gcp.bigquery.sync.use_file_listing_from_metadata=true \
+--hoodie-conf hoodie.gcp.bigquery.sync.assume_date_partitioning=false \
+--hoodie-conf hoodie.datasource.hive_sync.mode=jdbc \
+--hoodie-conf hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://localhost:10000 \
+--hoodie-conf hoodie.datasource.hive_sync.skip_ro_suffix=true \
+--hoodie-conf hoodie.datasource.hive_sync.ignore_exceptions=false \
+--hoodie-conf hoodie.datasource.hive_sync.database=mydataset \
+--hoodie-conf hoodie.datasource.hive_sync.table=mytable \
+--hoodie-conf hoodie.datasource.write.recordkey.field=mykey \
+--hoodie-conf hoodie.datasource.write.partitionpath.field=year,month,day \
+--hoodie-conf hoodie.datasource.write.precombine.field=ts \
+--hoodie-conf hoodie.datasource.write.keygenerator.type=COMPLEX \
+--hoodie-conf hoodie.datasource.write.hive_style_partitioning=true \
+--hoodie-conf hoodie.datasource.write.drop.partition.columns=true \
+--hoodie-conf hoodie.partition.metafile.use.base.format=true \
+--hoodie-conf hoodie.metadata.enable=true \
+```
diff --git a/website/versioned_docs/version-0.15.1/gcs_hoodie.md b/website/versioned_docs/version-0.15.1/gcs_hoodie.md
new file mode 100644
index 000000000000..f0171aff16b5
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/gcs_hoodie.md
@@ -0,0 +1,60 @@
+---
+title: Google Cloud
+keywords: [ hudi, hive, google cloud, storage, spark, presto]
+summary: In this page, we go over how to configure hudi with Google Cloud Storage.
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+For Hudi storage on GCS, **regional** buckets provide an DFS API with strong consistency.
+
+## GCS Configs
+
+There are two configurations required for Hudi GCS compatibility:
+
+- Adding GCS Credentials for Hudi
+- Adding required jars to classpath
+
+### GCS Credentials
+
+Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the `fs.defaultFS` with your GCS bucket name and Hudi should be able to read/write from the bucket.
+
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>gs://hudi-bucket</value>
+ </property>
+
+ <property>
+ <name>fs.gs.impl</name>
+ <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem</value>
+ <description>The FileSystem for gs: (GCS) uris.</description>
+ </property>
+
+ <property>
+ <name>fs.AbstractFileSystem.gs.impl</name>
+ <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS</value>
+ <description>The AbstractFileSystem for gs: (GCS) uris.</description>
+ </property>
+
+ <property>
+ <name>fs.gs.project.id</name>
+ <value>GCS_PROJECT_ID</value>
+ </property>
+ <property>
+ <name>google.cloud.auth.service.account.enable</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>google.cloud.auth.service.account.email</name>
+ <value>GCS_SERVICE_ACCOUNT_EMAIL</value>
+ </property>
+ <property>
+ <name>google.cloud.auth.service.account.keyfile</name>
+ <value>GCS_SERVICE_ACCOUNT_KEYFILE</value>
+ </property>
+```
+
+### GCS Libs
+
+GCS hadoop libraries to add to our classpath
+
+- com.google.cloud.bigdataoss:gcs-connector:1.6.0-hadoop2
diff --git a/website/versioned_docs/version-0.15.1/hoodie_cleaner.md b/website/versioned_docs/version-0.15.1/hoodie_cleaner.md
new file mode 100644
index 000000000000..d3963038ac0a
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/hoodie_cleaner.md
@@ -0,0 +1,148 @@
+---
+title: Cleaning
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+---
+## Background
+Cleaning is a table service employed by Hudi to reclaim space occupied by older versions of data and keep storage costs
+in check. Apache Hudi provides snapshot isolation between writers and readers by managing multiple versioned files with **MVCC**
+concurrency. These file versions provide history and enable time travel and rollbacks, but it is important to manage
+how much history you keep to balance your costs. Cleaning service plays a crucial role in managing the tradeoff between
+retaining the long history of data and the associated storage costs.
+
+Hudi enables [Automatic Hudi cleaning](configurations.md#hoodiecleanautomatic) by default. Cleaning is invoked
+immediately after each commit, to delete older file slices. It's recommended to leave this enabled to ensure metadata
+and data storage growth is bounded. Cleaner can also be scheduled after every few commits instead of after every commit by
+configuring [hoodie.clean.max.commits](https://hudi.apache.org/docs/configurations#hoodiecleanmaxcommits).
+
+### Cleaning Retention Policies
+When cleaning old files, you should be careful not to remove files that are being actively used by long running queries.
+
+For spark based:
+
+| Config Name | Default | Description |
+|----------------------------------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
+| hoodie.cleaner.policy | KEEP_LATEST_COMMITS (Optional) | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. <br /><br />`Config Param: CLEANER_POLICY` |
+
+The corresponding config for Flink based engine is [`clean.policy`](https://hudi.apache.org/docs/configurations/#cleanpolicy).
+
+Hudi cleaner currently supports the below cleaning policies to keep a certain number of commits or file versions:
+
+- **KEEP_LATEST_COMMITS**: This is the default policy. This is a temporal cleaning policy that ensures the effect of
+ having lookback into all the changes that happened in the last X commits. Suppose a writer is ingesting data
+ into a Hudi dataset every 30 minutes and the longest running query can take 5 hours to finish, then the user should
+ retain at least the last 10 commits. With such a configuration, we ensure that the oldest version of a file is kept on
+ disk for at least 5 hours, thereby preventing the longest running query from failing at any point in time. Incremental
+ cleaning is also possible using this policy.
+ Number of commits to retain can be configured by [`hoodie.cleaner.commits.retained`](https://analytics.google.com/analytics/web/#/p300324801/reports/intelligenthome).
+ The corresponding Flink related config is [`clean.retain_commits`](https://hudi.apache.org/docs/configurations/#cleanretain_commits).
+
+- **KEEP_LATEST_FILE_VERSIONS**: This policy has the effect of keeping N number of file versions irrespective of time.
+ This policy is useful when it is known how many MAX versions of the file does one want to keep at any given time.
+ To achieve the same behaviour as before of preventing long running queries from failing, one should do their calculations
+ based on data patterns. Alternatively, this policy is also useful if a user just wants to maintain 1 latest version of the file.
+ Number of file versions to retain can be configured by [`hoodie.cleaner.fileversions.retained`](https://hudi.apache.org/docs/configurations/#hoodiecleanerfileversionsretained).
+ The corresponding Flink related config is [`clean.retain_file_versions`](https://hudi.apache.org/docs/configurations/#cleanretain_file_versions).
+
+- **KEEP_LATEST_BY_HOURS**: This policy clean up based on hours.It is simple and useful when knowing that you want to
+ keep files at any given time. Corresponding to commits with commit times older than the configured number of hours to
+ be retained are cleaned. Currently you can configure by parameter [`hoodie.cleaner.hours.retained`](https://hudi.apache.org/docs/configurations/#hoodiecleanerhoursretained).
+ The corresponding Flink related config is [`clean.retain_hours`](https://hudi.apache.org/docs/configurations/#cleanretain_hours).
+
+### Configs
+For details about all possible configurations and their default values see the [configuration docs](https://hudi.apache.org/docs/next/configurations/#Clean-Configs).
+For Flink related configs refer [here](https://hudi.apache.org/docs/next/configurations/#FLINK_SQL).
+
+### Ways to trigger Cleaning
+
+#### Inline
+
+By default, in Spark based writing, cleaning is run inline after every commit using the default policy of `KEEP_LATEST_COMMITS`. It's recommended
+to keep this enabled, to ensure metadata and data storage growth is bounded. To enable this, users do not have to set any configs. Following are the relevant basic configs.
+
+| Config Name | Default | Description |
+|----------------------------------| -----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.clean.automatic | true (Optional) | When enabled, the cleaner table service is invoked immediately after each commit, to delete older file slices. It's recommended to enable this, to ensure metadata and data storage growth is bounded.<br /><br />`Config Param: AUTO_CLEAN` |
+| hoodie.cleaner.commits.retained | 10 (Optional) | Number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br /><br />`Config Param: CLEANER_COMMITS_RETAINED` |
+
+
+#### Async
+In case you wish to run the cleaner service asynchronously along with writing, please enable the [`hoodie.clean.async`](https://hudi.apache.org/docs/configurations#hoodiecleanasync) as shown below:
+```java
+hoodie.clean.automatic=true
+hoodie.clean.async=true
+```
+
+For Flink based writing, this is the default mode of cleaning. Please refer to [`clean.async.enabled`](https://hudi.apache.org/docs/configurations/#cleanasyncenabled) for details.
+
+#### Run independently
+Hoodie Cleaner can also be run as a separate process. Following is the command for running the cleaner independently:
+```
+spark-submit --master local --class org.apache.hudi.utilities.HoodieCleaner `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` --help
+ Usage: <main class> [options]
+ Options:
+ --help, -h
+
+ --hoodie-conf
+ Any configuration that can be set in the properties file (using the CLI
+ parameter "--props") can also be passed command line using this
+ parameter. This can be repeated
+ Default: []
+ --props
+ path to properties file on localfs or dfs, with configurations for
+ hoodie client for cleaning
+ --spark-master
+ spark master to use.
+ Default: local[2]
+ * --target-base-path
+ base path for the hoodie table to be cleaner.
+```
+Some examples to run the cleaner.
+Keep the latest 10 commits
+```
+spark-submit --master local --class org.apache.hudi.utilities.HoodieCleaner `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar`\
+ --target-base-path /path/to/hoodie_table \
+ --hoodie-conf hoodie.cleaner.policy=KEEP_LATEST_COMMITS \
+ --hoodie-conf hoodie.cleaner.commits.retained=10 \
+ --hoodie-conf hoodie.cleaner.parallelism=200
+```
+Keep the latest 3 file versions
+```
+spark-submit --master local --class org.apache.hudi.utilities.HoodieCleaner `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar`\
+ --target-base-path /path/to/hoodie_table \
+ --hoodie-conf hoodie.cleaner.policy=KEEP_LATEST_FILE_VERSIONS \
+ --hoodie-conf hoodie.cleaner.fileversions.retained=3 \
+ --hoodie-conf hoodie.cleaner.parallelism=200
+```
+Clean commits older than 24 hours
+```
+spark-submit --master local --class org.apache.hudi.utilities.HoodieCleaner `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar`\
+ --target-base-path /path/to/hoodie_table \
+ --hoodie-conf hoodie.cleaner.policy=KEEP_LATEST_BY_HOURS \
+ --hoodie-conf hoodie.cleaner.hours.retained=24 \
+ --hoodie-conf hoodie.cleaner.parallelism=200
+```
+Note: The parallelism takes the min value of number of partitions to clean and `hoodie.cleaner.parallelism`.
+
+#### CLI
+You can also use [Hudi CLI](cli.md) to run Hoodie Cleaner.
+
+CLI provides the below commands for cleaner service:
+- `cleans show`
+- `clean showpartitions`
+- `cleans run`
+
+Example of cleaner keeping the latest 3 commits
+```
+cleans run --sparkMaster local --hoodieConfigs hoodie.cleaner.policy=KEEP_LATEST_COMMITS hoodie.cleaner.commits.retained=3 hoodie.cleaner.parallelism=200
+```
+
+You can find more details and the relevant code for these commands in [`org.apache.hudi.cli.commands.CleansCommand`](https://github.com/apache/hudi/blob/master/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CleansCommand.java) class.
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Cleaner Service: Save up to 40% on data lake storage costs | Hudi Labs](https://youtu.be/mUvRhJDoO3w)
+* [Efficient Data Lake Management with Apache Hudi Cleaner: Benefits of Scheduling Data Cleaning #1](https://www.youtube.com/watch?v=CEzgFtmVjx4)
+* [Efficient Data Lake Management with Apache Hudi Cleaner: Benefits of Scheduling Data Cleaning #2](https://www.youtube.com/watch?v=RbBF9Ys2GqM)
diff --git a/website/versioned_docs/version-0.15.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-0.15.1/hoodie_streaming_ingestion.md
new file mode 100644
index 000000000000..1158c695df6c
--- /dev/null
+++ b/website/versioned_docs/version-0.15.1/hoodie_streaming_ingestion.md
@@ -0,0 +1,635 @@
+---
+title: Using Spark
+keywords: [hudi, streamer, hoodiestreamer, spark_streaming]
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Hudi Streamer
+
+The `HoodieStreamer` utility (part of `hudi-utilities-slim-bundle` and `hudi-utilities-bundle`) provides ways to ingest
+from different sources such as DFS or Kafka, with the following capabilities.
+
+- Exactly once ingestion of new events from
+ Kafka, [incremental imports](https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide#_incremental_imports) from Sqoop or
+ output of `HiveIncrementalPuller` or files under a DFS folder
+- Support json, avro or a custom record types for the incoming data
+- Manage checkpoints, rollback & recovery
+- Leverage Avro schemas from DFS or Confluent [schema registry](https://github.com/confluentinc/schema-registry).
+- Support for plugging in transformations
+
+:::danger Important
+
+The following classes were renamed and relocated to `org.apache.hudi.utilities.streamer` package.
+- `DeltastreamerMultiWriterCkptUpdateFunc` is renamed to `StreamerMultiWriterCkptUpdateFunc`
+- `DeltaSync` is renamed to `StreamSync`
+- `HoodieDeltaStreamer` is renamed to `HoodieStreamer`
+- `HoodieDeltaStreamerMetrics` is renamed to `HoodieStreamerMetrics`
+- `HoodieMultiTableDeltaStreamer` is renamed to `HoodieMultiTableStreamer`
+
+To maintain backward compatibility, the original classes are still present in the `org.apache.hudi.utilities.deltastreamer`
+package, but have been deprecated.
+
+:::
+
+### Options
+<details>
+
+<summary>
+Expand this to see HoodieStreamer's "--help" output describing its capabilities in more details.
+</summary>
+
+```shell
+[hoodie]$ spark-submit --class org.apache.hudi.utilities.streamer.HoodieStreamer `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` --help
+Usage: <main class> [options]
+ Options:
+ --allow-commit-on-no-checkpoint-change
+ allow commits even if checkpoint has not changed before and after fetch
+ datafrom source. This might be useful in sources like SqlSource where
+ there is not checkpoint. And is not recommended to enable in continuous
+ mode.
+ Default: false
+ --base-file-format
+ File format for the base files. PARQUET (or) HFILE
+ Default: PARQUET
+ --bootstrap-index-class
+ subclass of BootstrapIndex
+ Default: org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex
+ --bootstrap-overwrite
+ Overwrite existing target table, default false
+ Default: false
+ --checkpoint
+ Resume Hudi Streamer from this checkpoint.
+ --cluster-scheduling-minshare
+ Minshare for clustering as defined in
+ https://spark.apache.org/docs/latest/job-scheduling.html
+ Default: 0
+ --cluster-scheduling-weight
+ Scheduling weight for clustering as defined in
+ https://spark.apache.org/docs/latest/job-scheduling.html
+ Default: 1
+ --commit-on-errors
+ Commit even when some records failed to be written
+ Default: false
+ --compact-scheduling-minshare
+ Minshare for compaction as defined in
+ https://spark.apache.org/docs/latest/job-scheduling.html
+ Default: 0
+ --compact-scheduling-weight
+ Scheduling weight for compaction as defined in
+ https://spark.apache.org/docs/latest/job-scheduling.html
+ Default: 1
+ --config-hot-update-strategy-class
+ Configuration hot update in continuous mode
+ Default: <empty string>
+ --continuous
+ Hudi Streamer runs in continuous mode running source-fetch -> Transform
+ -> Hudi Write in loop
+ Default: false
+ --delta-sync-scheduling-minshare
+ Minshare for delta sync as defined in
+ https://spark.apache.org/docs/latest/job-scheduling.html
+ Default: 0
+ --delta-sync-scheduling-weight
+ Scheduling weight for delta sync as defined in
+ https://spark.apache.org/docs/latest/job-scheduling.html
+ Default: 1
+ --disable-compaction
+ Compaction is enabled for MoR table by default. This flag disables it
+ Default: false
+ --enable-hive-sync
+ Enable syncing to hive
+ Default: false
+ --enable-sync
+ Enable syncing meta
+ Default: false
+ --filter-dupes
+ Should duplicate records from source be dropped/filtered out before
+ insert/bulk-insert
+ Default: false
+ --force-empty-sync
+ Force syncing meta even on empty commit
+ Default: false
+ --help, -h
+
+ --hoodie-conf
+ Any configuration that can be set in the properties file (using the CLI
+ parameter "--props") can also be passed command line using this
+ parameter. This can be repeated
+ Default: []
+ --ingestion-metrics-class
+ Ingestion metrics class for reporting metrics during ingestion
+ lifecycles.
+ Default: org.apache.hudi.utilities.streamer.HoodieStreamerMetrics
+ --initial-checkpoint-provider
+ subclass of
+ org.apache.hudi.utilities.checkpointing.InitialCheckpointProvider.
+ Generate check point for Hudi Streamer for the first run. This field
+ will override the checkpoint of last commit using the checkpoint field.
+ Use this field only when switching source, for example, from DFS source
+ to Kafka Source.
+ --max-pending-clustering
+ Maximum number of outstanding inflight/requested clustering. Delta Sync
+ will not happen unlessoutstanding clustering is less than this number
+ Default: 5
+ --max-pending-compactions
+ Maximum number of outstanding inflight/requested compactions. Delta Sync
+ will not happen unlessoutstanding compactions is less than this number
+ Default: 5
+ --max-retry-count
+ the max retry count if --retry-on-source-failures is enabled
+ Default: 3
+ --min-sync-interval-seconds
+ the min sync interval of each sync in continuous mode
+ Default: 0
+ --op
+ Takes one of these values : UPSERT (default), INSERT, BULK_INSERT,
+ INSERT_OVERWRITE, INSERT_OVERWRITE_TABLE, DELETE_PARTITION
+ Default: UPSERT
+ Possible Values: [INSERT, INSERT_PREPPED, UPSERT, UPSERT_PREPPED, BULK_INSERT, BULK_INSERT_PREPPED, DELETE, DELETE_PREPPED, BOOTSTRAP, INSERT_OVERWRITE, CLUSTER, DELETE_PARTITION, INSERT_OVERWRITE_TABLE, COMPACT, INDEX, ALTER_SCHEMA, LOG_COMPACT, UNKNOWN]
+ --payload-class
+ subclass of HoodieRecordPayload, that works off a GenericRecord.
+ Implement your own, if you want to do something other than overwriting
+ existing value
+ Default: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+ --post-write-termination-strategy-class
+ Post writer termination strategy class to gracefully shutdown
+ deltastreamer in continuous mode
+ Default: <empty string>
+ --props
+ path to properties file on localfs or dfs, with configurations for
+ hoodie client, schema provider, key generator and data source. For
+ hoodie client props, sane defaults are used, but recommend use to
+ provide basic things like metrics endpoints, hive configs etc. For
+ sources, referto individual classes, for supported properties.
+ Properties in this file can be overridden by "--hoodie-conf"
+ Default: file:///Users/shiyanxu/src/test/resources/streamer-config/dfs-source.properties
+ --retry-interval-seconds
+ the retry interval for source failures if --retry-on-source-failures is
+ enabled
+ Default: 30
+ --retry-last-pending-inline-clustering, -rc
+ Retry last pending inline clustering plan before writing to sink.
+ Default: false
+ --retry-last-pending-inline-compaction
+ Retry last pending inline compaction plan before writing to sink.
+ Default: false
+ --retry-on-source-failures
+ Retry on any source failures
+ Default: false
+ --run-bootstrap
+ Run bootstrap if bootstrap index is not found
+ Default: false
+ --schemaprovider-class
+ subclass of org.apache.hudi.utilities.schema.SchemaProvider to attach
+ schemas to input & target table data, built in options:
+ org.apache.hudi.utilities.schema.FilebasedSchemaProvider.Source (See
+ org.apache.hudi.utilities.sources.Source) implementation can implement
+ their own SchemaProvider. For Sources that return Dataset<Row>, the
+ schema is obtained implicitly. However, this CLI option allows
+ overriding the schemaprovider returned by Source.
+ --source-class
+ Subclass of org.apache.hudi.utilities.sources to read data. Built-in
+ options: org.apache.hudi.utilities.sources.{JsonDFSSource (default),
+ AvroDFSSource, JsonKafkaSource, AvroKafkaSource, HiveIncrPullSource}
+ Default: org.apache.hudi.utilities.sources.JsonDFSSource
+ --source-limit
+ Maximum amount of data to read from source. Default: No limit, e.g:
+ DFS-Source => max bytes to read, Kafka-Source => max events to read
+ Default: 9223372036854775807
+ --source-ordering-field
+ Field within source record to decide how to break ties between records
+ with same key in input data. Default: 'ts' holding unix timestamp of
+ record
+ Default: ts
+ --spark-master
+ spark master to use, if not defined inherits from your environment
+ taking into account Spark Configuration priority rules (e.g. not using
+ spark-submit command).
+ Default: <empty string>
+ --sync-tool-classes
+ Meta sync client tool, using comma to separate multi tools
+ Default: org.apache.hudi.hive.HiveSyncTool
+ * --table-type
+ Type of table. COPY_ON_WRITE (or) MERGE_ON_READ
+ * --target-base-path
+ base path for the target hoodie table. (Will be created if did not exist
+ first time around. If exists, expected to be a hoodie table)
+ * --target-table
+ name of the target table
+ --transformer-class
+ A subclass or a list of subclasses of
+ org.apache.hudi.utilities.transform.Transformer. Allows transforming raw
+ source Dataset to a target Dataset (conforming to target schema) before
+ writing. Default : Not set. E.g. -
+ org.apache.hudi.utilities.transform.SqlQueryBasedTransformer (which
+ allows a SQL query templated to be passed as a transformation function).
+ Pass a comma-separated list of subclass names to chain the
+ transformations. If there are two or more transformers using the same
+ config keys and expect different values for those keys, then transformer
+ can include an identifier. E.g. -
+ tr1:org.apache.hudi.utilities.transform.SqlQueryBasedTransformer. Here
+ the identifier tr1 can be used along with property key like
+ `hoodie.streamer.transformer.sql.tr1` to identify properties related to
+ the transformer. So effective value for
+ `hoodie.streamer.transformer.sql` is determined by key
+ `hoodie.streamer.transformer.sql.tr1` for this transformer. If
+ identifier is used, it should be specified for all the transformers.
+ Further the order in which transformer is applied is determined by the
+ occurrence of transformer irrespective of the identifier used for the
+ transformer. For example: In the configured value below tr2:org.apache.hudi.utilities.transform.SqlQueryBasedTransformer,tr1:org.apache.hudi.utilities.transform.SqlQueryBasedTransformer
+ , tr2 is applied before tr1 based on order of occurrence.
+```
+</details>
+
+The tool takes a hierarchically composed property file and has pluggable interfaces for extracting data, key generation and providing schema. Sample configs for ingesting from kafka and dfs are
+provided under `hudi-utilities/src/test/resources/streamer-config`.
+
+For e.g: once you have Confluent Kafka, Schema registry up & running, produce some test data using ([impressions.avro](https://docs.confluent.io/current/ksql/docs/tutorials/generate-custom-test-data) provided by schema-registry repo)
+
+```java
+[confluent-5.0.0]$ bin/ksql-datagen schema=../impressions.avro format=avro topic=impressions key=impressionid
+```
+
+and then ingest it as follows.
+
+```java
+[hoodie]$ spark-submit --class org.apache.hudi.utilities.streamer.HoodieStreamer `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` \
+ --props file://${PWD}/hudi-utilities/src/test/resources/streamer-config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --target-base-path file:\/\/\/tmp/hudi-streamer-op \
+ --target-table uber.impressions \
+ --op BULK_INSERT
+```
+
+In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide.md).
+
+### Using `hudi-utilities` bundle jars
+
+From 0.11.0 release, we start to provide a new `hudi-utilities-slim-bundle` which aims to exclude dependencies that can
+cause conflicts and compatibility issues with different versions of Spark.
+
+It is recommended to switch to `hudi-utilities-slim-bundle`, which should be used along with a Hudi Spark bundle
+corresponding the Spark version used to make utilities work with Spark, e.g.,
+`--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:0.13.0,org.apache.hudi:hudi-spark3.2-bundle_2.12:0.13.0`.
+
+`hudi-utilities-bundle` remains as a legacy bundle jar to work with Spark 2.4 and 3.1.
+
+### Concurrency Control
+
+Using optimistic concurrency control (OCC) via Hudi Streamer requires the configs below to the properties file that can be passed to the
+job.
+
+```properties
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.write.lock.provider=<lock-provider-classname>
+hoodie.cleaner.policy.failed.writes=LAZY
+```
+
+As an example, adding the configs to `kafka-source.properties` file and passing them to Hudi Streamer will enable OCC.
+A Hudi Streamer job can then be triggered as follows:
+
+```java
+[hoodie]$ spark-submit --class org.apache.hudi.utilities.streamer.HoodieStreamer `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` \
+ --props file://${PWD}/hudi-utilities/src/test/resources/streamer-config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --target-base-path file:///tmp/hudi-streamer-op \
+ --target-table uber.impressions \
+ --op BULK_INSERT
+```
+
+Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control.md) section
+
+### Checkpointing
+
+`HoodieStreamer` uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
+When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
+When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read.
+Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+
+If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
+
+- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint.
+- `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
+For Kafka, this is the max # of events to read.
+
+### Transformers
+
+`HoodieStreamer` supports custom transformation on records before writing to storage. This is done by supplying
+implementation of `org.apache.hudi.utilities.transform.Transformer` via `--transformer-class` option.
+
+#### SQL Query Transformer
+You can pass a SQL Query to be executed during write.
+
+```scala
+--transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer
+--hoodie-conf hoodie.streamer.transformer.sql=SELECT a.col1, a.col3, a.col4 FROM <SRC> a
+```
+
+#### SQL File Transformer
+You can specify a File with a SQL script to be executed during write. The SQL file is configured with this hoodie property:
+hoodie.streamer.transformer.sql.file
+
+The query should reference the source as a table named "\<SRC\>"
+
+The final sql statement result is used as the write payload.
+
+Example Spark SQL Query:
+```sql
+CACHE TABLE tmp_personal_trips AS
+SELECT * FROM <SRC> WHERE trip_type='personal_trips';
+
+SELECT * FROM tmp_personal_trips;
+```
+
+#### Flattening Transformer
+This transformer can flatten nested objects. It flattens the nested fields in the incoming records by prefixing
+inner-fields with outer-field and _ in a nested fashion. Currently flattening of arrays is not supported.
+
+An example schema may look something like the below where name is a nested field of StructType in the original source
+```scala
+age as intColumn,address as stringColumn,name.first as name_first,name.last as name_last, name.middle as name_middle
+```
+
+Set the config as:
+```scala
+--transformer-class org.apache.hudi.utilities.transform.FlatteningTransformer
+```
+
+#### Chained Transformer
+If you wish to use multiple transformers together, you can use the Chained transformers to pass multiple to be executed sequentially.
+
+Example below first flattens the incoming records and then does sql projection based on the query specified:
+```scala
+--transformer-class org.apache.hudi.utilities.transform.FlatteningTransformer,org.apache.hudi.utilities.transform.SqlQueryBasedTransformer
+--hoodie-conf hoodie.streamer.transformer.sql=SELECT a.col1, a.col3, a.col4 FROM <SRC> a
+```
+
+#### AWS DMS Transformer
+This transformer is specific for AWS DMS data. It adds `Op` field with value `I` if the field is not present.
+
+Set the config as:
+```scala
+--transformer-class org.apache.hudi.utilities.transform.AWSDmsTransformer
+```
+
+#### Custom Transformer Implementation
+You can write your own custom transformer by extending [this class](https://github.com/apache/hudi/tree/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/transform)
+
+#### Related Resources
+
+* [Learn about Apache Hudi Transformers with Hands on Lab](https://www.youtube.com/watch?v=AprlZ8hGdJo)
+* [Apache Hudi with DBT Hands on Lab.Transform Raw Hudi tables with DBT and Glue Interactive Session](https://youtu.be/DH3LEaPG6ss)
+
+### Schema Providers
+
+By default, Spark will infer the schema of the source and use that inferred schema when writing to a table. If you need
+to explicitly define the schema you can use one of the following Schema Providers below.
+
+#### Schema Registry Provider
+
+You can obtain the latest schema from an online registry. You pass a URL to the registry and if needed, you can also
+pass userinfo and credentials in the url like: `https://foo:bar@schemaregistry.org` The credentials are then extracted
+and are set on the request as an Authorization Header.
+
+When fetching schemas from a registry, you can specify both the source schema and the target schema separately.
+
+| Config | Description | Example |
+|---------------------------------------------------|------------------------------------------------|------------------------------------|
+| hoodie.streamer.schemaprovider.registry.url | The schema of the source you are reading from | https://foo:bar@schemaregistry.org |
+| hoodie.streamer.schemaprovider.registry.targetUrl | The schema of the target you are writing to | https://foo:bar@schemaregistry.org |
+
+The above configs are passed to Hudi Streamer spark-submit command like:
+
+```shell
+--hoodie-conf hoodie.streamer.schemaprovider.registry.url=https://foo:bar@schemaregistry.org
+```
+
+There are other optional configs to work with schema registry provider such as SSL-store related configs, and supporting
+custom transformation of schema returned by schema registry, e.g., converting the original json schema to avro schema
+via `org.apache.hudi.utilities.schema.converter.JsonToAvroSchemaConverter`.
+
+| Config | Description | Example |
+|---------------------------------------------------------|------------------------------------------------------|------------------------------------------------------------------------|
+| hoodie.streamer.schemaprovider.registry.schemaconverter | The class name of the custom schema converter to use | `org.apache.hudi.utilities.schema.converter.JsonToAvroSchemaConverter` |
+| schema.registry.ssl.keystore.location | SSL key store location | |
+| schema.registry.ssl.keystore.password | SSL key store password | |
+| schema.registry.ssl.truststore.location | SSL trust store location | |
+| schema.registry.ssl.truststore.password | SSL trust store password | |
+| schema.registry.ssl.key.password | SSL key password | |
+
+#### JDBC Schema Provider
+
+You can obtain the latest schema through a JDBC connection.
+
+| Config | Description | Example |
+|------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------|
+| hoodie.streamer.schemaprovider.source.schema.jdbc.connection.url | The JDBC URL to connect to. You can specify source specific connection properties in the URL | jdbc:postgresql://localhost/test?user=fred&password=secret |
+| hoodie.streamer.schemaprovider.source.schema.jdbc.driver.type | The class name of the JDBC driver to use to connect to this URL | org.h2.Driver |
+| hoodie.streamer.schemaprovider.source.schema.jdbc.username | username for the connection | fred |
+| hoodie.streamer.schemaprovider.source.schema.jdbc.password | password for the connection | secret |
+| hoodie.streamer.schemaprovider.source.schema.jdbc.dbtable | The table with the schema to reference | test_database.test1_table or test1_table |
+| hoodie.streamer.schemaprovider.source.schema.jdbc.timeout | The number of seconds the driver will wait for a Statement object to execute to the given number of seconds. Zero means there is no limit. In the write path, this option depends on how JDBC drivers implement the API setQueryTimeout, e.g., the h2 JDBC driver checks the timeout of each query instead of an entire JDBC batch. It defaults to 0. | 0 |
+| hoodie.streamer.schemaprovider.source.schema.jdbc.nullable | If true, all columns are nullable | true |
+
+The above configs are passed to Hudi Streamer spark-submit command like:
+```--hoodie-conf hoodie.streamer.jdbcbasedschemaprovider.connection.url=jdbc:postgresql://localhost/test?user=fred&password=secret```
+
+#### File Based Schema Provider
+
+You can use a .avsc file to define your schema. You can then point to this file on DFS as a schema provider.
+
+| Config | Description | Example |
+|---------------------------------------------------|-----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.streamer.schemaprovider.source.schema.file | The schema of the source you are reading from | [example schema file](https://github.com/apache/hudi/blob/a8fb69656f522648233f0310ca3756188d954281/docker/demo/config/test-suite/source.avsc) |
+| hoodie.streamer.schemaprovider.target.schema.file | The schema of the target you are writing to | [example schema file](https://github.com/apache/hudi/blob/a8fb69656f522648233f0310ca3756188d954281/docker/demo/config/test-suite/target.avsc) |
+
+#### Hive Schema Provider
+
+You can use hive tables to fetch source and target schema.
+
+| Config | Description |
+|------------------------------------------------------------|--------------------------------------------------------|
+| hoodie.streamer.schemaprovider.source.schema.hive.database | Hive database from where source schema can be fetched |
+| hoodie.streamer.schemaprovider.source.schema.hive.table | Hive table from where source schema can be fetched |
+| hoodie.streamer.schemaprovider.target.schema.hive.database | Hive database from where target schema can be fetched |
+| hoodie.streamer.schemaprovider.target.schema.hive.table | Hive table from where target schema can be fetched |
+
+
+#### Schema Provider with Post Processor
+The SchemaProviderWithPostProcessor, will extract the schema from one of the previously mentioned Schema Providers and
+then will apply a post processor to change the schema before it is used. You can write your own post processor by extending
+this class: https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaPostProcessor.java
+
+### Sources
+
+Hudi Streamer can read data from a wide variety of sources. The following are a list of supported sources:
+
+#### Distributed File System (DFS)
+See the storage configurations page to see some examples of DFS applications Hudi can read from. The following are the
+supported file formats Hudi can read/write with on DFS Sources. (Note: you can still use Spark/Flink readers to read from
+other formats and then write data as Hudi format.)
+
+- CSV
+- AVRO
+- JSON
+- PARQUET
+- ORC
+- HUDI
+
+For DFS sources the following behaviors are expected:
+
+- For JSON DFS source, you always need to set a schema. If the target Hudi table follows the same schema as from the source file, you just need to set the source schema. If not, you need to set schemas for both source and target.
+- `HoodieStreamer` reads the files under the source base path (`hoodie.streamer.source.dfs.root`) directly, and it won't use the partition paths under this base path as fields of the dataset. Detailed examples can be found [here](https://github.com/apache/hudi/issues/5485).
+
+#### Kafka
+Hudi can read directly from Kafka clusters. See more details on `HoodieStreamer` to learn how to setup streaming
+ingestion with exactly once semantics, checkpointing, and plugin transformations. The following formats are supported
+when reading data from Kafka:
+
+- AVRO: `org.apache.hudi.utilities.sources.AvroKafkaSource`
+- JSON: `org.apache.hudi.utilities.sources.JsonKafkaSource`
+- Proto: `org.apache.hudi.utilities.sources.ProtoKafkaSource`
+
+Check out [Kafka source config](https://hudi.apache.org/docs/configurations#Kafka-Source-Configs) for more details.
+
+#### Pulsar
+
+`HoodieStreamer` also supports ingesting from Apache Pulsar via `org.apache.hudi.utilities.sources.PulsarSource`.
+Check out [Pulsar source config](https://hudi.apache.org/docs/configurations#Pulsar-Source-Configs) for more details.
+
+#### Cloud storage event sources
+AWS S3 storage provides an event notification service which will post notifications when certain events happen in your S3 bucket:
+https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html
+AWS will put these events in a Simple Queue Service (SQS). Apache Hudi provides `S3EventsSource`
+and `S3EventsHoodieIncrSource` that can read from SQS to trigger/processing of new or changed data as soon as it is
+available on S3. Check out [S3 source configs](https://hudi.apache.org/docs/configurations#S3-Source-Configs) for more details.
+
+Similar to S3 event source, Google Cloud Storage (GCS) event source is also supported via `GcsEventsSource` and
+`GcsEventsHoodieIncrSource`. Check out [GCS events source configs](https://hudi.apache.org/docs/configurations#GCS-Events-Source-Configs) for more details.
+
+##### AWS Setup
+1. Enable S3 Event Notifications https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html
+2. Download the aws-java-sdk-sqs jar.
+3. Find the queue URL and Region to set these configurations:
+ 1. hoodie.streamer.s3.source.queue.url=https://sqs.us-west-2.amazonaws.com/queue/url
+ 2. hoodie.streamer.s3.source.queue.region=us-west-2
+4. Start the `S3EventsSource` and `S3EventsHoodieIncrSource` using the `HoodieStreamer` utility as shown in sample commands below:
+
+Insert code sample from this blog: https://hudi.apache.org/blog/2021/08/23/s3-events-source/#configuration-and-setup
+
+#### JDBC Source
+Hudi can read from a JDBC source with a full fetch of a table, or Hudi can even read incrementally with checkpointing from a JDBC source.
+
+| Config | Description | Example |
+|--------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.streamer.jdbc.url | URL of the JDBC connection | jdbc:postgresql://localhost/test |
+| hoodie.streamer.jdbc.user | User to use for authentication of the JDBC connection | fred |
+| hoodie.streamer.jdbc.password | Password to use for authentication of the JDBC connection | secret |
+| hoodie.streamer.jdbc.password.file | If you prefer to use a password file for the connection | |
+| hoodie.streamer.jdbc.driver.class | Driver class to use for the JDBC connection | |
+| hoodie.streamer.jdbc.table.name | | my_table |
+| hoodie.streamer.jdbc.table.incr.column.name | If run in incremental mode, this field will be used to pull new data incrementally | |
+| hoodie.streamer.jdbc.incr.pull | Will the JDBC connection perform an incremental pull? | |
+| hoodie.streamer.jdbc.extra.options. | How you pass extra configurations that would normally by specified as spark.read.option() | hoodie.streamer.jdbc.extra.options.fetchSize=100 hoodie.streamer.jdbc.extra.options.upperBound=1 hoodie.streamer.jdbc.extra.options.lowerBound=100 |
+| hoodie.streamer.jdbc.storage.level | Used to control the persistence level | Default = MEMORY_AND_DISK_SER |
+| hoodie.streamer.jdbc.incr.fallback.to.full.fetch | Boolean which if set true makes an incremental fetch fallback to a full fetch if there is any error in the incremental read | FALSE |
+
+#### SQL Sources
+
+SQL Source `org.apache.hudi.utilities.sources.SqlSource` reads from any table, used mainly for backfill jobs which will process specific partition dates.
+This won't update the streamer.checkpoint.key to the processed commit, instead it will fetch the latest successful
+checkpoint key and set that value as this backfill commits checkpoint so that it won't interrupt the regular incremental
+processing. To fetch and use the latest incremental checkpoint, you need to also set this hoodie_conf for Hudi Streamer
+jobs: `hoodie.write.meta.key.prefixes = 'streamer.checkpoint.key'`
+
+Spark SQL should be configured using this hoodie config:
+`hoodie.streamer.source.sql.sql.query = 'select * from source_table'`
+
+Using `org.apache.hudi.utilities.sources.SqlFileBasedSource` allows setting the SQL queries in a file to read from any
+table. SQL file path should be configured using this hoodie config:
+`hoodie.streamer.source.sql.file = 'hdfs://xxx/source.sql'`
+
+### Error Table
+
+`HoodieStreamer` supports segregating error records into a separate table called "Error table" alongside with the
+target data table. This allows easy integration with dead-letter queues (DLQ). Error Table is supported with
+user-provided subclass of `org.apache.hudi.utilities.streamer.BaseErrorTableWriter` supplied via
+config `hoodie.errortable.write.class`. Check out more in `org.apache.hudi.config.HoodieErrorTableConfig`.
+
+### Termination Strategy
+
+Users can configure a post-write termination strategy under `continuous` mode if need be. For instance,
+users can configure graceful shutdown if there is no new data from the configured source for 5 consecutive times.
+Here is the interface for the termination strategy.
+
+```java
+/**
+ * Post write termination strategy for deltastreamer in continuous mode.
+ */
+public interface PostWriteTerminationStrategy {
+
+ /**
+ * Returns whether HoodieStreamer needs to be shutdown.
+ * @param scheduledCompactionInstantAndWriteStatuses optional pair of scheduled compaction instant and write statuses.
+ * @return true if HoodieStreamer has to be shutdown. false otherwise.
+ */
+ boolean shouldShutdown(Option<Pair<Option<String>, JavaRDD<WriteStatus>>> scheduledCompactionInstantAndWriteStatuses);
+
+}
+```
+
+Also, this might help in bootstrapping a new table. Instead of doing one bulk load or bulk_insert leveraging a large
+cluster for a large input of data, one could start `HoodieStreamer` on the `continuous` mode and add a shutdown strategy
+to terminate, once all data has been bootstrapped. This way, each batch could be smaller and may not need a large
+cluster to bootstrap data. There is a concrete implementation provided out-of-the-box: [NoNewDataTerminationStrategy](https://github.com/apache/hudi/blob/0d0a4152cfd362185066519ae926ac4513c7a152/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/NoNewDataTerminationStrategy.java).
+Users can feel free to implement their own strategy as they see fit.
+
+### Dynamic configuration updates
+
+When Hudi Streamer is running in `continuous` mode, the properties can be refreshed/updated before each sync calls.
+Interested users can implement `org.apache.hudi.utilities.deltastreamer.ConfigurationHotUpdateStrategy` to leverage this.
+
+## MultiTableStreamer
+
+`HoodieMultiTableStreamer`, an extension of `HoodieStreamer`, facilitates the simultaneous ingestion of multiple tables into Hudi datasets. At present, it supports the sequential ingestion of tables and accommodates both COPY_ON_WRITE and MERGE_ON_READ storage types. The command line parameters for `HoodieMultiTableStreamer` largely mirror those of `HoodieStreamer`, with the notable difference being the necessity to supply table-specific configurations in separate files in a dedicated co [...]
+
+```java
+ * --config-folder
+ the path to the folder which contains all the table wise config files
+ --base-path-prefix
+ this is added to enable users to create all the hudi datasets for related tables under one path in FS. The datasets are then created under the path - <base_path_prefix>/<database>/<table_to_be_ingested>. However you can override the paths for every table by setting the property hoodie.streamer.ingestion.targetBasePath
+```
+
+The following properties are needed to be set properly to ingest data using `HoodieMultiTableStreamer`.
+
+```java
+hoodie.streamer.ingestion.tablesToBeIngested
+ comma separated names of tables to be ingested in the format <database>.<table>, for example db1.table1,db1.table2
+hoodie.streamer.ingestion.targetBasePath
+ if you wish to ingest a particular table in a separate path, you can mention that path here
+hoodie.streamer.ingestion.<database>.<table>.configFile
+ path to the config file in dedicated config folder which contains table overridden properties for the particular table to be ingested.
+```
+
+Sample config files for table wise overridden properties can be found
+under `hudi-utilities/src/test/resources/streamer-config`. The command to run `HoodieMultiTableStreamer` is also similar
+to how you run `HoodieStreamer`.
+
+```java
... 10593 lines suppressed ...