You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by si...@apache.org on 2021/08/27 22:17:39 UTC
[hudi] branch asf-site updated: [HUDI-2373] Generate versioned docs
for Hudi 0.9.0 release (#3551)
This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0834046 [HUDI-2373] Generate versioned docs for Hudi 0.9.0 release (#3551)
0834046 is described below
commit 0834046421bbca51fda3c53e79be1a094a5d6f42
Author: Udit Mehrotra <ud...@gmail.com>
AuthorDate: Fri Aug 27 15:17:26 2021 -0700
[HUDI-2373] Generate versioned docs for Hudi 0.9.0 release (#3551)
- Generate versioned docs for Hudi 0.9.0 release
- Updated configs for 0.9.0 and current unreleased version
---
hudi-utils/README.md | 2 +-
hudi-utils/pom.xml | 2 +-
website/docs/configurations.md | 2472 +++++++++++---------
website/docusaurus.config.js | 4 +-
.../versioned_docs/version-0.9.0/azure_hoodie.md | 50 +
website/versioned_docs/version-0.9.0/bos_hoodie.md | 57 +
website/versioned_docs/version-0.9.0/cloud.md | 27 +
website/versioned_docs/version-0.9.0/comparison.md | 56 +
website/versioned_docs/version-0.9.0/concepts.md | 172 ++
.../version-0.9.0/concurrency_control.md | 149 ++
.../version-0.9.0}/configurations.md | 2453 ++++++++++---------
website/versioned_docs/version-0.9.0/cos_hoodie.md | 71 +
website/versioned_docs/version-0.9.0/deployment.md | 577 +++++
.../versioned_docs/version-0.9.0/docker_demo.md | 1235 ++++++++++
.../version-0.9.0/flink-quick-start-guide.md | 556 +++++
website/versioned_docs/version-0.9.0/gcs_hoodie.md | 60 +
.../versioned_docs/version-0.9.0/ibm_cos_hoodie.md | 77 +
website/versioned_docs/version-0.9.0/metrics.md | 160 ++
.../version-0.9.0/migration_guide.md | 70 +
website/versioned_docs/version-0.9.0/oss_hoodie.md | 70 +
website/versioned_docs/version-0.9.0/overview.md | 172 ++
.../versioned_docs/version-0.9.0/performance.md | 64 +
website/versioned_docs/version-0.9.0/privacy.md | 22 +
.../versioned_docs/version-0.9.0/querying_data.md | 277 +++
.../version-0.9.0/quick-start-guide.md | 1075 +++++++++
website/versioned_docs/version-0.9.0/s3_hoodie.md | 80 +
.../version-0.9.0/schema_evolution.md | 193 ++
website/versioned_docs/version-0.9.0/structure.md | 20 +
website/versioned_docs/version-0.9.0/use_cases.md | 81 +
.../versioned_docs/version-0.9.0/writing_data.md | 425 ++++
.../versioned_sidebars/version-0.9.0-sidebars.json | 125 +
website/versions.json | 1 +
32 files changed, 8537 insertions(+), 2318 deletions(-)
diff --git a/hudi-utils/README.md b/hudi-utils/README.md
index 4143656..21fa4cf 100644
--- a/hudi-utils/README.md
+++ b/hudi-utils/README.md
@@ -18,6 +18,6 @@
mvn clean
mvn install
-java -cp target/hudi-utils-1.0-SNAPSHOT-jar-with-dependencies.jar:$HOME/.m2/repository/org/apache/hudi/hudi-utilities-bundle_2.11/0.9.0-SNAPSHOT/hudi-utilities-bundle_2.11-0.9.0-SNAPSHOT.jar:$HOME/.m2/repository/org/apache/hudi/hudi-spark-bundle_2.11/0.9.0-SNAPSHOT/hudi-spark-bundle_2.11-0.9.0-SNAPSHOT.jar:$HOME/.m2/repository/org/apache/hudi/hudi-flink-bundle_2.11/0.9.0-SNAPSHOT/hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar org.apache.hudi.utils.HoodieConfigDocGenerator
+java -cp target/hudi-utils-1.0-SNAPSHOT-jar-with-dependencies.jar:$HOME/.m2/repository/org/apache/hudi/hudi-utilities-bundle_2.11/0.10.0-SNAPSHOT/hudi-utilities-bundle_2.11-0.10.0-SNAPSHOT.jar:$HOME/.m2/repository/org/apache/hudi/hudi-spark-bundle_2.11/0.10.0-SNAPSHOT/hudi-spark-bundle_2.11-0.10.0-SNAPSHOT.jar:$HOME/.m2/repository/org/apache/hudi/hudi-flink-bundle_2.11/0.10.0-SNAPSHOT/hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar org.apache.hudi.utils.HoodieConfigDocGenerator
cp /tmp/configurations.md $HUDI-DIR/website/docs/configurations.md
diff --git a/hudi-utils/pom.xml b/hudi-utils/pom.xml
index 70701e0..97b5e4e 100644
--- a/hudi-utils/pom.xml
+++ b/hudi-utils/pom.xml
@@ -27,7 +27,7 @@
<properties>
<jdk.version>1.8</jdk.version>
- <hudi.version>0.9.0-SNAPSHOT</hudi.version>
+ <hudi.version>0.10.0-SNAPSHOT</hudi.version>
<scala.binary.version>2.11</scala.binary.version>
<hudi.spark.module>hudi-spark2</hudi.spark.module>
<scala.binary.version>2.11</scala.binary.version>
diff --git a/website/docs/configurations.md b/website/docs/configurations.md
index a8307db..a5db0a6 100644
--- a/website/docs/configurations.md
+++ b/website/docs/configurations.md
@@ -4,7 +4,7 @@ keywords: [ configurations, default, flink options, spark, configs, parameters ]
permalink: /docs/configurations.html
summary: This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
toc: true
-last_modified_at: 2021-08-09T17:00:50.653032
+last_modified_at: 2021-08-26T22:21:44.177
---
This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
@@ -24,6 +24,13 @@ Options useful for reading tables via `read.format.option(...)`
`Config Class`: org.apache.hudi.DataSourceOptions.scala<br></br>
+> #### hoodie.datasource.read.incr.path.glob
+> For the use-cases like users only want to incremental pull from certain partitions instead of the full table. This option allows using glob pattern to directly filter on path.<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: INCR_PATH_GLOB`<br></br>
+
+---
+
> #### hoodie.file.index.enable
> Enables use of the spark file index implementation for Hudi, that speeds up listing of large tables.<br></br>
> **Default Value**: true (Optional)<br></br>
@@ -31,24 +38,17 @@ Options useful for reading tables via `read.format.option(...)`
---
-> #### hoodie.datasource.merge.type
-> For Snapshot query on merge on read table, control whether we invoke the record payload implementation to merge (payload_combine) or skip merging altogetherskip_merge<br></br>
-> **Default Value**: payload_combine (Optional)<br></br>
-> `Config Param: REALTIME_MERGE_OPT_KEY`<br></br>
-
----
-
-> #### hoodie.datasource.read.incr.path.glob
-> For the use-cases like users only want to incremental pull from certain partitions instead of the full table. This option allows using glob pattern to directly filter on path.<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: INCR_PATH_GLOB_OPT_KEY`<br></br>
+> #### hoodie.datasource.read.paths
+> Comma separated list of file paths to read within a Hudi table.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: READ_PATHS`<br></br>
---
-> #### hoodie.datasource.query.type
-> Whether data needs to be read, in incremental mode (new data since an instantTime) (or) Read Optimized mode (obtain latest view, based on base files) (or) Snapshot mode (obtain latest view, by merging base and (if any) log files)<br></br>
-> **Default Value**: snapshot (Optional)<br></br>
-> `Config Param: QUERY_TYPE_OPT_KEY`<br></br>
+> #### hoodie.datasource.read.end.instanttime
+> Instant time to limit incrementally fetched data to. New data written with an instant_time <= END_INSTANTTIME are fetched out.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: END_INSTANTTIME`<br></br>
---
@@ -59,38 +59,45 @@ Options useful for reading tables via `read.format.option(...)`
---
-> #### hoodie.datasource.read.end.instanttime
-> Instant time to limit incrementally fetched data to. New data written with an instant_time <= END_INSTANTTIME are fetched out.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: END_INSTANTTIME_OPT_KEY`<br></br>
+> #### hoodie.datasource.read.incr.filters
+> For use-cases like DeltaStreamer which reads from Hoodie Incremental table and applies opaque map functions, filters appearing late in the sequence of transformations cannot be automatically pushed down. This option allows setting filters directly on Hoodie Source.<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: PUSH_DOWN_INCR_FILTERS`<br></br>
+
+---
+
+> #### hoodie.datasource.merge.type
+> For Snapshot query on merge on read table, control whether we invoke the record payload implementation to merge (payload_combine) or skip merging altogetherskip_merge<br></br>
+> **Default Value**: payload_combine (Optional)<br></br>
+> `Config Param: REALTIME_MERGE`<br></br>
---
> #### hoodie.datasource.read.begin.instanttime
> Instant time to start incrementally pulling data from. The instanttime here need not necessarily correspond to an instant on the timeline. New data written with an instant_time > BEGIN_INSTANTTIME are fetched out. For e.g: ‘20170901080000’ will get all new data written after Sep 1, 2017 08:00AM.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BEGIN_INSTANTTIME_OPT_KEY`<br></br>
+> `Config Param: BEGIN_INSTANTTIME`<br></br>
---
-> #### hoodie.datasource.read.schema.use.end.instanttime
-> Uses end instant schema when incrementally fetched data to. Default: users latest instant schema.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME_OPT_KEY`<br></br>
+> #### as.of.instant
+> The query instant for time travel. Without specified this option, we query the latest snapshot.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: TIME_TRAVEL_AS_OF_INSTANT`<br></br>
---
-> #### hoodie.datasource.read.incr.filters
-> For use-cases like DeltaStreamer which reads from Hoodie Incremental table and applies opaque map functions, filters appearing late in the sequence of transformations cannot be automatically pushed down. This option allows setting filters directly on Hoodie Source.<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: PUSH_DOWN_INCR_FILTERS_OPT_KEY`<br></br>
+> #### hoodie.datasource.query.type
+> Whether data needs to be read, in incremental mode (new data since an instantTime) (or) Read Optimized mode (obtain latest view, based on base files) (or) Snapshot mode (obtain latest view, by merging base and (if any) log files)<br></br>
+> **Default Value**: snapshot (Optional)<br></br>
+> `Config Param: QUERY_TYPE`<br></br>
---
-> #### hoodie.datasource.read.paths
-> Comma separated list of file paths to read within a Hudi table.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: READ_PATHS_OPT_KEY`<br></br>
+> #### hoodie.datasource.read.schema.use.end.instanttime
+> Uses end instant schema when incrementally fetched data to. Default: users latest instant schema.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME`<br></br>
---
@@ -114,311 +121,354 @@ Options useful for writing tables via `write.format.option(...)`
`Config Class`: org.apache.hudi.DataSourceOptions.scala<br></br>
-> #### hoodie.datasource.hive_sync.database
-> database to sync to<br></br>
-> **Default Value**: default (Optional)<br></br>
-> `Config Param: HIVE_DATABASE_OPT_KEY`<br></br>
+> #### hoodie.datasource.clustering.async.enable
+> Enable asynchronous clustering. Disabled by default.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ASYNC_CLUSTERING_ENABLE`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.datasource.write.precombine.field
-> Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br></br>
-> **Default Value**: ts (Optional)<br></br>
-> `Config Param: PRECOMBINE_FIELD_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.operation
+> Whether to do upsert, insert or bulkinsert for the write operation. Use bulkinsert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br></br>
+> **Default Value**: upsert (Optional)<br></br>
+> `Config Param: OPERATION`<br></br>
---
-> #### hoodie.datasource.hive_sync.username
-> hive user name to use<br></br>
-> **Default Value**: hive (Optional)<br></br>
-> `Config Param: HIVE_USER_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.reconcile.schema
+> When a new batch of write has records with old schema, but latest table schema got evolved, this config will upgrade the records to leverage latest table schema(default values will be injected to missing fields). If not, the write batch would fail.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: RECONCILE_SCHEMA`<br></br>
---
-> #### hoodie.datasource.hive_sync.table_properties
-> Additional properties to store with table.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_TABLE_PROPERTIES`<br></br>
+> #### hoodie.datasource.write.recordkey.field
+> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
+Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using
+the dot notation eg: `a.b.c`<br></br>
+> **Default Value**: uuid (Optional)<br></br>
+> `Config Param: RECORDKEY_FIELD`<br></br>
---
-> #### hoodie.datasource.hive_sync.batch_num
-> The number of partitions one batch when synchronous partitions to hive.<br></br>
-> **Default Value**: 1000 (Optional)<br></br>
-> `Config Param: HIVE_BATCH_SYNC_PARTITION_NUM`<br></br>
+> #### hoodie.datasource.hive_sync.skip_ro_suffix
+> Skip the _ro suffix for Read optimized table, when registering<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE`<br></br>
---
-> #### hoodie.datasource.write.keygenerator.class
-> Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: KEYGENERATOR_CLASS_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.partitionpath.urlencode
+> Should we url encode the partition path value, before creating the folder structure.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: URL_ENCODE_PARTITIONING`<br></br>
---
-> #### hoodie.datasource.hive_sync.base_file_format
-> Base file format for the sync.<br></br>
-> **Default Value**: PARQUET (Optional)<br></br>
-> `Config Param: HIVE_BASE_FILE_FORMAT_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.partition_extractor_class
+> <br></br>
+> **Default Value**: org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor (Optional)<br></br>
+> `Config Param: HIVE_PARTITION_EXTRACTOR_CLASS`<br></br>
---
-> #### hoodie.datasource.write.table.type
-> The table type for the underlying data, for this write. This can’t change between writes.<br></br>
-> **Default Value**: COPY_ON_WRITE (Optional)<br></br>
-> `Config Param: TABLE_TYPE_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.serde_properties
+> <br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_TABLE_SERDE_PROPERTIES`<br></br>
---
-> #### hoodie.datasource.write.operation
-> Whether to do upsert, insert or bulkinsert for the write operation. Use bulkinsert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br></br>
-> **Default Value**: upsert (Optional)<br></br>
-> `Config Param: OPERATION_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.password
+> hive password to use<br></br>
+> **Default Value**: hive (Optional)<br></br>
+> `Config Param: HIVE_PASS`<br></br>
---
-> #### hoodie.datasource.write.streaming.retry.interval.ms
-> Config to indicate how long (by millisecond) before a retry should issued for failed microbatch<br></br>
-> **Default Value**: 2000 (Optional)<br></br>
-> `Config Param: STREAMING_RETRY_INTERVAL_MS_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.support_timestamp
+> ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE`<br></br>
---
-> #### hoodie.datasource.write.hive_style_partitioning
-> Flag to indicate whether to use Hive style partitioning.
-If set true, the names of partition folders follow <partition_column_name>=<partition_value> format.
-By default false (the names of partition folders are only partition values)<br></br>
+> #### hoodie.datasource.hive_sync.create_managed_table
+> Whether to sync the table as managed table.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_STYLE_PARTITIONING_OPT_KEY`<br></br>
+> `Config Param: HIVE_CREATE_MANAGED_TABLE`<br></br>
---
-> #### hoodie.datasource.hive_sync.serde_properties
-> <br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_TABLE_SERDE_PROPERTIES`<br></br>
+> #### hoodie.datasource.clustering.inline.enable
+> Enable inline clustering. Disabled by default.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INLINE_CLUSTERING_ENABLE`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.datasource.write.payload.class
-> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: PAYLOAD_CLASS_OPT_KEY`<br></br>
+> #### hoodie.datasource.compaction.async.enable
+> Controls whether async compaction should be turned on for MOR table writing.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: ASYNC_COMPACT_ENABLE`<br></br>
---
-> #### hoodie.datasource.hive_sync.assume_date_partitioning
-> Assume partitioning is yyyy/mm/dd<br></br>
+> #### hoodie.datasource.meta.sync.enable
+> <br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_ASSUME_DATE_PARTITION_OPT_KEY`<br></br>
+> `Config Param: META_SYNC_ENABLED`<br></br>
---
-> #### hoodie.meta.sync.client.tool.class
-> Sync tool class name used to sync to metastore. Defaults to Hive.<br></br>
-> **Default Value**: org.apache.hudi.hive.HiveSyncTool (Optional)<br></br>
-> `Config Param: META_SYNC_CLIENT_TOOL_CLASS`<br></br>
+> #### hoodie.datasource.write.streaming.ignore.failed.batch
+> Config to indicate whether to ignore any non exception error (e.g. writestatus error) within a streaming microbatch<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: STREAMING_IGNORE_FAILED_BATCH`<br></br>
---
-> #### hoodie.datasource.hive_sync.password
-> hive password to use<br></br>
+> #### hoodie.datasource.write.precombine.field
+> Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br></br>
+> **Default Value**: ts (Optional)<br></br>
+> `Config Param: PRECOMBINE_FIELD`<br></br>
+
+---
+
+> #### hoodie.datasource.hive_sync.username
+> hive user name to use<br></br>
> **Default Value**: hive (Optional)<br></br>
-> `Config Param: HIVE_PASS_OPT_KEY`<br></br>
+> `Config Param: HIVE_USER`<br></br>
---
-> #### hoodie.datasource.hive_sync.create_managed_table
-> Whether to sync the table as managed table.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_CREATE_MANAGED_TABLE`<br></br>
+> #### hoodie.datasource.write.partitionpath.field
+> Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value ontained by invoking .toString()<br></br>
+> **Default Value**: partitionpath (Optional)<br></br>
+> `Config Param: PARTITIONPATH_FIELD`<br></br>
+
+---
+
+> #### hoodie.datasource.write.streaming.retry.count
+> Config to indicate how many times streaming job should retry for a failed micro batch.<br></br>
+> **Default Value**: 3 (Optional)<br></br>
+> `Config Param: STREAMING_RETRY_CNT`<br></br>
---
> #### hoodie.datasource.hive_sync.partition_fields
> field in the table to use for determining hive partition columns.<br></br>
> **Default Value**: (Optional)<br></br>
-> `Config Param: HIVE_PARTITION_FIELDS_OPT_KEY`<br></br>
+> `Config Param: HIVE_PARTITION_FIELDS`<br></br>
---
-> #### hoodie.datasource.write.recordkey.field
-> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
-Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using
-the dot notation eg: `a.b.c`<br></br>
-> **Default Value**: uuid (Optional)<br></br>
-> `Config Param: RECORDKEY_FIELD_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.sync_as_datasource
+> <br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE`<br></br>
---
-> #### hoodie.datasource.write.commitmeta.key.prefix
-> Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata. This is useful to store checkpointing information, in a consistent way with the hudi timeline<br></br>
-> **Default Value**: _ (Optional)<br></br>
-> `Config Param: COMMIT_METADATA_KEYPREFIX_OPT_KEY`<br></br>
+> #### hoodie.sql.insert.mode
+> Insert mode when insert data to pk-table. The optional modes are: upsert, strict and non-strict.For upsert mode, insert statement do the upsert operation for the pk-table which will update the duplicate record.For strict mode, insert statement will keep the primary key uniqueness constraint which do not allow duplicate record.While for non-strict mode, hudi just do the insert operation for the pk-table.<br></br>
+> **Default Value**: upsert (Optional)<br></br>
+> `Config Param: SQL_INSERT_MODE`<br></br>
---
-> #### hoodie.datasource.hive_sync.table
-> table to sync to<br></br>
-> **Default Value**: unknown (Optional)<br></br>
-> `Config Param: HIVE_TABLE_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.use_jdbc
+> Use JDBC when hive synchronization is enabled<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: HIVE_USE_JDBC`<br></br>
+> `Deprecated Version: 0.9.0`<br></br>
---
-> #### hoodie.datasource.hive_sync.auto_create_database
-> Auto create hive database if does not exists<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: HIVE_AUTO_CREATE_DATABASE_OPT_KEY`<br></br>
+> #### hoodie.meta.sync.client.tool.class
+> Sync tool class name used to sync to metastore. Defaults to Hive.<br></br>
+> **Default Value**: org.apache.hudi.hive.HiveSyncTool (Optional)<br></br>
+> `Config Param: META_SYNC_CLIENT_TOOL_CLASS_NAME`<br></br>
---
-> #### hoodie.datasource.write.table.name
-> Table name for the datasource write. Also used to register the table into meta stores.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: TABLE_NAME_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.keygenerator.class
+> Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`<br></br>
+> **Default Value**: org.apache.hudi.keygen.SimpleKeyGenerator (Optional)<br></br>
+> `Config Param: KEYGENERATOR_CLASS_NAME`<br></br>
---
-> #### hoodie.datasource.write.partitionpath.urlencode
-> Should we url encode the partition path value, before creating the folder structure.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: URL_ENCODE_PARTITIONING_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.payload.class
+> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: PAYLOAD_CLASS_NAME`<br></br>
---
-> #### hoodie.datasource.hive_sync.support_timestamp
-> ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_SUPPORT_TIMESTAMP`<br></br>
+> #### hoodie.datasource.hive_sync.table_properties
+> Additional properties to store with table.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_TABLE_PROPERTIES`<br></br>
---
> #### hoodie.datasource.hive_sync.jdbcurl
> Hive metastore url<br></br>
> **Default Value**: jdbc:hive2://localhost:10000 (Optional)<br></br>
-> `Config Param: HIVE_URL_OPT_KEY`<br></br>
+> `Config Param: HIVE_URL`<br></br>
---
-> #### hoodie.datasource.compaction.async.enable
-> Controls whether async compaction should be turned on for MOR table writing.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: ASYNC_COMPACT_ENABLE_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.batch_num
+> The number of partitions one batch when synchronous partitions to hive.<br></br>
+> **Default Value**: 1000 (Optional)<br></br>
+> `Config Param: HIVE_BATCH_SYNC_PARTITION_NUM`<br></br>
---
-> #### hoodie.datasource.write.streaming.retry.count
-> Config to indicate how many times streaming job should retry for a failed micro batch.<br></br>
-> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: STREAMING_RETRY_CNT_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.assume_date_partitioning
+> Assume partitioning is yyyy/mm/dd<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: HIVE_ASSUME_DATE_PARTITION`<br></br>
---
-> #### hoodie.datasource.hive_sync.ignore_exceptions
-> <br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_IGNORE_EXCEPTIONS_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.auto_create_database
+> Auto create hive database if does not exists<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: HIVE_AUTO_CREATE_DATABASE`<br></br>
---
-> #### hoodie.datasource.write.insert.drop.duplicates
-> If set to true, filters out all duplicate records from incoming dataframe, during insert operations.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INSERT_DROP_DUPS_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.database
+> database to sync to<br></br>
+> **Default Value**: default (Optional)<br></br>
+> `Config Param: HIVE_DATABASE`<br></br>
---
-> #### hoodie.datasource.hive_sync.partition_extractor_class
-> <br></br>
-> **Default Value**: org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor (Optional)<br></br>
-> `Config Param: HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.streaming.retry.interval.ms
+> Config to indicate how long (by millisecond) before a retry should issued for failed microbatch<br></br>
+> **Default Value**: 2000 (Optional)<br></br>
+> `Config Param: STREAMING_RETRY_INTERVAL_MS`<br></br>
---
-> #### hoodie.datasource.hive_sync.use_pre_apache_input_format
-> <br></br>
+> #### hoodie.sql.bulk.insert.enable
+> When set to true, the sql insert statement will use bulk insert.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMAT_OPT_KEY`<br></br>
+> `Config Param: SQL_ENABLE_BULK_INSERT`<br></br>
---
-> #### hoodie.datasource.write.partitionpath.field
-> Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value ontained by invoking .toString()<br></br>
-> **Default Value**: partitionpath (Optional)<br></br>
-> `Config Param: PARTITIONPATH_FIELD_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.commitmeta.key.prefix
+> Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata. This is useful to store checkpointing information, in a consistent way with the hudi timeline<br></br>
+> **Default Value**: _ (Optional)<br></br>
+> `Config Param: COMMIT_METADATA_KEYPREFIX`<br></br>
---
-> #### hoodie.datasource.hive_sync.sync_as_datasource
-> <br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE`<br></br>
+> #### hoodie.datasource.write.drop.partition.columns
+> When set to true, will not write the partition columns into hudi. By default, false.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: DROP_PARTITION_COLUMNS`<br></br>
---
-> #### hoodie.deltastreamer.source.kafka.value.deserializer.class
-> This class is used by kafka client to deserialize the records<br></br>
-> **Default Value**: io.confluent.kafka.serializers.KafkaAvroDeserializer (Optional)<br></br>
-> `Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br></br>
-> `Since Version: 0.9.0`<br></br>
+> #### hoodie.datasource.hive_sync.enable
+> When set to true, register/sync the table to Apache Hive metastore<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: HIVE_SYNC_ENABLED`<br></br>
---
-> #### hoodie.datasource.hive_sync.skip_ro_suffix
-> Skip the _ro suffix for Read optimized table, when registering<br></br>
+> #### hoodie.datasource.hive_sync.table
+> table to sync to<br></br>
+> **Default Value**: unknown (Optional)<br></br>
+> `Config Param: HIVE_TABLE`<br></br>
+
+---
+
+> #### hoodie.datasource.hive_sync.ignore_exceptions
+> <br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_SKIP_RO_SUFFIX`<br></br>
+> `Config Param: HIVE_IGNORE_EXCEPTIONS`<br></br>
---
-> #### hoodie.datasource.meta.sync.enable
+> #### hoodie.datasource.hive_sync.use_pre_apache_input_format
> <br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: META_SYNC_ENABLED_OPT_KEY`<br></br>
+> `Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMAT`<br></br>
+
+---
+
+> #### hoodie.datasource.write.table.type
+> The table type for the underlying data, for this write. This can’t change between writes.<br></br>
+> **Default Value**: COPY_ON_WRITE (Optional)<br></br>
+> `Config Param: TABLE_TYPE`<br></br>
---
> #### hoodie.datasource.write.row.writer.enable
> When set to true, will perform write operations directly using the spark native `Row` representation, avoiding any additional conversion costs.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: ENABLE_ROW_WRITER`<br></br>
+
+---
+
+> #### hoodie.datasource.write.hive_style_partitioning
+> Flag to indicate whether to use Hive style partitioning.
+If set true, the names of partition folders follow <partition_column_name>=<partition_value> format.
+By default false (the names of partition folders are only partition values)<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: ENABLE_ROW_WRITER_OPT_KEY`<br></br>
+> `Config Param: HIVE_STYLE_PARTITIONING`<br></br>
---
-> #### hoodie.datasource.write.streaming.ignore.failed.batch
-> Config to indicate whether to ignore any non exception error (e.g. writestatus error) within a streaming microbatch<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: STREAMING_IGNORE_FAILED_BATCH_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.mode
+> Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_SYNC_MODE`<br></br>
---
-> #### hoodie.datasource.clustering.async.enable
-> Enable asynchronous clustering. Disabled by default.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: ASYNC_CLUSTERING_ENABLE_OPT_KEY`<br></br>
-> `Since Version: 0.9.0`<br></br>
+> #### hoodie.datasource.write.table.name
+> Table name for the datasource write. Also used to register the table into meta stores.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: TABLE_NAME`<br></br>
---
-> #### hoodie.datasource.clustering.inline.enable
-> Enable inline clustering. Disabled by default.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INLINE_CLUSTERING_ENABLE_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.base_file_format
+> Base file format for the sync.<br></br>
+> **Default Value**: PARQUET (Optional)<br></br>
+> `Config Param: HIVE_BASE_FILE_FORMAT`<br></br>
+
+---
+
+> #### hoodie.deltastreamer.source.kafka.value.deserializer.class
+> This class is used by kafka client to deserialize the records<br></br>
+> **Default Value**: io.confluent.kafka.serializers.KafkaAvroDeserializer (Optional)<br></br>
+> `Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br></br>
> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.datasource.hive_sync.enable
-> When set to true, register/sync the table to Apache Hive metastore<br></br>
+> #### hoodie.datasource.write.insert.drop.duplicates
+> If set to true, filters out all duplicate records from incoming dataframe, during insert operations.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_SYNC_ENABLED_OPT_KEY`<br></br>
+> `Config Param: INSERT_DROP_DUPS`<br></br>
---
-> #### hoodie.datasource.hive_sync.use_jdbc
-> Use JDBC when hive synchronization is enabled<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: HIVE_USE_JDBC_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.partitions.to.delete
+> Comma separated list of partitions to delete<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: PARTITIONS_TO_DELETE`<br></br>
---
@@ -430,6 +480,13 @@ These configs control the Hudi Flink SQL source/sink connectors, providing abili
Flink jobs using the SQL can be configured through the options in WITH clause. The actual datasource level configs are listed below.
`Config Class`: org.apache.hudi.configuration.FlinkOptions<br></br>
+> #### write.bulk_insert.sort_by_partition
+> Whether to sort the inputs by partition path for bulk insert tasks, default true<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: WRITE_BULK_INSERT_SORT_BY_PARTITION`<br></br>
+
+---
+
> #### read.streaming.enabled
> Whether to read as streaming source, default false<br></br>
> **Default Value**: false (Optional)<br></br>
@@ -484,6 +541,13 @@ Disabled by default for backward compatibility.<br></br>
---
+> #### metadata.compaction.delta_commits
+> Max delta commits for metadata table to trigger compaction, default 24<br></br>
+> **Default Value**: 24 (Optional)<br></br>
+> `Config Param: METADATA_COMPACTION_DELTA_COMMITS`<br></br>
+
+---
+
> #### hive_sync.assume_date_partitioning
> Assume partitioning is yyyy/mm/dd, default false<br></br>
> **Default Value**: false (Optional)<br></br>
@@ -491,6 +555,13 @@ Disabled by default for backward compatibility.<br></br>
---
+> #### write.bulk_insert.shuffle_by_partition
+> Whether to shuffle the inputs by partition path for bulk insert tasks, default true<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: WRITE_BULK_INSERT_SHUFFLE_BY_PARTITION`<br></br>
+
+---
+
> #### hive_sync.table
> Table name for hive sync, default 'unknown'<br></br>
> **Default Value**: unknown (Optional)<br></br>
@@ -498,6 +569,14 @@ Disabled by default for backward compatibility.<br></br>
---
+> #### write.payload.class
+> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting.
+This will render any value set for the option in-effective<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: PAYLOAD_CLASS_NAME`<br></br>
+
+---
+
> #### compaction.tasks
> Parallelism of tasks that do actual compaction, default is 10<br></br>
> **Default Value**: 10 (Optional)<br></br>
@@ -521,13 +600,6 @@ By default false (the names of partition folders are only partition values)<br><
---
-> #### hive_sync.partition_extractor_class
-> Tool to extract the partition value from HDFS path, default 'SlashEncodedDayPartitionValueExtractor'<br></br>
-> **Default Value**: org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor (Optional)<br></br>
-> `Config Param: HIVE_SYNC_PARTITION_EXTRACTOR_CLASS`<br></br>
-
----
-
> #### hive_sync.auto_create_db
> Auto create hive database if it does not exists, default true<br></br>
> **Default Value**: true (Optional)<br></br>
@@ -542,6 +614,13 @@ By default false (the names of partition folders are only partition values)<br><
---
+> #### write.sort.memory
+> Sort memory in MB, default 128MB<br></br>
+> **Default Value**: 128 (Optional)<br></br>
+> `Config Param: WRITE_SORT_MEMORY`<br></br>
+
+---
+
> #### hive_sync.enable
> Asynchronously sync Hive meta to HMS, default false<br></br>
> **Default Value**: false (Optional)<br></br>
@@ -549,6 +628,17 @@ By default false (the names of partition folders are only partition values)<br><
---
+> #### changelog.enabled
+> Whether to keep all the intermediate changes, we try to keep all the changes of a record when enabled:
+1). The sink accept the UPDATE_BEFORE message;
+2). The source try to emit every changes of a record.
+The semantics is best effort because the compaction job would finally merge all changes of a record into one.
+ default false to have UPSERT semantics<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: CHANGELOG_ENABLED`<br></br>
+
+---
+
> #### read.streaming.check-interval
> Check interval for streaming read of SECOND, default 1 minute<br></br>
> **Default Value**: 60 (Optional)<br></br>
@@ -574,6 +664,13 @@ By default 3<br></br>
---
+> #### metadata.enabled
+> Enable the internal metadata table which serves table metadata like level file listings, default false<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: METADATA_ENABLED`<br></br>
+
+---
+
> #### read.tasks
> Parallelism of tasks that do actual read, default is 4<br></br>
> **Default Value**: 4 (Optional)<br></br>
@@ -645,6 +742,13 @@ By default 2000 and it will be doubled by every retry<br></br>
---
+> #### hive_sync.partition_extractor_class
+> Tool to extract the partition value from HDFS path, default 'SlashEncodedDayPartitionValueExtractor'<br></br>
+> **Default Value**: org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor (Optional)<br></br>
+> `Config Param: HIVE_SYNC_PARTITION_EXTRACTOR_CLASS_NAME`<br></br>
+
+---
+
> #### write.batch.size
> Batch buffer size in MB to flush data into the underneath filesystem, default 64MB<br></br>
> **Default Value**: 64.0 (Optional)<br></br>
@@ -659,10 +763,10 @@ By default 2000 and it will be doubled by every retry<br></br>
---
-> #### sink.shuffle-by-partition.enable
-> The option to enable shuffle data by dynamic partition fields in sink phase, this can greatly reduce the number of file for filesystem sink but may lead data skew.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: SINK_SHUFFLE_BY_PARTITION`<br></br>
+> #### hoodie.datasource.write.keygenerator.class
+> Key generator class, that implements will extract the key out of incoming record<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: KEYGEN_CLASS_NAME`<br></br>
---
@@ -682,6 +786,13 @@ By default insert will accept duplicates, to gain extra performance<br></br>
---
+> #### write.insert.deduplicate
+> Whether to deduplicate for INSERT operation, if disabled, writes the base files directly, default true<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: INSERT_DEDUP`<br></br>
+
+---
+
> #### index.partition.regex
> Whether to load partitions in state if partition path matching， default *<br></br>
> **Default Value**: .* (Optional)<br></br>
@@ -756,14 +867,6 @@ waits for the instant commit success, only for internal use<br></br>
---
-> #### write.payload.class
-> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting.
-This will render any value set for the option in-effective<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: PAYLOAD_CLASS`<br></br>
-
----
-
> #### write.operation
> The write operation, that this write should do<br></br>
> **Default Value**: upsert (Optional)<br></br>
@@ -906,13 +1009,6 @@ it flushes the max size data bucket to avoid OOM, default 1GB<br></br>
---
-> #### hoodie.datasource.write.keygenerator.class
-> Key generator class, that implements will extract the key out of incoming record<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: KEYGEN_CLASS`<br></br>
-
----
-
> #### hoodie.datasource.write.recordkey.field
> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br></br>
@@ -921,13 +1017,6 @@ Actual value will be obtained by invoking .toString() on the field value. Nested
---
-> #### write.bulk_insert.partition.sorted
-> Whether the bulk insert write task input records are already sorted by the partition path<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: WRITE_BULK_INSERT_PARTITION_SORTED`<br></br>
-
----
-
> #### compaction.delta_seconds
> Max delta seconds time needed to trigger compaction, default 1 hour<br></br>
> **Default Value**: 3600 (Optional)<br></br>
@@ -967,7 +1056,7 @@ The consistency guard related config options, to help talk to eventually consist
> #### hoodie.optimistic.consistency.guard.sleep_time_ms
> Amount of time (in ms), to wait after which we assume storage is consistent.<br></br>
> **Default Value**: 500 (Optional)<br></br>
-> `Config Param: OPTIMISTIC_CONSISTENCY_GUARD_SLEEP_TIME_MS_PROP`<br></br>
+> `Config Param: OPTIMISTIC_CONSISTENCY_GUARD_SLEEP_TIME_MS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -975,25 +1064,24 @@ The consistency guard related config options, to help talk to eventually consist
> #### hoodie.consistency.check.max_interval_ms
> Maximum amount of time (in ms), to wait for consistency checking.<br></br>
> **Default Value**: 20000 (Optional)<br></br>
-> `Config Param: MAX_CONSISTENCY_CHECK_INTERVAL_MS_PROP`<br></br>
+> `Config Param: MAX_CHECK_INTERVAL_MS`<br></br>
> `Since Version: 0.5.0`<br></br>
> `Deprecated Version: 0.7.0`<br></br>
---
-> #### hoodie.consistency.check.enabled
-> Enabled to handle S3 eventual consistency issue. This property is no longer required since S3 is now strongly consistent. Will be removed in the future releases.<br></br>
+> #### _hoodie.optimistic.consistency.guard.enable
+> Enable consistency guard, which optimistically assumes consistency is achieved after a certain time period.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: CONSISTENCY_CHECK_ENABLED_PROP`<br></br>
-> `Since Version: 0.5.0`<br></br>
-> `Deprecated Version: 0.7.0`<br></br>
+> `Config Param: OPTIMISTIC_CONSISTENCY_GUARD_ENABLE`<br></br>
+> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.consistency.check.initial_interval_ms
-> Amount of time (in ms) to wait, before checking for consistency after an operation on storage.<br></br>
-> **Default Value**: 400 (Optional)<br></br>
-> `Config Param: INITIAL_CONSISTENCY_CHECK_INTERVAL_MS_PROP`<br></br>
+> #### hoodie.consistency.check.enabled
+> Enabled to handle S3 eventual consistency issue. This property is no longer required since S3 is now strongly consistent. Will be removed in the future releases.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ENABLE`<br></br>
> `Since Version: 0.5.0`<br></br>
> `Deprecated Version: 0.7.0`<br></br>
@@ -1002,17 +1090,18 @@ The consistency guard related config options, to help talk to eventually consist
> #### hoodie.consistency.check.max_checks
> Maximum number of consistency checks to perform, with exponential backoff.<br></br>
> **Default Value**: 6 (Optional)<br></br>
-> `Config Param: MAX_CONSISTENCY_CHECKS_PROP`<br></br>
+> `Config Param: MAX_CHECKS`<br></br>
> `Since Version: 0.5.0`<br></br>
> `Deprecated Version: 0.7.0`<br></br>
---
-> #### _hoodie.optimistic.consistency.guard.enable
-> Enable consistency guard, which optimistically assumes consistency is achieved after a certain time period.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: ENABLE_OPTIMISTIC_CONSISTENCY_GUARD_PROP`<br></br>
-> `Since Version: 0.6.0`<br></br>
+> #### hoodie.consistency.check.initial_interval_ms
+> Amount of time (in ms) to wait, before checking for consistency after an operation on storage.<br></br>
+> **Default Value**: 400 (Optional)<br></br>
+> `Config Param: INITIAL_CHECK_INTERVAL_MS`<br></br>
+> `Since Version: 0.5.0`<br></br>
+> `Deprecated Version: 0.7.0`<br></br>
---
@@ -1021,333 +1110,372 @@ The consistency guard related config options, to help talk to eventually consist
Configurations that control write behavior on Hudi tables. These can be directly passed down from even higher level frameworks (e.g Spark datasources, Flink sink) and utilities (e.g DeltaStreamer).
`Config Class`: org.apache.hudi.config.HoodieWriteConfig<br></br>
-> #### hoodie.embed.timeline.server.reuse.enabled
-> Controls whether the timeline server instance should be cached and reused across the JVM (across task lifecycles)to avoid startup costs. This should rarely be changed.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_REUSE_ENABLED`<br></br>
+> #### hoodie.combine.before.upsert
+> When upserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage. This should be turned off only if you are absolutely certain that there are no duplicates incoming, otherwise it can lead to duplicate keys and violate the uniqueness guarantees.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: COMBINE_BEFORE_UPSERT`<br></br>
---
-> #### hoodie.datasource.write.keygenerator.class
-> Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: KEYGENERATOR_CLASS_PROP`<br></br>
+> #### hoodie.write.markers.type
+> Marker type to use. Two modes are supported: - DIRECT: individual marker file corresponding to each data file is directly created by the writer. - TIMELINE_SERVER_BASED: marker operations are all handled at the timeline service which serves as a proxy. New marker entries are batch processed and stored in a limited number of underlying files for efficiency.<br></br>
+> **Default Value**: DIRECT (Optional)<br></br>
+> `Config Param: MARKERS_TYPE`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.bulkinsert.shuffle.parallelism
-> For large initial imports using bulk_insert operation, controls the parallelism to use for sort modes or custom partitioning donebefore writing records to the table.<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: BULKINSERT_PARALLELISM`<br></br>
+> #### hoodie.consistency.check.max_interval_ms
+> Max time to wait between successive attempts at performing consistency checks<br></br>
+> **Default Value**: 300000 (Optional)<br></br>
+> `Config Param: MAX_CONSISTENCY_CHECK_INTERVAL_MS`<br></br>
---
-> #### hoodie.rollback.parallelism
-> Parallelism for rollback of commits. Rollbacks perform delete of files or logging delete blocks to file groups on storage in parallel.<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: ROLLBACK_PARALLELISM`<br></br>
+> #### hoodie.embed.timeline.server.port
+> Port at which the timeline server listens for requests. When running embedded in each writer, it picks a free port and communicates to all the executors. This should rarely be changed.<br></br>
+> **Default Value**: 0 (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_PORT_NUM`<br></br>
---
-> #### hoodie.write.schema
-> The specified write schema. In most case, we do not need set this parameter, but for the case the write schema is not equal to the specified table schema, we can specify the write schema by this parameter. Used by MergeIntoHoodieTableCommand<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: WRITE_SCHEMA_PROP`<br></br>
+> #### hoodie.write.meta.key.prefixes
+> Comma separated metadata key prefixes to override from latest commit during overlapping commits via multi writing<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: WRITE_META_KEY_PREFIXES`<br></br>
---
-> #### hoodie.timeline.layout.version
-> Controls the layout of the timeline. Version 0 relied on renames, Version 1 (default) models the timeline as an immutable log relying only on atomic writes for object storage.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: TIMELINE_LAYOUT_VERSION`<br></br>
-> `Since Version: 0.5.1`<br></br>
+> #### hoodie.table.base.file.format
+> <br></br>
+> **Default Value**: PARQUET (Optional)<br></br>
+> `Config Param: BASE_FILE_FORMAT`<br></br>
---
> #### hoodie.avro.schema.validate
> Validate the schema used for the write against the latest schema, for backwards compatibility.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: AVRO_SCHEMA_VALIDATE`<br></br>
+> `Config Param: AVRO_SCHEMA_VALIDATE_ENABLE`<br></br>
---
-> #### hoodie.bulkinsert.sort.mode
-> Sorting modes to use for sorting records for bulk insert. This is user when user hoodie.bulkinsert.user.defined.partitioner.classis not configured. Available values are - GLOBAL_SORT: this ensures best file sizes, with lowest memory overhead at cost of sorting. PARTITION_SORT: Strikes a balance by only sorting within a partition, still keeping the memory overhead of writing lowest and best effort file sizing. NONE: No sorting. Fastest and matches `spark.write.parquet()` in terms of num [...]
-> **Default Value**: GLOBAL_SORT (Optional)<br></br>
-> `Config Param: BULKINSERT_SORT_MODE`<br></br>
+> #### hoodie.write.buffer.limit.bytes
+> Size of in-memory buffer used for parallelizing network reads and lake storage writes.<br></br>
+> **Default Value**: 4194304 (Optional)<br></br>
+> `Config Param: WRITE_BUFFER_LIMIT_BYTES_VALUE`<br></br>
---
-> #### hoodie.upsert.shuffle.parallelism
-> Parallelism to use for upsert operation on the table. Upserts can shuffle data to perform index lookups, file sizing, bin packing records optimallyinto file groups.<br></br>
+> #### hoodie.insert.shuffle.parallelism
+> Parallelism for inserting records into the table. Inserts can shuffle data before writing to tune file sizes and optimize the storage layout.<br></br>
> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: UPSERT_PARALLELISM`<br></br>
+> `Config Param: INSERT_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.embed.timeline.server.gzip
-> Controls whether gzip compression is used, for large responses from the timeline server, to improve latency.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_COMPRESS_OUTPUT`<br></br>
+> #### hoodie.embed.timeline.server.async
+> Controls whether or not, the requests to the timeline server are processed in asynchronous fashion, potentially improving throughput.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_USE_ASYNC_ENABLE`<br></br>
+
+---
+
+> #### hoodie.rollback.parallelism
+> Parallelism for rollback of commits. Rollbacks perform delete of files or logging delete blocks to file groups on storage in parallel.<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: ROLLBACK_PARALLELISM_VALUE`<br></br>
+
+---
+
+> #### hoodie.write.status.storage.level
+> Write status objects hold metadata about a write (stats, errors), that is not yet committed to storage. This controls the how that information is cached for inspection by clients. We rarely expect this to be changed.<br></br>
+> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
+> `Config Param: WRITE_STATUS_STORAGE_LEVEL_VALUE`<br></br>
+
+---
+
+> #### hoodie.writestatus.class
+> Subclass of org.apache.hudi.client.WriteStatus to be used to collect information about a write. Can be overridden to collection additional metrics/statistics about the data if needed.<br></br>
+> **Default Value**: org.apache.hudi.client.WriteStatus (Optional)<br></br>
+> `Config Param: WRITE_STATUS_CLASS_NAME`<br></br>
+
+---
+
+> #### hoodie.base.path
+> Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: BASE_PATH`<br></br>
---
-> #### hoodie.fail.on.timeline.archiving
-> Timeline archiving removes older instants from the timeline, after each write operation, to minimize metadata overhead. Controls whether or not, the write should be failed as well, if such archiving fails.<br></br>
+> #### hoodie.allow.empty.commit
+> Whether to allow generation of empty commits, even if no data was written in the commit. It's useful in cases where extra metadata needs to be published regardless e.g tracking source offsets when ingesting data<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: FAIL_ON_TIMELINE_ARCHIVING_ENABLED_PROP`<br></br>
+> `Config Param: ALLOW_EMPTY_COMMIT`<br></br>
---
-> #### hoodie.write.status.storage.level
-> Write status objects hold metadata about a write (stats, errors), that is not yet committed to storage. This controls the how that information is cached for inspection by clients. We rarely expect this to be changed.<br></br>
-> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
-> `Config Param: WRITE_STATUS_STORAGE_LEVEL`<br></br>
+> #### hoodie.bulkinsert.user.defined.partitioner.class
+> If specified, this class will be used to re-partition records before they are bulk inserted. This can be used to sort, pack, cluster data optimally for common query patterns. For now we support a build-in user defined bulkinsert partitioner org.apache.hudi.execution.bulkinsert.RDDCustomColumnsSortPartitioner which can does sorting based on specified column values set by hoodie.bulkinsert.user.defined.partitioner.sort.columns<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: BULKINSERT_USER_DEFINED_PARTITIONER_CLASS_NAME`<br></br>
---
-> #### hoodie.table.base.file.format
-> <br></br>
-> **Default Value**: PARQUET (Optional)<br></br>
-> `Config Param: BASE_FILE_FORMAT`<br></br>
+> #### hoodie.table.name
+> Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: TBL_NAME`<br></br>
---
-> #### _.hoodie.allow.multi.write.on.same.instant
-> <br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: ALLOW_MULTI_WRITE_ON_SAME_INSTANT`<br></br>
+> #### hoodie.combine.before.delete
+> During delete operations, controls whether we should combine deletes (and potentially also upserts) before writing to storage.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: COMBINE_BEFORE_DELETE`<br></br>
---
-> #### hoodie.embed.timeline.server.port
-> Port at which the timeline server listens for requests. When running embedded in each writer, it picks a free port and communicates to all the executors. This should rarely be changed.<br></br>
-> **Default Value**: 0 (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_PORT`<br></br>
+> #### hoodie.embed.timeline.server.threads
+> Number of threads to serve requests in the timeline server. By default, auto configured based on the number of underlying cores.<br></br>
+> **Default Value**: -1 (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_NUM_SERVER_THREADS`<br></br>
---
-> #### hoodie.consistency.check.max_interval_ms
-> Max time to wait between successive attempts at performing consistency checks<br></br>
-> **Default Value**: 300000 (Optional)<br></br>
-> `Config Param: MAX_CONSISTENCY_CHECK_INTERVAL_MS_PROP`<br></br>
+> #### hoodie.fail.on.timeline.archiving
+> Timeline archiving removes older instants from the timeline, after each write operation, to minimize metadata overhead. Controls whether or not, the write should be failed as well, if such archiving fails.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: FAIL_ON_TIMELINE_ARCHIVING_ENABLE`<br></br>
---
-> #### hoodie.write.concurrency.mode
-> Enable different concurrency modes. Options are SINGLE_WRITER: Only one active writer to the table. Maximizes throughputOPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table and exactly one of them succeed if a conflict (writes affect the same file group) is detected.<br></br>
-> **Default Value**: SINGLE_WRITER (Optional)<br></br>
-> `Config Param: WRITE_CONCURRENCY_MODE_PROP`<br></br>
+> #### hoodie.datasource.write.keygenerator.class
+> Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: KEYGENERATOR_CLASS_NAME`<br></br>
---
-> #### hoodie.delete.shuffle.parallelism
-> Parallelism used for “delete” operation. Delete operations also performs shuffles, similar to upsert operation.<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: DELETE_PARALLELISM`<br></br>
+> #### hoodie.combine.before.insert
+> When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: COMBINE_BEFORE_INSERT`<br></br>
---
-> #### hoodie.embed.timeline.server
-> When true, spins up an instance of the timeline server (meta server that serves cached file listings, statistics),running on each writer's driver process, accepting requests during the write from executors.<br></br>
+> #### hoodie.embed.timeline.server.gzip
+> Controls whether gzip compression is used, for large responses from the timeline server, to improve latency.<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_ENABLED`<br></br>
-
----
-
-> #### hoodie.base.path
-> Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BASE_PATH_PROP`<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_COMPRESS_ENABLE`<br></br>
---
-> #### hoodie.avro.schema
-> Schema string representing the current write schema of the table. Hudi passes this to implementations of HoodieRecordPayload to convert incoming records to avro. This is also used as the write schema evolving records during an update.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: AVRO_SCHEMA`<br></br>
+> #### hoodie.markers.timeline_server_based.batch.interval_ms
+> The batch interval in milliseconds for marker creation batch processing<br></br>
+> **Default Value**: 50 (Optional)<br></br>
+> `Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_INTERVAL_MS`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.allow.empty.commit
-> Whether to allow generation of empty commits, even if no data was written in the commit. It's useful in cases where extra metadata needs to be published regardless e.g tracking source offsets when ingesting data<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: ALLOW_EMPTY_COMMIT`<br></br>
+> #### hoodie.markers.timeline_server_based.batch.num_threads
+> Number of threads to use for batch processing marker creation requests at the timeline server<br></br>
+> **Default Value**: 20 (Optional)<br></br>
+> `Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_NUM_THREADS`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.table.name
-> Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: TABLE_NAME`<br></br>
+> #### _.hoodie.allow.multi.write.on.same.instant
+> <br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ALLOW_MULTI_WRITE_ON_SAME_INSTANT_ENABLE`<br></br>
---
-> #### hoodie.embed.timeline.server.async
-> Controls whether or not, the requests to the timeline server are processed in asynchronous fashion, potentially improving throughput.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_USE_ASYNC`<br></br>
+> #### hoodie.datasource.write.payload.class
+> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: WRITE_PAYLOAD_CLASS_NAME`<br></br>
---
-> #### hoodie.combine.before.upsert
-> When upserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage. This should be turned off only if you are absolutely certain that there are no duplicates incoming, otherwise it can lead to duplicate keys and violate the uniqueness guarantees.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: COMBINE_BEFORE_UPSERT_PROP`<br></br>
+> #### hoodie.bulkinsert.shuffle.parallelism
+> For large initial imports using bulk_insert operation, controls the parallelism to use for sort modes or custom partitioning donebefore writing records to the table.<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: BULKINSERT_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.embed.timeline.server.threads
-> Number of threads to serve requests in the timeline server. By default, auto configured based on the number of underlying cores.<br></br>
-> **Default Value**: -1 (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_THREADS`<br></br>
+> #### hoodie.delete.shuffle.parallelism
+> Parallelism used for “delete” operation. Delete operations also performs shuffles, similar to upsert operation.<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: DELETE_PARALLELISM_VALUE`<br></br>
---
> #### hoodie.consistency.check.max_checks
> Maximum number of checks, for consistency of written data.<br></br>
> **Default Value**: 7 (Optional)<br></br>
-> `Config Param: MAX_CONSISTENCY_CHECKS_PROP`<br></br>
-
----
-
-> #### hoodie.combine.before.delete
-> During delete operations, controls whether we should combine deletes (and potentially also upserts) before writing to storage.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: COMBINE_BEFORE_DELETE_PROP`<br></br>
+> `Config Param: MAX_CONSISTENCY_CHECKS`<br></br>
---
> #### hoodie.datasource.write.keygenerator.type
> Easily configure one the built-in key generators, instead of specifying the key generator class.Currently supports SIMPLE, COMPLEX, TIMESTAMP, CUSTOM, NON_PARTITION, GLOBAL_DELETE<br></br>
> **Default Value**: SIMPLE (Optional)<br></br>
-> `Config Param: KEYGENERATOR_TYPE_PROP`<br></br>
+> `Config Param: KEYGENERATOR_TYPE`<br></br>
---
-> #### hoodie.write.buffer.limit.bytes
-> Size of in-memory buffer used for parallelizing network reads and lake storage writes.<br></br>
-> **Default Value**: 4194304 (Optional)<br></br>
-> `Config Param: WRITE_BUFFER_LIMIT_BYTES`<br></br>
+> #### hoodie.merge.allow.duplicate.on.inserts
+> When enabled, we allow duplicate keys even if inserts are routed to merge with an existing file (for ensuring file sizing). This is only relevant for insert operation, since upsert, delete operations will ensure unique key constraints are maintained.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE`<br></br>
---
-> #### hoodie.bulkinsert.user.defined.partitioner.class
-> If specified, this class will be used to re-partition records before they are bulk inserted. This can be used to sort, pack, cluster data optimally for common query patterns.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BULKINSERT_USER_DEFINED_PARTITIONER_CLASS`<br></br>
+> #### hoodie.embed.timeline.server.reuse.enabled
+> Controls whether the timeline server instance should be cached and reused across the JVM (across task lifecycles)to avoid startup costs. This should rarely be changed.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_REUSE_ENABLED`<br></br>
---
-> #### hoodie.client.heartbeat.interval_in_ms
-> Writers perform heartbeats to indicate liveness. Controls how often (in ms), such heartbeats are registered to lake storage.<br></br>
-> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: CLIENT_HEARTBEAT_INTERVAL_IN_MS_PROP`<br></br>
+> #### hoodie.datasource.write.precombine.field
+> Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br></br>
+> **Default Value**: ts (Optional)<br></br>
+> `Config Param: PRECOMBINE_FIELD_NAME`<br></br>
---
-> #### hoodie.avro.schema.external.transformation
-> When enabled, records in older schema are rewritten into newer schema during upsert,delete and background compaction,clustering operations.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION`<br></br>
+> #### hoodie.bulkinsert.sort.mode
+> Sorting modes to use for sorting records for bulk insert. This is user when user hoodie.bulkinsert.user.defined.partitioner.classis not configured. Available values are - GLOBAL_SORT: this ensures best file sizes, with lowest memory overhead at cost of sorting. PARTITION_SORT: Strikes a balance by only sorting within a partition, still keeping the memory overhead of writing lowest and best effort file sizing. NONE: No sorting. Fastest and matches `spark.write.parquet()` in terms of num [...]
+> **Default Value**: GLOBAL_SORT (Optional)<br></br>
+> `Config Param: BULK_INSERT_SORT_MODE`<br></br>
---
-> #### hoodie.client.heartbeat.tolerable.misses
-> Number of heartbeat misses, before a writer is deemed not alive and all pending writes are aborted.<br></br>
-> **Default Value**: 2 (Optional)<br></br>
-> `Config Param: CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES_PROP`<br></br>
+> #### hoodie.avro.schema
+> Schema string representing the current write schema of the table. Hudi passes this to implementations of HoodieRecordPayload to convert incoming records to avro. This is also used as the write schema evolving records during an update.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: AVRO_SCHEMA_STRING`<br></br>
---
> #### hoodie.auto.commit
> Controls whether a write operation should auto commit. This can be turned off to perform inspection of the uncommitted write before deciding to commit.<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: HOODIE_AUTO_COMMIT_PROP`<br></br>
+> `Config Param: AUTO_COMMIT_ENABLE`<br></br>
---
-> #### hoodie.combine.before.insert
-> When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: COMBINE_BEFORE_INSERT_PROP`<br></br>
+> #### hoodie.embed.timeline.server
+> When true, spins up an instance of the timeline server (meta server that serves cached file listings, statistics),running on each writer's driver process, accepting requests during the write from executors.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_ENABLE`<br></br>
---
-> #### hoodie.writestatus.class
-> Subclass of org.apache.hudi.client.WriteStatus to be used to collect information about a write. Can be overridden to collection additional metrics/statistics about the data if needed.<br></br>
-> **Default Value**: org.apache.hudi.client.WriteStatus (Optional)<br></br>
-> `Config Param: HOODIE_WRITE_STATUS_CLASS_PROP`<br></br>
+> #### hoodie.timeline.layout.version
+> Controls the layout of the timeline. Version 0 relied on renames, Version 1 (default) models the timeline as an immutable log relying only on atomic writes for object storage.<br></br>
+> **Default Value**: 1 (Optional)<br></br>
+> `Config Param: TIMELINE_LAYOUT_VERSION_NUM`<br></br>
+> `Since Version: 0.5.1`<br></br>
---
-> #### hoodie.markers.delete.parallelism
-> Determines the parallelism for deleting marker files, which are used to track all files (valid or invalid/partial) written during a write operation. Increase this value if delays are observed, with large batch writes.<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: MARKERS_DELETE_PARALLELISM`<br></br>
+> #### hoodie.upsert.shuffle.parallelism
+> Parallelism to use for upsert operation on the table. Upserts can shuffle data to perform index lookups, file sizing, bin packing records optimallyinto file groups.<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: UPSERT_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.consistency.check.initial_interval_ms
-> Initial time between successive attempts to ensure written data's metadata is consistent on storage. Grows with exponential backoff after the initial value.<br></br>
-> **Default Value**: 2000 (Optional)<br></br>
-> `Config Param: INITIAL_CONSISTENCY_CHECK_INTERVAL_MS_PROP`<br></br>
+> #### hoodie.write.schema
+> The specified write schema. In most case, we do not need set this parameter, but for the case the write schema is not equal to the specified table schema, we can specify the write schema by this parameter. Used by MergeIntoHoodieTableCommand<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: WRITE_SCHEMA`<br></br>
+
+---
+
+> #### hoodie.rollback.using.markers
+> Enables a more efficient mechanism for rollbacks based on the marker files generated during the writes. Turned off by default.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ROLLBACK_USING_MARKERS_ENABLE`<br></br>
---
> #### hoodie.merge.data.validation.enabled
> When enabled, data validation checks are performed during merges to ensure expected number of records after merge operation.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: MERGE_DATA_VALIDATION_CHECK_ENABLED`<br></br>
+> `Config Param: MERGE_DATA_VALIDATION_CHECK_ENABLE`<br></br>
---
-> #### hoodie.datasource.write.payload.class
-> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: WRITE_PAYLOAD_CLASS`<br></br>
+> #### hoodie.client.heartbeat.tolerable.misses
+> Number of heartbeat misses, before a writer is deemed not alive and all pending writes are aborted.<br></br>
+> **Default Value**: 2 (Optional)<br></br>
+> `Config Param: CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES`<br></br>
---
-> #### hoodie.insert.shuffle.parallelism
-> Parallelism for inserting records into the table. Inserts can shuffle data before writing to tune file sizes and optimize the storage layout.<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: INSERT_PARALLELISM`<br></br>
+> #### hoodie.write.concurrency.mode
+> Enable different concurrency modes. Options are SINGLE_WRITER: Only one active writer to the table. Maximizes throughputOPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table and exactly one of them succeed if a conflict (writes affect the same file group) is detected.<br></br>
+> **Default Value**: SINGLE_WRITER (Optional)<br></br>
+> `Config Param: WRITE_CONCURRENCY_MODE`<br></br>
---
-> #### hoodie.datasource.write.precombine.field
-> Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br></br>
-> **Default Value**: ts (Optional)<br></br>
-> `Config Param: PRECOMBINE_FIELD_PROP`<br></br>
+> #### hoodie.markers.delete.parallelism
+> Determines the parallelism for deleting marker files, which are used to track all files (valid or invalid/partial) written during a write operation. Increase this value if delays are observed, with large batch writes.<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: MARKERS_DELETE_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.write.meta.key.prefixes
-> Comma separated metadata key prefixes to override from latest commit during overlapping commits via multi writing<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: WRITE_META_KEY_PREFIXES_PROP`<br></br>
+> #### hoodie.bulkinsert.user.defined.partitioner.sort.columns
+> Columns to sort the data by when use org.apache.hudi.execution.bulkinsert.RDDCustomColumnsSortPartitioner as user defined partitioner during bulk_insert. For example 'column1,column2'<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: BULKINSERT_USER_DEFINED_PARTITIONER_SORT_COLUMNS`<br></br>
---
> #### hoodie.finalize.write.parallelism
> Parallelism for the write finalization internal operation, which involves removing any partially written files from lake storage, before committing the write. Reduce this value, if the high number of tasks incur delays for smaller tables or low latency writes.<br></br>
> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: FINALIZE_WRITE_PARALLELISM`<br></br>
+> `Config Param: FINALIZE_WRITE_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.merge.allow.duplicate.on.inserts
-> When enabled, we allow duplicate keys even if inserts are routed to merge with an existing file (for ensuring file sizing). This is only relevant for insert operation, since upsert, delete operations will ensure unique key constraints are maintained.<br></br>
+> #### hoodie.client.heartbeat.interval_in_ms
+> Writers perform heartbeats to indicate liveness. Controls how often (in ms), such heartbeats are registered to lake storage.<br></br>
+> **Default Value**: 60000 (Optional)<br></br>
+> `Config Param: CLIENT_HEARTBEAT_INTERVAL_IN_MS`<br></br>
+
+---
+
+> #### hoodie.allow.operation.metadata.field
+> Whether to include '_hoodie_operation' in the metadata fields. Once enabled, all the changes of a record are persisted to the delta log directly without merge<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: MERGE_ALLOW_DUPLICATE_ON_INSERTS`<br></br>
+> `Config Param: ALLOW_OPERATION_METADATA_FIELD`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.rollback.using.markers
-> Enables a more efficient mechanism for rollbacks based on the marker files generated during the writes. Turned off by default.<br></br>
+> #### hoodie.consistency.check.initial_interval_ms
+> Initial time between successive attempts to ensure written data's metadata is consistent on storage. Grows with exponential backoff after the initial value.<br></br>
+> **Default Value**: 2000 (Optional)<br></br>
+> `Config Param: INITIAL_CONSISTENCY_CHECK_INTERVAL_MS`<br></br>
+
+---
+
+> #### hoodie.avro.schema.external.transformation
+> When enabled, records in older schema are rewritten into newer schema during upsert,delete and background compaction,clustering operations.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: ROLLBACK_USING_MARKERS`<br></br>
+> `Config Param: AVRO_EXTERNAL_SCHEMA_TRANSFORMATION_ENABLE`<br></br>
---
@@ -1359,16 +1487,7 @@ Hudi maintains keys (record key + partition path) for uniquely identifying a par
> #### hoodie.datasource.write.partitionpath.urlencode
> Should we url encode the partition path value, before creating the folder structure.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: URL_ENCODE_PARTITIONING_OPT_KEY`<br></br>
-
----
-
-> #### hoodie.datasource.write.recordkey.field
-> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
-Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using
-the dot notation eg: `a.b.c`<br></br>
-> **Default Value**: uuid (Optional)<br></br>
-> `Config Param: RECORDKEY_FIELD_OPT_KEY`<br></br>
+> `Config Param: URL_ENCODE_PARTITIONING`<br></br>
---
@@ -1377,14 +1496,23 @@ the dot notation eg: `a.b.c`<br></br>
If set true, the names of partition folders follow <partition_column_name>=<partition_value> format.
By default false (the names of partition folders are only partition values)<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_STYLE_PARTITIONING_OPT_KEY`<br></br>
+> `Config Param: HIVE_STYLE_PARTITIONING_ENABLE`<br></br>
---
> #### hoodie.datasource.write.partitionpath.field
> Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value ontained by invoking .toString()<br></br>
> **Default Value**: partitionpath (Optional)<br></br>
-> `Config Param: PARTITIONPATH_FIELD_OPT_KEY`<br></br>
+> `Config Param: PARTITIONPATH_FIELD_NAME`<br></br>
+
+---
+
+> #### hoodie.datasource.write.recordkey.field
+> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
+Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using
+the dot notation eg: `a.b.c`<br></br>
+> **Default Value**: uuid (Optional)<br></br>
+> `Config Param: RECORDKEY_FIELD_NAME`<br></br>
---
@@ -1393,38 +1521,52 @@ By default false (the names of partition folders are only partition values)<br><
Configurations that control indexing behavior (when HBase based indexing is enabled), which tags incoming records as either inserts or updates to older records.
`Config Class`: org.apache.hudi.config.HoodieHBaseIndexConfig<br></br>
-> #### hoodie.index.hbase.qps.fraction
-> Property to set the fraction of the global share of QPS that should be allocated to this job. Let's say there are 3 jobs which have input size in terms of number of rows required for HbaseIndexing as x, 2x, 3x respectively. Then this fraction for the jobs would be (0.17) 1/6, 0.33 (2/6) and 0.5 (3/6) respectively. Default is 50%, which means a total of 2 jobs can run using HbaseIndex without overwhelming Region Servers.<br></br>
-> **Default Value**: 0.5 (Optional)<br></br>
-> `Config Param: HBASE_QPS_FRACTION_PROP`<br></br>
+> #### hoodie.index.hbase.zkport
+> Only applies if index type is HBASE. HBase ZK Quorum port to connect to<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: ZKPORT`<br></br>
---
-> #### hoodie.index.hbase.zknode.path
-> Only applies if index type is HBASE. This is the root znode that will contain all the znodes created/used by HBase<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_ZK_ZNODEPARENT`<br></br>
+> #### hoodie.hbase.index.update.partition.path
+> Only applies if index type is HBASE. When an already existing record is upserted to a new partition compared to whats in storage, this config when set, will delete old record in old paritition and will insert it as new record in new partition.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: UPDATE_PARTITION_PATH_ENABLE`<br></br>
---
-> #### hoodie.index.hbase.zkpath.qps_root
-> chroot in zookeeper, to use for all qps allocation co-ordination.<br></br>
-> **Default Value**: /QPS_ROOT (Optional)<br></br>
-> `Config Param: HBASE_ZK_PATH_QPS_ROOT`<br></br>
+> #### hoodie.index.hbase.qps.allocator.class
+> Property to set which implementation of HBase QPS resource allocator to be used, whichcontrols the batching rate dynamically.<br></br>
+> **Default Value**: org.apache.hudi.index.hbase.DefaultHBaseQPSResourceAllocator (Optional)<br></br>
+> `Config Param: QPS_ALLOCATOR_CLASS_NAME`<br></br>
---
-> #### hoodie.index.hbase.put.batch.size
-> Controls the batch size for performing puts against HBase. Batching improves throughput, by saving round trips.<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: HBASE_PUT_BATCH_SIZE_PROP`<br></br>
+> #### hoodie.index.hbase.put.batch.size.autocompute
+> Property to set to enable auto computation of put batch size<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: PUT_BATCH_SIZE_AUTO_COMPUTE`<br></br>
---
-> #### hoodie.index.hbase.dynamic_qps
-> Property to decide if HBASE_QPS_FRACTION_PROP is dynamically calculated based on write volume.<br></br>
+> #### hoodie.index.hbase.rollback.sync
+> When set to true, the rollback method will delete the last failed task index. The default value is false. Because deleting the index will add extra load on the Hbase cluster for each rollback<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HOODIE_INDEX_COMPUTE_QPS_DYNAMICALLY`<br></br>
+> `Config Param: ROLLBACK_SYNC_ENABLE`<br></br>
+
+---
+
+> #### hoodie.index.hbase.get.batch.size
+> Controls the batch size for performing gets against HBase. Batching improves throughput, by saving round trips.<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: GET_BATCH_SIZE`<br></br>
+
+---
+
+> #### hoodie.index.hbase.zkpath.qps_root
+> chroot in zookeeper, to use for all qps allocation co-ordination.<br></br>
+> **Default Value**: /QPS_ROOT (Optional)<br></br>
+> `Config Param: ZKPATH_QPS_ROOT`<br></br>
---
@@ -1434,112 +1576,98 @@ Configurations that control indexing behavior (when HBase based indexing is enab
value based on global indexing throughput needs and most importantly, how much the HBase installation in use is
able to tolerate without Region Servers going down.<br></br>
> **Default Value**: 1000 (Optional)<br></br>
-> `Config Param: HBASE_MAX_QPS_PER_REGION_SERVER_PROP`<br></br>
-
----
-
-> #### hoodie.index.hbase.zk.session_timeout_ms
-> Session timeout value to use for Zookeeper failure detection, for the HBase client.Lower this value, if you want to fail faster.<br></br>
-> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: HOODIE_INDEX_HBASE_ZK_SESSION_TIMEOUT_MS`<br></br>
+> `Config Param: MAX_QPS_PER_REGION_SERVER`<br></br>
---
-> #### hoodie.index.hbase.sleep.ms.for.get.batch
-> <br></br>
+> #### hoodie.index.hbase.max.qps.fraction
+> Maximum for HBASE_QPS_FRACTION_PROP to stabilize skewed write workloads<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_SLEEP_MS_GET_BATCH_PROP`<br></br>
+> `Config Param: MAX_QPS_FRACTION`<br></br>
---
> #### hoodie.index.hbase.min.qps.fraction
> Minimum for HBASE_QPS_FRACTION_PROP to stabilize skewed write workloads<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_MIN_QPS_FRACTION_PROP`<br></br>
+> `Config Param: MIN_QPS_FRACTION`<br></br>
---
-> #### hoodie.index.hbase.desired_puts_time_in_secs
-> <br></br>
-> **Default Value**: 600 (Optional)<br></br>
-> `Config Param: HOODIE_INDEX_DESIRED_PUTS_TIME_IN_SECS`<br></br>
+> #### hoodie.index.hbase.zk.connection_timeout_ms
+> Timeout to use for establishing connection with zookeeper, from HBase client.<br></br>
+> **Default Value**: 15000 (Optional)<br></br>
+> `Config Param: ZK_CONNECTION_TIMEOUT_MS`<br></br>
---
-> #### hoodie.hbase.index.update.partition.path
-> Only applies if index type is HBASE. When an already existing record is upserted to a new partition compared to whats in storage, this config when set, will delete old record in old paritition and will insert it as new record in new partition.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HBASE_INDEX_UPDATE_PARTITION_PATH`<br></br>
+> #### hoodie.index.hbase.table
+> Only applies if index type is HBASE. HBase Table name to use as the index. Hudi stores the row_key and [partition_path, fileID, commitTime] mapping in the table<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: TABLENAME`<br></br>
---
-> #### hoodie.index.hbase.qps.allocator.class
-> Property to set which implementation of HBase QPS resource allocator to be used, whichcontrols the batching rate dynamically.<br></br>
-> **Default Value**: org.apache.hudi.index.hbase.DefaultHBaseQPSResourceAllocator (Optional)<br></br>
-> `Config Param: HBASE_INDEX_QPS_ALLOCATOR_CLASS`<br></br>
+> #### hoodie.index.hbase.dynamic_qps
+> Property to decide if HBASE_QPS_FRACTION_PROP is dynamically calculated based on write volume.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: COMPUTE_QPS_DYNAMICALLY`<br></br>
---
-> #### hoodie.index.hbase.sleep.ms.for.put.batch
-> <br></br>
+> #### hoodie.index.hbase.zknode.path
+> Only applies if index type is HBASE. This is the root znode that will contain all the znodes created/used by HBase<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_SLEEP_MS_PUT_BATCH_PROP`<br></br>
-
----
-
-> #### hoodie.index.hbase.rollback.sync
-> When set to true, the rollback method will delete the last failed task index. The default value is false. Because deleting the index will add extra load on the Hbase cluster for each rollback<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HBASE_INDEX_ROLLBACK_SYNC`<br></br>
+> `Config Param: ZK_NODE_PATH`<br></br>
---
> #### hoodie.index.hbase.zkquorum
> Only applies if index type is HBASE. HBase ZK Quorum url to connect to<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_ZKQUORUM_PROP`<br></br>
+> `Config Param: ZKQUORUM`<br></br>
---
-> #### hoodie.index.hbase.get.batch.size
-> Controls the batch size for performing gets against HBase. Batching improves throughput, by saving round trips.<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: HBASE_GET_BATCH_SIZE_PROP`<br></br>
+> #### hoodie.index.hbase.qps.fraction
+> Property to set the fraction of the global share of QPS that should be allocated to this job. Let's say there are 3 jobs which have input size in terms of number of rows required for HbaseIndexing as x, 2x, 3x respectively. Then this fraction for the jobs would be (0.17) 1/6, 0.33 (2/6) and 0.5 (3/6) respectively. Default is 50%, which means a total of 2 jobs can run using HbaseIndex without overwhelming Region Servers.<br></br>
+> **Default Value**: 0.5 (Optional)<br></br>
+> `Config Param: QPS_FRACTION`<br></br>
---
-> #### hoodie.index.hbase.zk.connection_timeout_ms
-> Timeout to use for establishing connection with zookeeper, from HBase client.<br></br>
-> **Default Value**: 15000 (Optional)<br></br>
-> `Config Param: HOODIE_INDEX_HBASE_ZK_CONNECTION_TIMEOUT_MS`<br></br>
+> #### hoodie.index.hbase.zk.session_timeout_ms
+> Session timeout value to use for Zookeeper failure detection, for the HBase client.Lower this value, if you want to fail faster.<br></br>
+> **Default Value**: 60000 (Optional)<br></br>
+> `Config Param: ZK_SESSION_TIMEOUT_MS`<br></br>
---
-> #### hoodie.index.hbase.put.batch.size.autocompute
-> Property to set to enable auto computation of put batch size<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HBASE_PUT_BATCH_SIZE_AUTO_COMPUTE_PROP`<br></br>
+> #### hoodie.index.hbase.put.batch.size
+> Controls the batch size for performing puts against HBase. Batching improves throughput, by saving round trips.<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: PUT_BATCH_SIZE`<br></br>
---
-> #### hoodie.index.hbase.zkport
-> Only applies if index type is HBASE. HBase ZK Quorum port to connect to<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_ZKPORT_PROP`<br></br>
+> #### hoodie.index.hbase.desired_puts_time_in_secs
+> <br></br>
+> **Default Value**: 600 (Optional)<br></br>
+> `Config Param: DESIRED_PUTS_TIME_IN_SECONDS`<br></br>
---
-> #### hoodie.index.hbase.max.qps.fraction
-> Maximum for HBASE_QPS_FRACTION_PROP to stabilize skewed write workloads<br></br>
+> #### hoodie.index.hbase.sleep.ms.for.put.batch
+> <br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_MAX_QPS_FRACTION_PROP`<br></br>
+> `Config Param: SLEEP_MS_FOR_PUT_BATCH`<br></br>
---
-> #### hoodie.index.hbase.table
-> Only applies if index type is HBASE. HBase Table name to use as the index. Hudi stores the row_key and [partition_path, fileID, commitTime] mapping in the table<br></br>
+> #### hoodie.index.hbase.sleep.ms.for.get.batch
+> <br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_TABLENAME_PROP`<br></br>
+> `Config Param: SLEEP_MS_FOR_GET_BATCH`<br></br>
---
@@ -1548,18 +1676,10 @@ Configurations that control indexing behavior (when HBase based indexing is enab
Controls callback behavior into HTTP endpoints, to push notifications on commits on hudi tables.
`Config Class`: org.apache.hudi.config.HoodieWriteCommitCallbackConfig<br></br>
-> #### hoodie.write.commit.callback.http.api.key
-> Http callback API key. hudi_write_commit_http_callback by default<br></br>
-> **Default Value**: hudi_write_commit_http_callback (Optional)<br></br>
-> `Config Param: CALLBACK_HTTP_API_KEY`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
> #### hoodie.write.commit.callback.on
> Turn commit callback on/off. off by default.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: CALLBACK_ON`<br></br>
+> `Config Param: TURN_CALLBACK_ON`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -1567,7 +1687,7 @@ Controls callback behavior into HTTP endpoints, to push notifications on commit
> #### hoodie.write.commit.callback.http.url
> Callback host to be sent along with callback messages<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CALLBACK_HTTP_URL_PROP`<br></br>
+> `Config Param: CALLBACK_HTTP_URL`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -1575,7 +1695,7 @@ Controls callback behavior into HTTP endpoints, to push notifications on commit
> #### hoodie.write.commit.callback.http.timeout.seconds
> Callback timeout in seconds. 3 by default<br></br>
> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: CALLBACK_HTTP_TIMEOUT_SECONDS`<br></br>
+> `Config Param: CALLBACK_HTTP_TIMEOUT_IN_SECONDS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -1583,7 +1703,15 @@ Controls callback behavior into HTTP endpoints, to push notifications on commit
> #### hoodie.write.commit.callback.class
> Full path of callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default<br></br>
> **Default Value**: org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback (Optional)<br></br>
-> `Config Param: CALLBACK_CLASS_PROP`<br></br>
+> `Config Param: CALLBACK_CLASS_NAME`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.write.commit.callback.http.api.key
+> Http callback API key. hudi_write_commit_http_callback by default<br></br>
+> **Default Value**: hudi_write_commit_http_callback (Optional)<br></br>
+> `Config Param: CALLBACK_HTTP_API_KEY_VALUE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -1593,26 +1721,18 @@ Controls callback behavior into HTTP endpoints, to push notifications on commit
Controls notifications sent to Kafka, on events happening to a hudi table.
`Config Class`: org.apache.hudi.utilities.callback.kafka.HoodieWriteCommitKafkaCallbackConfig<br></br>
-> #### hoodie.write.commit.callback.kafka.acks
-> kafka acks level, all by default to ensure strong durability.<br></br>
-> **Default Value**: all (Optional)<br></br>
-> `Config Param: CALLBACK_KAFKA_ACKS`<br></br>
-> `Since Version: 0.7.0`<br></br>
-
----
-
-> #### hoodie.write.commit.callback.kafka.bootstrap.servers
-> Bootstrap servers of kafka cluster, to be used for publishing commit metadata.<br></br>
+> #### hoodie.write.commit.callback.kafka.topic
+> Kafka topic name to publish timeline activity into.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CALLBACK_KAFKA_BOOTSTRAP_SERVERS`<br></br>
+> `Config Param: TOPIC`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.write.commit.callback.kafka.topic
-> Kafka topic name to publish timeline activity into.<br></br>
+> #### hoodie.write.commit.callback.kafka.partition
+> It may be desirable to serialize all changes into a single Kafka partition for providing strict ordering. By default, Kafka messages are keyed by table name, which guarantees ordering at the table level, but not globally (or when new partitions are added)<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CALLBACK_KAFKA_TOPIC`<br></br>
+> `Config Param: PARTITION`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -1620,15 +1740,23 @@ Controls notifications sent to Kafka, on events happening to a hudi table.
> #### hoodie.write.commit.callback.kafka.retries
> Times to retry the produce. 3 by default<br></br>
> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: CALLBACK_KAFKA_RETRIES`<br></br>
+> `Config Param: RETRIES`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.write.commit.callback.kafka.partition
-> It may be desirable to serialize all changes into a single Kafka partition for providing strict ordering. By default, Kafka messages are keyed by table name, which guarantees ordering at the table level, but not globally (or when new partitions are added)<br></br>
+> #### hoodie.write.commit.callback.kafka.acks
+> kafka acks level, all by default to ensure strong durability.<br></br>
+> **Default Value**: all (Optional)<br></br>
+> `Config Param: ACKS`<br></br>
+> `Since Version: 0.7.0`<br></br>
+
+---
+
+> #### hoodie.write.commit.callback.kafka.bootstrap.servers
+> Bootstrap servers of kafka cluster, to be used for publishing commit metadata.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CALLBACK_KAFKA_PARTITION`<br></br>
+> `Config Param: BOOTSTRAP_SERVERS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -1638,34 +1766,26 @@ Controls notifications sent to Kafka, on events happening to a hudi table.
Configs that control locking mechanisms required for concurrency control between writers to a Hudi table. Concurrency between Hudi's own table services are auto managed internally.
`Config Class`: org.apache.hudi.config.HoodieLockConfig<br></br>
-> #### hoodie.write.lock.zookeeper.url
-> Zookeeper URL to connect to.<br></br>
+> #### hoodie.write.lock.zookeeper.base_path
+> The base path on Zookeeper under which to create lock related ZNodes. This should be same for all concurrent writers to the same table<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ZK_CONNECT_URL_PROP`<br></br>
-> `Since Version: 0.8.0`<br></br>
-
----
-
-> #### hoodie.write.lock.max_wait_time_ms_between_retry
-> Maximum amount of time to wait between retries by lock provider client. This bounds the maximum delay from the exponential backoff. Currently used by ZK based lock provider only.<br></br>
-> **Default Value**: 5000 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_RETRY_MAX_WAIT_TIME_IN_MILLIS_PROP`<br></br>
+> `Config Param: ZK_BASE_PATH`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.filesystem.path
-> For DFS based lock providers, path to store the locks under.<br></br>
+> #### hoodie.write.lock.zookeeper.lock_key
+> Key name under base_path at which to create a ZNode and acquire lock. Final path on zk will look like base_path/lock_key. We recommend setting this to the table name<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: FILESYSTEM_LOCK_PATH_PROP`<br></br>
+> `Config Param: ZK_LOCK_KEY`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.hivemetastore.uris
-> For Hive based lock provider, the Hive metastore URI to acquire locks against.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_METASTORE_URI_PROP`<br></br>
+> #### hoodie.write.lock.client.num_retries
+> Maximum number of times to retry to acquire lock additionally from the lock manager.<br></br>
+> **Default Value**: 0 (Optional)<br></br>
+> `Config Param: LOCK_ACQUIRE_CLIENT_NUM_RETRIES`<br></br>
> `Since Version: 0.8.0`<br></br>
---
@@ -1673,7 +1793,15 @@ Configs that control locking mechanisms required for concurrency control betwee
> #### hoodie.write.lock.wait_time_ms_between_retry
> Initial amount of time to wait between retries to acquire locks, subsequent retries will exponentially backoff.<br></br>
> **Default Value**: 5000 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP`<br></br>
+> `Config Param: LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS`<br></br>
+> `Since Version: 0.8.0`<br></br>
+
+---
+
+> #### hoodie.write.lock.num_retries
+> Maximum number of times to retry lock acquire, at each lock provider<br></br>
+> **Default Value**: 3 (Optional)<br></br>
+> `Config Param: LOCK_ACQUIRE_NUM_RETRIES`<br></br>
> `Since Version: 0.8.0`<br></br>
---
@@ -1681,103 +1809,103 @@ Configs that control locking mechanisms required for concurrency control betwee
> #### hoodie.write.lock.wait_time_ms
> Timeout in ms, to wait on an individual lock acquire() call, at the lock provider.<br></br>
> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP`<br></br>
+> `Config Param: LOCK_ACQUIRE_WAIT_TIMEOUT_MS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.conflict.resolution.strategy
-> Lock provider class name, this should be subclass of org.apache.hudi.client.transaction.ConflictResolutionStrategy<br></br>
-> **Default Value**: org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy (Optional)<br></br>
-> `Config Param: WRITE_CONFLICT_RESOLUTION_STRATEGY_CLASS_PROP`<br></br>
+> #### hoodie.write.lock.zookeeper.connection_timeout_ms
+> Timeout in ms, to wait for establishing connection with Zookeeper.<br></br>
+> **Default Value**: 15000 (Optional)<br></br>
+> `Config Param: ZK_CONNECTION_TIMEOUT_MS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.num_retries
-> Maximum number of times to retry lock acquire, at each lock provider<br></br>
-> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_NUM_RETRIES_PROP`<br></br>
+> #### hoodie.write.lock.zookeeper.port
+> Zookeeper port to connect to.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: ZK_PORT`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.connection_timeout_ms
-> Timeout in ms, to wait for establishing connection with Zookeeper.<br></br>
-> **Default Value**: 15000 (Optional)<br></br>
-> `Config Param: ZK_CONNECTION_TIMEOUT_MS_PROP`<br></br>
+> #### hoodie.write.lock.hivemetastore.table
+> For Hive based lock provider, the Hive table to acquire lock against<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_TABLE_NAME`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.client.num_retries
-> Maximum number of times to retry to acquire lock additionally from the lock manager.<br></br>
-> **Default Value**: 0 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_CLIENT_NUM_RETRIES_PROP`<br></br>
+> #### hoodie.write.lock.zookeeper.url
+> Zookeeper URL to connect to.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: ZK_CONNECT_URL`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.hivemetastore.database
-> For Hive based lock provider, the Hive database to acquire lock against<br></br>
+> #### hoodie.write.lock.filesystem.path
+> For DFS based lock providers, path to store the locks under.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_DATABASE_NAME_PROP`<br></br>
+> `Config Param: FILESYSTEM_LOCK_PATH`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.port
-> Zookeeper port to connect to.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ZK_PORT_PROP`<br></br>
+> #### hoodie.write.lock.provider
+> Lock provider class name, user can provide their own implementation of LockProvider which should be subclass of org.apache.hudi.common.lock.LockProvider<br></br>
+> **Default Value**: org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider (Optional)<br></br>
+> `Config Param: LOCK_PROVIDER_CLASS_NAME`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.client.wait_time_ms_between_retry
-> Amount of time to wait between retries on the lock provider by the lock manager<br></br>
-> **Default Value**: 10000 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_CLIENT_RETRY_WAIT_TIME_IN_MILLIS_PROP`<br></br>
+> #### hoodie.write.lock.zookeeper.session_timeout_ms
+> Timeout in ms, to wait after losing connection to ZooKeeper, before the session is expired<br></br>
+> **Default Value**: 60000 (Optional)<br></br>
+> `Config Param: ZK_SESSION_TIMEOUT_MS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.lock_key
-> Key name under base_path at which to create a ZNode and acquire lock. Final path on zk will look like base_path/lock_key. We recommend setting this to the table name<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ZK_LOCK_KEY_PROP`<br></br>
+> #### hoodie.write.lock.conflict.resolution.strategy
+> Lock provider class name, this should be subclass of org.apache.hudi.client.transaction.ConflictResolutionStrategy<br></br>
+> **Default Value**: org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy (Optional)<br></br>
+> `Config Param: WRITE_CONFLICT_RESOLUTION_STRATEGY_CLASS_NAME`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.base_path
-> The base path on Zookeeper under which to create lock related ZNodes. This should be same for all concurrent writers to the same table<br></br>
+> #### hoodie.write.lock.hivemetastore.database
+> For Hive based lock provider, the Hive database to acquire lock against<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ZK_BASE_PATH_PROP`<br></br>
+> `Config Param: HIVE_DATABASE_NAME`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.session_timeout_ms
-> Timeout in ms, to wait after losing connection to ZooKeeper, before the session is expired<br></br>
-> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: ZK_SESSION_TIMEOUT_MS_PROP`<br></br>
+> #### hoodie.write.lock.hivemetastore.uris
+> For Hive based lock provider, the Hive metastore URI to acquire locks against.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_METASTORE_URI`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.hivemetastore.table
-> For Hive based lock provider, the Hive table to acquire lock against<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_TABLE_NAME_PROP`<br></br>
+> #### hoodie.write.lock.max_wait_time_ms_between_retry
+> Maximum amount of time to wait between retries by lock provider client. This bounds the maximum delay from the exponential backoff. Currently used by ZK based lock provider only.<br></br>
+> **Default Value**: 5000 (Optional)<br></br>
+> `Config Param: LOCK_ACQUIRE_RETRY_MAX_WAIT_TIME_IN_MILLIS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.provider
-> Lock provider class name, user can provide their own implementation of LockProvider which should be subclass of org.apache.hudi.common.lock.LockProvider<br></br>
-> **Default Value**: org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider (Optional)<br></br>
-> `Config Param: LOCK_PROVIDER_CLASS_PROP`<br></br>
+> #### hoodie.write.lock.client.wait_time_ms_between_retry
+> Amount of time to wait between retries on the lock provider by the lock manager<br></br>
+> **Default Value**: 10000 (Optional)<br></br>
+> `Config Param: LOCK_ACQUIRE_CLIENT_RETRY_WAIT_TIME_IN_MILLIS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
@@ -1787,192 +1915,192 @@ Configs that control locking mechanisms required for concurrency control betwee
Configurations that control compaction (merging of log files onto a new base files) as well as cleaning (reclamation of older/unused file groups/slices).
`Config Class`: org.apache.hudi.config.HoodieCompactionConfig<br></br>
-> #### hoodie.compact.inline.trigger.strategy
-> Controls how compaction scheduling is triggered, by time or num delta commits or combination of both. Valid options: NUM_COMMITS,TIME_ELAPSED,NUM_AND_TIME,NUM_OR_TIME<br></br>
-> **Default Value**: NUM_COMMITS (Optional)<br></br>
-> `Config Param: INLINE_COMPACT_TRIGGER_STRATEGY_PROP`<br></br>
+> #### hoodie.compaction.payload.class
+> This needs to be same as class used during insert/upserts. Just like writing, compaction also uses the record payload class to merge records in the log against each other, merge again with the base file and produce the final record to be written after compaction.<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: PAYLOAD_CLASS_NAME`<br></br>
---
-> #### hoodie.cleaner.fileversions.retained
-> When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning.<br></br>
-> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: CLEANER_FILE_VERSIONS_RETAINED_PROP`<br></br>
+> #### hoodie.copyonwrite.record.size.estimate
+> The average record size. If not explicitly specified, hudi will compute the record size estimate compute dynamically based on commit metadata. This is critical in computing the insert parallelism and bin-packing inserts into small files.<br></br>
+> **Default Value**: 1024 (Optional)<br></br>
+> `Config Param: COPY_ON_WRITE_RECORD_SIZE_ESTIMATE`<br></br>
---
-> #### hoodie.cleaner.policy.failed.writes
-> Cleaning policy for failed writes to be used. Hudi will delete any files written by failed writes to re-claim space. Choose to perform this rollback of failed writes eagerly before every writer starts (only supported for single writer) or lazily by the cleaner (required for multi-writers)<br></br>
-> **Default Value**: EAGER (Optional)<br></br>
-> `Config Param: FAILED_WRITES_CLEANER_POLICY_PROP`<br></br>
+> #### hoodie.cleaner.policy
+> Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. By default, cleaner spares the file slices written by the last N commits, determined by hoodie.cleaner.commits.retained Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query execution time<br></br>
+> **Default Value**: KEEP_LATEST_COMMITS (Optional)<br></br>
+> `Config Param: CLEANER_POLICY`<br></br>
---
-> #### hoodie.parquet.small.file.limit
-> During upsert operation, we opportunistically expand existing small files on storage, instead of writing new files, to keep number of files to an optimum. This config sets the file size limit below which a file on storage becomes a candidate to be selected as such a `small file`. By default, treat any file <= 100MB as a small file.<br></br>
-> **Default Value**: 104857600 (Optional)<br></br>
-> `Config Param: PARQUET_SMALL_FILE_LIMIT_BYTES`<br></br>
+> #### hoodie.compact.inline.max.delta.seconds
+> Number of elapsed seconds after the last compaction, before scheduling a new one.<br></br>
+> **Default Value**: 3600 (Optional)<br></br>
+> `Config Param: INLINE_COMPACT_TIME_DELTA_SECONDS`<br></br>
---
-> #### hoodie.keep.max.commits
-> Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows.This config controls the maximum number of instants to retain in the active timeline. <br></br>
-> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: MAX_COMMITS_TO_KEEP_PROP`<br></br>
+> #### hoodie.cleaner.delete.bootstrap.base.file
+> When set to true, cleaner also deletes the bootstrap base file when it's skeleton base file is cleaned. Turn this to true, if you want to ensure the bootstrap dataset storage is reclaimed over time, as the table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap base files are also physically deleted, to comply with data privacy enforcement processes.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: CLEANER_BOOTSTRAP_BASE_FILE_ENABLE`<br></br>
---
-> #### hoodie.compaction.lazy.block.read
-> When merging the delta log files, this config helps to choose whether the log blocks should be read lazily or not. Choose true to use lazy block reading (low memory usage, but incurs seeks to each block header) or false for immediate block read (higher memory usage)<br></br>
+> #### hoodie.cleaner.commits.retained
+> Number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br></br>
+> **Default Value**: 10 (Optional)<br></br>
+> `Config Param: CLEANER_COMMITS_RETAINED`<br></br>
+
+---
+
+> #### hoodie.cleaner.policy.failed.writes
+> Cleaning policy for failed writes to be used. Hudi will delete any files written by failed writes to re-claim space. Choose to perform this rollback of failed writes eagerly before every writer starts (only supported for single writer) or lazily by the cleaner (required for multi-writers)<br></br>
+> **Default Value**: EAGER (Optional)<br></br>
+> `Config Param: FAILED_WRITES_CLEANER_POLICY`<br></br>
+
+---
+
+> #### hoodie.clean.async
+> Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: COMPACTION_LAZY_BLOCK_READ_ENABLED_PROP`<br></br>
+> `Config Param: ASYNC_CLEAN`<br></br>
---
> #### hoodie.clean.automatic
> When enabled, the cleaner table service is invoked immediately after each commit, to delete older file slices. It's recommended to enable this, to ensure metadata and data storage growth is bounded.<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: AUTO_CLEAN_PROP`<br></br>
+> `Config Param: AUTO_CLEAN`<br></br>
---
> #### hoodie.commits.archival.batch
> Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size.<br></br>
> **Default Value**: 10 (Optional)<br></br>
-> `Config Param: COMMITS_ARCHIVAL_BATCH_SIZE_PROP`<br></br>
+> `Config Param: COMMITS_ARCHIVAL_BATCH_SIZE`<br></br>
+
+---
+
+> #### hoodie.compaction.reverse.log.read
+> HoodieLogFormatReader reads a logfile in the forward direction starting from pos=0 to pos=file_length. If this config is set to true, the reader reads the logfile in reverse direction, from pos=file_length to pos=0<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: COMPACTION_REVERSE_LOG_READ_ENABLE`<br></br>
---
-> #### hoodie.cleaner.parallelism
-> Parallelism for the cleaning operation. Increase this if cleaning becomes slow.<br></br>
-> **Default Value**: 200 (Optional)<br></br>
-> `Config Param: CLEANER_PARALLELISM`<br></br>
+> #### hoodie.cleaner.fileversions.retained
+> When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning.<br></br>
+> **Default Value**: 3 (Optional)<br></br>
+> `Config Param: CLEANER_FILE_VERSIONS_RETAINED`<br></br>
---
-> #### hoodie.cleaner.incremental.mode
-> When enabled, the plans for each cleaner service run is computed incrementally off the events in the timeline, since the last cleaner run. This is much more efficient than obtaining listings for the full table for each planning (even with a metadata table).<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: CLEANER_INCREMENTAL_MODE`<br></br>
+> #### hoodie.compact.inline
+> When set to true, compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INLINE_COMPACT`<br></br>
---
-> #### hoodie.cleaner.delete.bootstrap.base.file
-> When set to true, cleaner also deletes the bootstrap base file when it's skeleton base file is cleaned. Turn this to true, if you want to ensure the bootstrap dataset storage is reclaimed over time, as the table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap base files are also physically deleted, to comply with data privacy enforcement processes.<br></br>
+> #### hoodie.compaction.lazy.block.read
+> When merging the delta log files, this config helps to choose whether the log blocks should be read lazily or not. Choose true to use lazy block reading (low memory usage, but incurs seeks to each block header) or false for immediate block read (higher memory usage)<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: CLEANER_BOOTSTRAP_BASE_FILE_ENABLED`<br></br>
+> `Config Param: COMPACTION_LAZY_BLOCK_READ_ENABLE`<br></br>
---
-> #### hoodie.copyonwrite.insert.split.size
-> Number of inserts assigned for each partition/bucket for writing. We based the default on writing out 100MB files, with at least 1kb records (100K records per file), and over provision to 500K. As long as auto-tuning of splits is turned on, this only affects the first write, where there is no history to learn record sizes from.<br></br>
-> **Default Value**: 500000 (Optional)<br></br>
-> `Config Param: COPY_ON_WRITE_TABLE_INSERT_SPLIT_SIZE`<br></br>
+> #### hoodie.parquet.small.file.limit
+> During upsert operation, we opportunistically expand existing small files on storage, instead of writing new files, to keep number of files to an optimum. This config sets the file size limit below which a file on storage becomes a candidate to be selected as such a `small file`. By default, treat any file <= 100MB as a small file.<br></br>
+> **Default Value**: 104857600 (Optional)<br></br>
+> `Config Param: PARQUET_SMALL_FILE_LIMIT`<br></br>
---
> #### hoodie.compaction.strategy
> Compaction strategy decides which file groups are picked up for compaction during each compaction run. By default. Hudi picks the log file with most accumulated unmerged data<br></br>
> **Default Value**: org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy (Optional)<br></br>
-> `Config Param: COMPACTION_STRATEGY_PROP`<br></br>
+> `Config Param: COMPACTION_STRATEGY`<br></br>
---
> #### hoodie.compaction.target.io
> Amount of MBs to spend during compaction run for the LogFileSizeBasedCompactionStrategy. This value helps bound ingestion latency while compaction is run inline mode.<br></br>
> **Default Value**: 512000 (Optional)<br></br>
-> `Config Param: TARGET_IO_PER_COMPACTION_IN_MB_PROP`<br></br>
-
----
-
-> #### hoodie.compaction.payload.class
-> This needs to be same as class used during insert/upserts. Just like writing, compaction also uses the record payload class to merge records in the log against each other, merge again with the base file and produce the final record to be written after compaction.<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: PAYLOAD_CLASS_PROP`<br></br>
-
----
-
-> #### hoodie.cleaner.commits.retained
-> Number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br></br>
-> **Default Value**: 10 (Optional)<br></br>
-> `Config Param: CLEANER_COMMITS_RETAINED_PROP`<br></br>
+> `Config Param: TARGET_IO_PER_COMPACTION_IN_MB`<br></br>
---
-> #### hoodie.record.size.estimation.threshold
-> We use the previous commits' metadata to calculate the estimated record size and use it to bin pack records into partitions. If the previous commit is too small to make an accurate estimation, Hudi will search commits in the reverse order, until we find a commit that has totalBytesWritten larger than (PARQUET_SMALL_FILE_LIMIT_BYTES * this_threshold)<br></br>
-> **Default Value**: 1.0 (Optional)<br></br>
-> `Config Param: RECORD_SIZE_ESTIMATION_THRESHOLD_PROP`<br></br>
+> #### hoodie.copyonwrite.insert.auto.split
+> Config to control whether we control insert split sizes automatically based on average record sizes. It's recommended to keep this turned on, since hand tuning is otherwise extremely cumbersome.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: COPY_ON_WRITE_AUTO_SPLIT_INSERTS`<br></br>
---
-> #### hoodie.compaction.daybased.target.partitions
-> Used by org.apache.hudi.io.compact.strategy.DayBasedCompactionStrategy to denote the number of latest partitions to compact during a compaction run.<br></br>
-> **Default Value**: 10 (Optional)<br></br>
-> `Config Param: TARGET_PARTITIONS_PER_DAYBASED_COMPACTION_PROP`<br></br>
+> #### hoodie.compact.inline.max.delta.commits
+> Number of delta commits after the last compaction, before scheduling of a new compaction is attempted.<br></br>
+> **Default Value**: 5 (Optional)<br></br>
+> `Config Param: INLINE_COMPACT_NUM_DELTA_COMMITS`<br></br>
---
> #### hoodie.keep.min.commits
> Similar to hoodie.keep.max.commits, but controls the minimum number ofinstants to retain in the active timeline.<br></br>
> **Default Value**: 20 (Optional)<br></br>
-> `Config Param: MIN_COMMITS_TO_KEEP_PROP`<br></br>
-
----
-
-> #### hoodie.compaction.reverse.log.read
-> HoodieLogFormatReader reads a logfile in the forward direction starting from pos=0 to pos=file_length. If this config is set to true, the reader reads the logfile in reverse direction, from pos=file_length to pos=0<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: COMPACTION_REVERSE_LOG_READ_ENABLED_PROP`<br></br>
+> `Config Param: MIN_COMMITS_TO_KEEP`<br></br>
---
-> #### hoodie.compact.inline
-> When set to true, compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INLINE_COMPACT_PROP`<br></br>
+> #### hoodie.cleaner.parallelism
+> Parallelism for the cleaning operation. Increase this if cleaning becomes slow.<br></br>
+> **Default Value**: 200 (Optional)<br></br>
+> `Config Param: CLEANER_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.clean.async
-> Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: ASYNC_CLEAN_PROP`<br></br>
+> #### hoodie.cleaner.incremental.mode
+> When enabled, the plans for each cleaner service run is computed incrementally off the events in the timeline, since the last cleaner run. This is much more efficient than obtaining listings for the full table for each planning (even with a metadata table).<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: CLEANER_INCREMENTAL_MODE_ENABLE`<br></br>
---
-> #### hoodie.copyonwrite.insert.auto.split
-> Config to control whether we control insert split sizes automatically based on average record sizes. It's recommended to keep this turned on, since hand tuning is otherwise extremely cumbersome.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: COPY_ON_WRITE_TABLE_AUTO_SPLIT_INSERTS`<br></br>
+> #### hoodie.record.size.estimation.threshold
+> We use the previous commits' metadata to calculate the estimated record size and use it to bin pack records into partitions. If the previous commit is too small to make an accurate estimation, Hudi will search commits in the reverse order, until we find a commit that has totalBytesWritten larger than (PARQUET_SMALL_FILE_LIMIT_BYTES * this_threshold)<br></br>
+> **Default Value**: 1.0 (Optional)<br></br>
+> `Config Param: RECORD_SIZE_ESTIMATION_THRESHOLD`<br></br>
---
-> #### hoodie.copyonwrite.record.size.estimate
-> The average record size. If not explicitly specified, hudi will compute the record size estimate compute dynamically based on commit metadata. This is critical in computing the insert parallelism and bin-packing inserts into small files.<br></br>
-> **Default Value**: 1024 (Optional)<br></br>
-> `Config Param: COPY_ON_WRITE_TABLE_RECORD_SIZE_ESTIMATE`<br></br>
+> #### hoodie.compact.inline.trigger.strategy
+> Controls how compaction scheduling is triggered, by time or num delta commits or combination of both. Valid options: NUM_COMMITS,TIME_ELAPSED,NUM_AND_TIME,NUM_OR_TIME<br></br>
+> **Default Value**: NUM_COMMITS (Optional)<br></br>
+> `Config Param: INLINE_COMPACT_TRIGGER_STRATEGY`<br></br>
---
-> #### hoodie.compact.inline.max.delta.commits
-> Number of delta commits after the last compaction, before scheduling of a new compaction is attempted.<br></br>
-> **Default Value**: 5 (Optional)<br></br>
-> `Config Param: INLINE_COMPACT_NUM_DELTA_COMMITS_PROP`<br></br>
+> #### hoodie.keep.max.commits
+> Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows.This config controls the maximum number of instants to retain in the active timeline. <br></br>
+> **Default Value**: 30 (Optional)<br></br>
+> `Config Param: MAX_COMMITS_TO_KEEP`<br></br>
---
-> #### hoodie.compact.inline.max.delta.seconds
-> Number of elapsed seconds after the last compaction, before scheduling a new one.<br></br>
-> **Default Value**: 3600 (Optional)<br></br>
-> `Config Param: INLINE_COMPACT_TIME_DELTA_SECONDS_PROP`<br></br>
+> #### hoodie.copyonwrite.insert.split.size
+> Number of inserts assigned for each partition/bucket for writing. We based the default on writing out 100MB files, with at least 1kb records (100K records per file), and over provision to 500K. As long as auto-tuning of splits is turned on, this only affects the first write, where there is no history to learn record sizes from.<br></br>
+> **Default Value**: 500000 (Optional)<br></br>
+> `Config Param: COPY_ON_WRITE_INSERT_SPLIT_SIZE`<br></br>
---
-> #### hoodie.cleaner.policy
-> Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. By default, cleaner spares the file slices written by the last N commits, determined by hoodie.cleaner.commits.retained Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query execution time<br></br>
-> **Default Value**: KEEP_LATEST_COMMITS (Optional)<br></br>
-> `Config Param: CLEANER_POLICY_PROP`<br></br>
+> #### hoodie.compaction.daybased.target.partitions
+> Used by org.apache.hudi.io.compact.strategy.DayBasedCompactionStrategy to denote the number of latest partitions to compact during a compaction run.<br></br>
+> **Default Value**: 10 (Optional)<br></br>
+> `Config Param: TARGET_PARTITIONS_PER_DAYBASED_COMPACTION`<br></br>
---
@@ -1984,98 +2112,98 @@ Configurations that control how file metadata is stored by Hudi, for transaction
> #### hoodie.filesystem.view.spillable.replaced.mem.fraction
> Fraction of the file system view memory, to be used for holding replace commit related metadata.<br></br>
> **Default Value**: 0.01 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_REPLACED_MEM_FRACTION`<br></br>
+> `Config Param: SPILLABLE_REPLACED_MEM_FRACTION`<br></br>
---
-> #### hoodie.filesystem.view.incr.timeline.sync.enable
-> Controls whether or not, the file system view is incrementally updated as new actions are performed on the timeline.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_INCREMENTAL_SYNC_MODE`<br></br>
+> #### hoodie.filesystem.view.spillable.dir
+> Path on local storage to use, when file system view is held in a spillable map.<br></br>
+> **Default Value**: /tmp/ (Optional)<br></br>
+> `Config Param: SPILLABLE_DIR`<br></br>
---
-> #### hoodie.filesystem.view.secondary.type
-> Specifies the secondary form of storage for file system view, if the primary (e.g timeline server) is unavailable.<br></br>
-> **Default Value**: MEMORY (Optional)<br></br>
-> `Config Param: FILESYSTEM_SECONDARY_VIEW_STORAGE_TYPE`<br></br>
+> #### hoodie.filesystem.remote.backup.view.enable
+> Config to control whether backup needs to be configured if clients were not able to reach timeline service.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: REMOTE_BACKUP_VIEW_ENABLE`<br></br>
---
> #### hoodie.filesystem.view.spillable.compaction.mem.fraction
> Fraction of the file system view memory, to be used for holding compaction related metadata.<br></br>
> **Default Value**: 0.8 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_PENDING_COMPACTION_MEM_FRACTION`<br></br>
+> `Config Param: SPILLABLE_COMPACTION_MEM_FRACTION`<br></br>
---
> #### hoodie.filesystem.view.spillable.mem
> Amount of memory to be used for holding file system view, before spilling to disk.<br></br>
> **Default Value**: 104857600 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_SPILLABLE_MEM`<br></br>
+> `Config Param: SPILLABLE_MEMORY`<br></br>
---
-> #### hoodie.filesystem.view.spillable.clustering.mem.fraction
-> Fraction of the file system view memory, to be used for holding clustering related metadata.<br></br>
-> **Default Value**: 0.01 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_PENDING_CLUSTERING_MEM_FRACTION`<br></br>
+> #### hoodie.filesystem.view.secondary.type
+> Specifies the secondary form of storage for file system view, if the primary (e.g timeline server) is unavailable.<br></br>
+> **Default Value**: MEMORY (Optional)<br></br>
+> `Config Param: SECONDARY_VIEW_TYPE`<br></br>
---
-> #### hoodie.filesystem.view.spillable.dir
-> Path on local storage to use, when file system view is held in a spillable map.<br></br>
-> **Default Value**: /tmp/view_map/ (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_SPILLABLE_DIR`<br></br>
+> #### hoodie.filesystem.view.remote.host
+> We expect this to be rarely hand configured.<br></br>
+> **Default Value**: localhost (Optional)<br></br>
+> `Config Param: REMOTE_HOST_NAME`<br></br>
---
-> #### hoodie.filesystem.view.remote.timeout.secs
-> Timeout in seconds, to wait for API requests against a remote file system view. e.g timeline server.<br></br>
-> **Default Value**: 300 (Optional)<br></br>
-> `Config Param: FILESYSTEM_REMOTE_TIMELINE_CLIENT_TIMEOUT_SECS`<br></br>
+> #### hoodie.filesystem.view.type
+> File system view provides APIs for viewing the files on the underlying lake storage, as file groups and file slices. This config controls how such a view is held. Options include MEMORY,SPILLABLE_DISK,EMBEDDED_KV_STORE,REMOTE_ONLY,REMOTE_FIRST which provide different trade offs for memory usage and API request performance.<br></br>
+> **Default Value**: MEMORY (Optional)<br></br>
+> `Config Param: VIEW_TYPE`<br></br>
---
-> #### hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction
-> Fraction of the file system view memory, to be used for holding mapping to bootstrap base files.<br></br>
-> **Default Value**: 0.05 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_BOOTSTRAP_BASE_FILE_FRACTION`<br></br>
+> #### hoodie.filesystem.view.remote.timeout.secs
+> Timeout in seconds, to wait for API requests against a remote file system view. e.g timeline server.<br></br>
+> **Default Value**: 300 (Optional)<br></br>
+> `Config Param: REMOTE_TIMEOUT_SECS`<br></br>
---
> #### hoodie.filesystem.view.remote.port
> Port to serve file system view queries, when remote. We expect this to be rarely hand configured.<br></br>
> **Default Value**: 26754 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_REMOTE_PORT`<br></br>
+> `Config Param: REMOTE_PORT_NUM`<br></br>
---
-> #### hoodie.filesystem.view.type
-> File system view provides APIs for viewing the files on the underlying lake storage, as file groups and file slices. This config controls how such a view is held. Options include MEMORY,SPILLABLE_DISK,EMBEDDED_KV_STORE,REMOTE_ONLY,REMOTE_FIRST which provide different trade offs for memory usage and API request performance.<br></br>
-> **Default Value**: MEMORY (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_STORAGE_TYPE`<br></br>
+> #### hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction
+> Fraction of the file system view memory, to be used for holding mapping to bootstrap base files.<br></br>
+> **Default Value**: 0.05 (Optional)<br></br>
+> `Config Param: BOOTSTRAP_BASE_FILE_MEM_FRACTION`<br></br>
---
-> #### hoodie.filesystem.view.remote.host
-> We expect this to be rarely hand configured.<br></br>
-> **Default Value**: localhost (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_REMOTE_HOST`<br></br>
+> #### hoodie.filesystem.view.spillable.clustering.mem.fraction
+> Fraction of the file system view memory, to be used for holding clustering related metadata.<br></br>
+> **Default Value**: 0.01 (Optional)<br></br>
+> `Config Param: SPILLABLE_CLUSTERING_MEM_FRACTION`<br></br>
---
> #### hoodie.filesystem.view.rocksdb.base.path
> Path on local storage to use, when storing file system view in embedded kv store/rocksdb.<br></br>
> **Default Value**: /tmp/hoodie_timeline_rocksdb (Optional)<br></br>
-> `Config Param: ROCKSDB_BASE_PATH_PROP`<br></br>
+> `Config Param: ROCKSDB_BASE_PATH`<br></br>
---
-> #### hoodie.filesystem.remote.backup.view.enable
-> Config to control whether backup needs to be configured if clients were not able to reach timeline service.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: REMOTE_BACKUP_VIEW_HANDLER_ENABLE`<br></br>
+> #### hoodie.filesystem.view.incr.timeline.sync.enable
+> Controls whether or not, the file system view is incrementally updated as new actions are performed on the timeline.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INCREMENTAL_TIMELINE_SYNC_ENABLE`<br></br>
---
@@ -2084,122 +2212,122 @@ Configurations that control how file metadata is stored by Hudi, for transaction
Configurations that persist across writes and read on a Hudi table like base, log file formats, table name, creation schema, table version layouts. Configurations are loaded from hoodie.properties, these properties are usually set during initializing a path as hoodie base path and rarely changes during the lifetime of the table. Writers/Queries' configurations are validated against these each time for compatibility.
`Config Class`: org.apache.hudi.common.table.HoodieTableConfig<br></br>
-> #### hoodie.table.recordkey.fields
-> Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br></br>
+> #### hoodie.bootstrap.index.enable
+> Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_RECORDKEY_FIELDS`<br></br>
+> `Config Param: BOOTSTRAP_INDEX_ENABLE`<br></br>
---
-> #### hoodie.archivelog.folder
-> path under the meta folder, to store archived timeline instants at.<br></br>
-> **Default Value**: archived (Optional)<br></br>
-> `Config Param: HOODIE_ARCHIVELOG_FOLDER_PROP`<br></br>
+> #### hoodie.table.precombine.field
+> Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: PRECOMBINE_FIELD`<br></br>
---
-> #### hoodie.table.create.schema
-> Schema used when creating the table, for the first time.<br></br>
+> #### hoodie.table.partition.fields
+> Fields used to partition the table. Concatenated values of these fields are used as the partition path, by invoking toString()<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_CREATE_SCHEMA`<br></br>
+> `Config Param: PARTITION_FIELDS`<br></br>
---
-> #### hoodie.table.type
-> The table type for the underlying data, for this write. This can’t change between writes.<br></br>
-> **Default Value**: COPY_ON_WRITE (Optional)<br></br>
-> `Config Param: HOODIE_TABLE_TYPE_PROP`<br></br>
+> #### hoodie.populate.meta.fields
+> When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: POPULATE_META_FIELDS`<br></br>
---
-> #### hoodie.table.base.file.format
-> Base file format to store all the base file data.<br></br>
-> **Default Value**: PARQUET (Optional)<br></br>
-> `Config Param: HOODIE_BASE_FILE_FORMAT_PROP`<br></br>
+> #### hoodie.compaction.payload.class
+> Payload class to use for performing compactions, i.e merge delta logs with current base file and then produce a new base file.<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: PAYLOAD_CLASS_NAME`<br></br>
---
-> #### hoodie.table.keygenerator.class
-> Key Generator class property for the hoodie table<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_KEY_GENERATOR_CLASS`<br></br>
+> #### hoodie.archivelog.folder
+> path under the meta folder, to store archived timeline instants at.<br></br>
+> **Default Value**: archived (Optional)<br></br>
+> `Config Param: ARCHIVELOG_FOLDER`<br></br>
---
-> #### hoodie.bootstrap.base.path
-> Base path of the dataset that needs to be bootstrapped as a Hudi table<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_BOOTSTRAP_BASE_PATH_PROP`<br></br>
+> #### hoodie.bootstrap.index.class
+> Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br></br>
+> **Default Value**: org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex (Optional)<br></br>
+> `Config Param: BOOTSTRAP_INDEX_CLASS_NAME`<br></br>
---
-> #### hoodie.table.name
-> Table name that will be used for registering with Hive. Needs to be same across runs.<br></br>
+> #### hoodie.table.type
+> The table type for the underlying data, for this write. This can’t change between writes.<br></br>
+> **Default Value**: COPY_ON_WRITE (Optional)<br></br>
+> `Config Param: TYPE`<br></br>
+
+---
+
+> #### hoodie.table.keygenerator.class
+> Key Generator class property for the hoodie table<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_NAME_PROP`<br></br>
+> `Config Param: KEY_GENERATOR_CLASS_NAME`<br></br>
---
> #### hoodie.table.version
> Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br></br>
> **Default Value**: ZERO (Optional)<br></br>
-> `Config Param: HOODIE_TABLE_VERSION_PROP`<br></br>
-
----
-
-> #### hoodie.table.precombine.field
-> Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_PRECOMBINE_FIELD_PROP`<br></br>
+> `Config Param: VERSION`<br></br>
---
-> #### hoodie.bootstrap.index.class
-> Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br></br>
-> **Default Value**: org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex (Optional)<br></br>
-> `Config Param: HOODIE_BOOTSTRAP_INDEX_CLASS_PROP`<br></br>
+> #### hoodie.table.base.file.format
+> Base file format to store all the base file data.<br></br>
+> **Default Value**: PARQUET (Optional)<br></br>
+> `Config Param: BASE_FILE_FORMAT`<br></br>
---
-> #### hoodie.table.log.file.format
-> Log format used for the delta logs.<br></br>
-> **Default Value**: HOODIE_LOG (Optional)<br></br>
-> `Config Param: HOODIE_LOG_FILE_FORMAT_PROP`<br></br>
+> #### hoodie.bootstrap.base.path
+> Base path of the dataset that needs to be bootstrapped as a Hudi table<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: BOOTSTRAP_BASE_PATH`<br></br>
---
-> #### hoodie.bootstrap.index.enable
-> Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined.<br></br>
+> #### hoodie.table.create.schema
+> Schema used when creating the table, for the first time.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_BOOTSTRAP_INDEX_ENABLE_PROP`<br></br>
+> `Config Param: CREATE_SCHEMA`<br></br>
---
> #### hoodie.timeline.layout.version
> Version of timeline used, by the table.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TIMELINE_LAYOUT_VERSION_PROP`<br></br>
+> `Config Param: TIMELINE_LAYOUT_VERSION`<br></br>
---
-> #### hoodie.table.partition.fields
-> Fields used to partition the table. Concatenated values of these fields are used as the partition path, by invoking toString()<br></br>
+> #### hoodie.table.name
+> Table name that will be used for registering with Hive. Needs to be same across runs.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_PARTITION_FIELDS_PROP`<br></br>
+> `Config Param: NAME`<br></br>
---
-> #### hoodie.compaction.payload.class
-> Payload class to use for performing compactions, i.e merge delta logs with current base file and then produce a new base file.<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: HOODIE_PAYLOAD_CLASS_PROP`<br></br>
+> #### hoodie.table.recordkey.fields
+> Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: RECORDKEY_FIELDS`<br></br>
---
-> #### hoodie.populate.meta.fields
-> When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: HOODIE_POPULATE_META_FIELDS`<br></br>
+> #### hoodie.table.log.file.format
+> Log format used for the delta logs.<br></br>
+> **Default Value**: HOODIE_LOG (Optional)<br></br>
+> `Config Param: LOG_FILE_FORMAT`<br></br>
---
@@ -2211,220 +2339,199 @@ Controls memory usage for compaction and merges, performed internally by Hudi.
> #### hoodie.memory.merge.fraction
> This fraction is multiplied with the user memory fraction (1 - spark.memory.fraction) to get a final fraction of heap space to use during merge<br></br>
> **Default Value**: 0.6 (Optional)<br></br>
-> `Config Param: MAX_MEMORY_FRACTION_FOR_MERGE_PROP`<br></br>
-
----
-
-> #### hoodie.memory.compaction.max.size
-> Maximum amount of memory used for compaction operations, before spilling to local storage.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: MAX_MEMORY_FOR_COMPACTION_PROP`<br></br>
-
----
-
-> #### hoodie.memory.spillable.map.path
-> Default file path prefix for spillable map<br></br>
-> **Default Value**: /tmp/ (Optional)<br></br>
-> `Config Param: SPILLABLE_MAP_BASE_PATH_PROP`<br></br>
+> `Config Param: MAX_MEMORY_FRACTION_FOR_MERGE`<br></br>
---
-> #### hoodie.memory.compaction.fraction
-> HoodieCompactedLogScanner reads logblocks, converts records to HoodieRecords and then merges these log blocks and records. At any point, the number of entries in a log block can be less than or equal to the number of entries in the corresponding parquet file. This can lead to OOM in the Scanner. Hence, a spillable map helps alleviate the memory pressure. Use this config to set the max allowable inMemory footprint of the spillable map<br></br>
-> **Default Value**: 0.6 (Optional)<br></br>
-> `Config Param: MAX_MEMORY_FRACTION_FOR_COMPACTION_PROP`<br></br>
+> #### hoodie.memory.dfs.buffer.max.size
+> Property to control the max memory for dfs input stream buffer size<br></br>
+> **Default Value**: 16777216 (Optional)<br></br>
+> `Config Param: MAX_DFS_STREAM_BUFFER_SIZE`<br></br>
---
> #### hoodie.memory.writestatus.failure.fraction
> Property to control how what fraction of the failed record, exceptions we report back to driver. Default is 10%. If set to 100%, with lot of failures, this can cause memory pressure, cause OOMs and mask actual data errors.<br></br>
> **Default Value**: 0.1 (Optional)<br></br>
-> `Config Param: WRITESTATUS_FAILURE_FRACTION_PROP`<br></br>
+> `Config Param: WRITESTATUS_FAILURE_FRACTION`<br></br>
+
+---
+
+> #### hoodie.memory.compaction.fraction
+> HoodieCompactedLogScanner reads logblocks, converts records to HoodieRecords and then merges these log blocks and records. At any point, the number of entries in a log block can be less than or equal to the number of entries in the corresponding parquet file. This can lead to OOM in the Scanner. Hence, a spillable map helps alleviate the memory pressure. Use this config to set the max allowable inMemory footprint of the spillable map<br></br>
+> **Default Value**: 0.6 (Optional)<br></br>
+> `Config Param: MAX_MEMORY_FRACTION_FOR_COMPACTION`<br></br>
---
> #### hoodie.memory.merge.max.size
> Maximum amount of memory used for merge operations, before spilling to local storage.<br></br>
> **Default Value**: 1073741824 (Optional)<br></br>
-> `Config Param: MAX_MEMORY_FOR_MERGE_PROP`<br></br>
+> `Config Param: MAX_MEMORY_FOR_MERGE`<br></br>
---
-> #### hoodie.memory.dfs.buffer.max.size
-> Property to control the max memory for dfs input stream buffer size<br></br>
-> **Default Value**: 16777216 (Optional)<br></br>
-> `Config Param: MAX_DFS_STREAM_BUFFER_SIZE_PROP`<br></br>
+> #### hoodie.memory.spillable.map.path
+> Default file path prefix for spillable map<br></br>
+> **Default Value**: /tmp/ (Optional)<br></br>
+> `Config Param: SPILLABLE_MAP_BASE_PATH`<br></br>
---
-### Index Configs {#Index-Configs}
-
-Configurations that control indexing behavior, which tags incoming records as either inserts or updates to older records.
-
-`Config Class`: org.apache.hudi.config.HoodieIndexConfig<br></br>
-> #### hoodie.index.type
-> Type of index to use. Default is Bloom filter. Possible options are [BLOOM | GLOBAL_BLOOM |SIMPLE | GLOBAL_SIMPLE | INMEMORY | HBASE]. Bloom filters removes the dependency on a external system and is stored in the footer of the Parquet Data Files<br></br>
+> #### hoodie.memory.compaction.max.size
+> Maximum amount of memory used for compaction operations, before spilling to local storage.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: INDEX_TYPE_PROP`<br></br>
+> `Config Param: MAX_MEMORY_FOR_COMPACTION`<br></br>
---
-> #### hoodie.bloom.index.use.treebased.filter
-> Only applies if index type is BLOOM. When true, interval tree based file pruning optimization is enabled. This mode speeds-up file-pruning based on key ranges when compared with the brute-force mode<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_TREE_BASED_FILTER_PROP`<br></br>
+### Index Configs {#Index-Configs}
----
+Configurations that control indexing behavior, which tags incoming records as either inserts or updates to older records.
+`Config Class`: org.apache.hudi.config.HoodieIndexConfig<br></br>
> #### hoodie.index.bloom.num_entries
> Only applies if index type is BLOOM. This is the number of entries to be stored in the bloom filter. The rationale for the default: Assume the maxParquetFileSize is 128MB and averageRecordSize is 1kb and hence we approx a total of 130K records in a file. The default (60000) is roughly half of this approximation. Warning: Setting this very low, will generate a lot of false positives and index lookup will have to scan a lot more files than it has to and setting this to a very high number [...]
> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: BLOOM_FILTER_NUM_ENTRIES`<br></br>
+> `Config Param: BLOOM_FILTER_NUM_ENTRIES_VALUE`<br></br>
---
-> #### hoodie.bloom.index.bucketized.checking
-> Only applies if index type is BLOOM. When true, bucketized bloom filtering is enabled. This reduces skew seen in sort based bloom index lookup<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_BUCKETIZED_CHECKING_PROP`<br></br>
+> #### hoodie.bloom.index.keys.per.bucket
+> Only applies if bloomIndexBucketizedChecking is enabled and index type is bloom. This configuration controls the “bucket” size which tracks the number of record-key checks made against a single file and is the unit of work allocated to each partition performing bloom filter lookup. A higher value would amortize the fixed cost of reading a bloom filter to memory.<br></br>
+> **Default Value**: 10000000 (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_KEYS_PER_BUCKET`<br></br>
---
-> #### hoodie.bloom.index.update.partition.path
-> Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_UPDATE_PARTITION_PATH`<br></br>
+> #### hoodie.simple.index.input.storage.level
+> Only applies when #simpleIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br></br>
+> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
+> `Config Param: SIMPLE_INDEX_INPUT_STORAGE_LEVEL_VALUE`<br></br>
---
-> #### hoodie.bloom.index.parallelism
-> Only applies if index type is BLOOM. This is the amount of parallelism for index lookup, which involves a shuffle. By default, this is auto computed based on input workload characteristics.<br></br>
-> **Default Value**: 0 (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_PARALLELISM_PROP`<br></br>
+> #### hoodie.simple.index.parallelism
+> Only applies if index type is SIMPLE. This is the amount of parallelism for index lookup, which involves a Spark Shuffle<br></br>
+> **Default Value**: 50 (Optional)<br></br>
+> `Config Param: SIMPLE_INDEX_PARALLELISM`<br></br>
---
-> #### hoodie.bloom.index.input.storage.level
-> Only applies when #bloomIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br></br>
-> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_INPUT_STORAGE_LEVEL`<br></br>
+> #### hoodie.global.simple.index.parallelism
+> Only applies if index type is GLOBAL_SIMPLE. This is the amount of parallelism for index lookup, which involves a Spark Shuffle<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: GLOBAL_SIMPLE_INDEX_PARALLELISM`<br></br>
---
-> #### hoodie.bloom.index.keys.per.bucket
-> Only applies if bloomIndexBucketizedChecking is enabled and index type is bloom. This configuration controls the “bucket” size which tracks the number of record-key checks made against a single file and is the unit of work allocated to each partition performing bloom filter lookup. A higher value would amortize the fixed cost of reading a bloom filter to memory.<br></br>
-> **Default Value**: 10000000 (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_KEYS_PER_BUCKET_PROP`<br></br>
+> #### hoodie.simple.index.update.partition.path
+> Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: true description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition since version: version is not defined deprecated after: version i [...]
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: SIMPLE_INDEX_UPDATE_PARTITION_PATH_ENABLE`<br></br>
---
-> #### hoodie.simple.index.update.partition.path
-> Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: false description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition since version: version is not defined deprecated after: version [...]
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: SIMPLE_INDEX_UPDATE_PARTITION_PATH`<br></br>
+> #### hoodie.bloom.index.bucketized.checking
+> Only applies if index type is BLOOM. When true, bucketized bloom filtering is enabled. This reduces skew seen in sort based bloom index lookup<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_BUCKETIZED_CHECKING`<br></br>
---
-> #### hoodie.simple.index.input.storage.level
-> Only applies when #simpleIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br></br>
-> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
-> `Config Param: SIMPLE_INDEX_INPUT_STORAGE_LEVEL`<br></br>
+> #### hoodie.index.type
+> Type of index to use. Default is Bloom filter. Possible options are [BLOOM | GLOBAL_BLOOM |SIMPLE | GLOBAL_SIMPLE | INMEMORY | HBASE]. Bloom filters removes the dependency on a external system and is stored in the footer of the Parquet Data Files<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: INDEX_TYPE`<br></br>
---
> #### hoodie.index.bloom.fpp
> Only applies if index type is BLOOM. Error rate allowed given the number of entries. This is used to calculate how many bits should be assigned for the bloom filter and the number of hash functions. This is usually set very low (default: 0.000000001), we like to tradeoff disk space for lower false positives. If the number of entries added to bloom filter exceeds the configured value (hoodie.index.bloom.num_entries), then this fpp may not be honored.<br></br>
> **Default Value**: 0.000000001 (Optional)<br></br>
-> `Config Param: BLOOM_FILTER_FPP`<br></br>
+> `Config Param: BLOOM_FILTER_FPP_VALUE`<br></br>
---
-> #### hoodie.bloom.index.filter.dynamic.max.entries
-> The threshold for the maximum number of keys to record in a dynamic Bloom filter row. Only applies if filter type is BloomFilterTypeCode.DYNAMIC_V0.<br></br>
-> **Default Value**: 100000 (Optional)<br></br>
-> `Config Param: HOODIE_BLOOM_INDEX_FILTER_DYNAMIC_MAX_ENTRIES`<br></br>
+> #### hoodie.bloom.index.update.partition.path
+> Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_UPDATE_PARTITION_PATH_ENABLE`<br></br>
---
-> #### hoodie.index.class
-> Full path of user-defined index class and must be a subclass of HoodieIndex class. It will take precedence over the hoodie.index.type configuration if specified<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: INDEX_CLASS_PROP`<br></br>
+> #### hoodie.bloom.index.use.caching
+> Only applies if index type is BLOOM.When true, the input RDD will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_USE_CACHING`<br></br>
---
-> #### hoodie.bloom.index.filter.type
-> Filter type used. Default is BloomFilterTypeCode.SIMPLE. Available values are [BloomFilterTypeCode.SIMPLE , BloomFilterTypeCode.DYNAMIC_V0]. Dynamic bloom filters auto size themselves based on number of keys.<br></br>
-> **Default Value**: SIMPLE (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_FILTER_TYPE`<br></br>
+> #### hoodie.bloom.index.input.storage.level
+> Only applies when #bloomIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br></br>
+> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_INPUT_STORAGE_LEVEL_VALUE`<br></br>
---
-> #### hoodie.global.simple.index.parallelism
-> Only applies if index type is GLOBAL_SIMPLE. This is the amount of parallelism for index lookup, which involves a Spark Shuffle<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: GLOBAL_SIMPLE_INDEX_PARALLELISM_PROP`<br></br>
+> #### hoodie.bloom.index.use.treebased.filter
+> Only applies if index type is BLOOM. When true, interval tree based file pruning optimization is enabled. This mode speeds-up file-pruning based on key ranges when compared with the brute-force mode<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_TREE_BASED_FILTER`<br></br>
---
-> #### hoodie.bloom.index.use.caching
-> Only applies if index type is BLOOM.When true, the input RDD will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_USE_CACHING_PROP`<br></br>
+> #### hoodie.bloom.index.parallelism
+> Only applies if index type is BLOOM. This is the amount of parallelism for index lookup, which involves a shuffle. By default, this is auto computed based on input workload characteristics.<br></br>
+> **Default Value**: 0 (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_PARALLELISM`<br></br>
---
-> #### hoodie.bloom.index.prune.by.ranges
-> Only applies if index type is BLOOM. When true, range information from files to leveraged speed up index lookups. Particularly helpful, if the key has a monotonously increasing prefix, such as timestamp. If the record key is completely random, it is better to turn this off, since range pruning will only add extra overhead to the index lookup.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_PRUNE_BY_RANGES_PROP`<br></br>
+> #### hoodie.bloom.index.filter.dynamic.max.entries
+> The threshold for the maximum number of keys to record in a dynamic Bloom filter row. Only applies if filter type is BloomFilterTypeCode.DYNAMIC_V0.<br></br>
+> **Default Value**: 100000 (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_FILTER_DYNAMIC_MAX_ENTRIES`<br></br>
---
> #### hoodie.simple.index.use.caching
> Only applies if index type is SIMPLE. When true, the incoming writes will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: SIMPLE_INDEX_USE_CACHING_PROP`<br></br>
+> `Config Param: SIMPLE_INDEX_USE_CACHING`<br></br>
---
-> #### hoodie.simple.index.parallelism
-> Only applies if index type is SIMPLE. This is the amount of parallelism for index lookup, which involves a Spark Shuffle<br></br>
-> **Default Value**: 50 (Optional)<br></br>
-> `Config Param: SIMPLE_INDEX_PARALLELISM_PROP`<br></br>
+> #### hoodie.bloom.index.prune.by.ranges
+> Only applies if index type is BLOOM. When true, range information from files to leveraged speed up index lookups. Particularly helpful, if the key has a monotonously increasing prefix, such as timestamp. If the record key is completely random, it is better to turn this off, since range pruning will only add extra overhead to the index lookup.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_PRUNE_BY_RANGES`<br></br>
---
-### Storage Configs {#Storage-Configs}
+> #### hoodie.bloom.index.filter.type
+> Filter type used. Default is BloomFilterTypeCode.DYNAMIC_V0. Available values are [BloomFilterTypeCode.SIMPLE , BloomFilterTypeCode.DYNAMIC_V0]. Dynamic bloom filters auto size themselves based on number of keys.<br></br>
+> **Default Value**: DYNAMIC_V0 (Optional)<br></br>
+> `Config Param: BLOOM_FILTER_TYPE`<br></br>
-Configurations that control aspects around writing, sizing, reading base and log files.
+---
-`Config Class`: org.apache.hudi.config.HoodieStorageConfig<br></br>
-> #### hoodie.parquet.compression.ratio
-> Expected compression of parquet data used by Hudi, when it tries to size new parquet files. Increase this value, if bulk_insert is producing smaller than expected sized files<br></br>
-> **Default Value**: 0.1 (Optional)<br></br>
-> `Config Param: PARQUET_COMPRESSION_RATIO`<br></br>
+> #### hoodie.index.class
+> Full path of user-defined index class and must be a subclass of HoodieIndex class. It will take precedence over the hoodie.index.type configuration if specified<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: INDEX_CLASS_NAME`<br></br>
---
-> #### hoodie.hfile.max.file.size
-> Target file size for HFile base files.<br></br>
-> **Default Value**: 125829120 (Optional)<br></br>
-> `Config Param: HFILE_FILE_MAX_BYTES`<br></br>
+### Storage Configs {#Storage-Configs}
----
+Configurations that control aspects around writing, sizing, reading base and log files.
+`Config Class`: org.apache.hudi.config.HoodieStorageConfig<br></br>
> #### hoodie.logfile.data.block.max.size
> LogFile Data block max size. This is the maximum size allowed for a single data block to be appended to a log file. This helps to make sure the data appended to the log file is broken up into sizable blocks to prevent from OOM errors. This size should be greater than the JVM memory.<br></br>
> **Default Value**: 268435456 (Optional)<br></br>
-> `Config Param: LOGFILE_DATA_BLOCK_SIZE_MAX_BYTES`<br></br>
-
----
-
-> #### hoodie.parquet.block.size
-> Parquet RowGroup size. It's recommended to make this large enough that scan costs can be amortized by packing enough column values into a single row group.<br></br>
-> **Default Value**: 125829120 (Optional)<br></br>
-> `Config Param: PARQUET_BLOCK_SIZE_BYTES`<br></br>
+> `Config Param: LOGFILE_DATA_BLOCK_MAX_SIZE`<br></br>
---
@@ -2442,66 +2549,87 @@ Configurations that control aspects around writing, sizing, reading base and log
---
-> #### hoodie.orc.max.file.size
-> Target file size for ORC base files.<br></br>
-> **Default Value**: 125829120 (Optional)<br></br>
-> `Config Param: ORC_FILE_MAX_BYTES`<br></br>
+> #### hoodie.orc.compression.codec
+> Compression codec to use for ORC base files.<br></br>
+> **Default Value**: ZLIB (Optional)<br></br>
+> `Config Param: ORC_COMPRESSION_CODEC_NAME`<br></br>
---
-> #### hoodie.hfile.compression.algorithm
-> Compression codec to use for hfile base files.<br></br>
-> **Default Value**: GZ (Optional)<br></br>
-> `Config Param: HFILE_COMPRESSION_ALGORITHM`<br></br>
+> #### hoodie.parquet.max.file.size
+> Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br></br>
+> **Default Value**: 125829120 (Optional)<br></br>
+> `Config Param: PARQUET_MAX_FILE_SIZE`<br></br>
---
-> #### hoodie.parquet.max.file.size
-> Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br></br>
+> #### hoodie.hfile.max.file.size
+> Target file size for HFile base files.<br></br>
> **Default Value**: 125829120 (Optional)<br></br>
-> `Config Param: PARQUET_FILE_MAX_BYTES`<br></br>
+> `Config Param: HFILE_MAX_FILE_SIZE`<br></br>
---
-> #### hoodie.orc.compression.codec
-> Compression codec to use for ORC base files.<br></br>
-> **Default Value**: ZLIB (Optional)<br></br>
-> `Config Param: ORC_COMPRESSION_CODEC`<br></br>
+> #### hoodie.parquet.block.size
+> Parquet RowGroup size. It's recommended to make this large enough that scan costs can be amortized by packing enough column values into a single row group.<br></br>
+> **Default Value**: 125829120 (Optional)<br></br>
+> `Config Param: PARQUET_BLOCK_SIZE`<br></br>
---
> #### hoodie.logfile.max.size
> LogFile max size. This is the maximum size allowed for a log file before it is rolled over to the next version.<br></br>
> **Default Value**: 1073741824 (Optional)<br></br>
-> `Config Param: LOGFILE_SIZE_MAX_BYTES`<br></br>
+> `Config Param: LOGFILE_MAX_SIZE`<br></br>
---
-> #### hoodie.parquet.compression.codec
-> Compression Codec for parquet files<br></br>
-> **Default Value**: gzip (Optional)<br></br>
-> `Config Param: PARQUET_COMPRESSION_CODEC`<br></br>
+> #### hoodie.hfile.block.size
+> Lower values increase the size of metadata tracked within HFile, but can offer potentially faster lookup times.<br></br>
+> **Default Value**: 1048576 (Optional)<br></br>
+> `Config Param: HFILE_BLOCK_SIZE`<br></br>
+
+---
+
+> #### hoodie.parquet.page.size
+> Parquet page size. Page is the unit of read within a parquet file. Within a block, pages are compressed separately.<br></br>
+> **Default Value**: 1048576 (Optional)<br></br>
+> `Config Param: PARQUET_PAGE_SIZE`<br></br>
+
+---
+
+> #### hoodie.hfile.compression.algorithm
+> Compression codec to use for hfile base files.<br></br>
+> **Default Value**: GZ (Optional)<br></br>
+> `Config Param: HFILE_COMPRESSION_ALGORITHM_NAME`<br></br>
+
+---
+
+> #### hoodie.orc.max.file.size
+> Target file size for ORC base files.<br></br>
+> **Default Value**: 125829120 (Optional)<br></br>
+> `Config Param: ORC_FILE_MAX_SIZE`<br></br>
---
> #### hoodie.logfile.to.parquet.compression.ratio
> Expected additional compression as records move from log files to parquet. Used for merge_on_read table to send inserts into log files & control the size of compacted parquet file.<br></br>
> **Default Value**: 0.35 (Optional)<br></br>
-> `Config Param: LOGFILE_TO_PARQUET_COMPRESSION_RATIO`<br></br>
+> `Config Param: LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION`<br></br>
---
-> #### hoodie.parquet.page.size
-> Parquet page size. Page is the unit of read within a parquet file. Within a block, pages are compressed separately.<br></br>
-> **Default Value**: 1048576 (Optional)<br></br>
-> `Config Param: PARQUET_PAGE_SIZE_BYTES`<br></br>
+> #### hoodie.parquet.compression.ratio
+> Expected compression of parquet data used by Hudi, when it tries to size new parquet files. Increase this value, if bulk_insert is producing smaller than expected sized files<br></br>
+> **Default Value**: 0.1 (Optional)<br></br>
+> `Config Param: PARQUET_COMPRESSION_RATIO_FRACTION`<br></br>
---
-> #### hoodie.hfile.block.size
-> Lower values increase the size of metadata tracked within HFile, but can offer potentially faster lookup times.<br></br>
-> **Default Value**: 1048576 (Optional)<br></br>
-> `Config Param: HFILE_BLOCK_SIZE_BYTES`<br></br>
+> #### hoodie.parquet.compression.codec
+> Compression Codec for parquet files<br></br>
+> **Default Value**: gzip (Optional)<br></br>
+> `Config Param: PARQUET_COMPRESSION_CODEC_NAME`<br></br>
---
@@ -2510,10 +2638,18 @@ Configurations that control aspects around writing, sizing, reading base and log
Configurations that control the clustering table service in hudi, which optimizes the storage layout for better query performance by sorting and sizing data files.
`Config Class`: org.apache.hudi.config.HoodieClusteringConfig<br></br>
-> #### hoodie.clustering.updates.strategy
-> Determines how to handle updates, deletes to file groups that are under clustering. Default strategy just rejects the update<br></br>
-> **Default Value**: org.apache.hudi.client.clustering.update.strategy.SparkRejectUpdateStrategy (Optional)<br></br>
-> `Config Param: CLUSTERING_UPDATES_STRATEGY_PROP`<br></br>
+> #### hoodie.clustering.preserve.commit.metadata
+> When rewriting data, preserves existing hoodie_commit_time<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: PRESERVE_COMMIT_METADATA`<br></br>
+> `Since Version: 0.9.0`<br></br>
+
+---
+
+> #### hoodie.clustering.plan.strategy.max.num.groups
+> Maximum number of groups to create as part of ClusteringPlan. Increasing groups will increase parallelism<br></br>
+> **Default Value**: 30 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_MAX_GROUPS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2521,95 +2657,103 @@ Configurations that control the clustering table service in hudi, which optimize
> #### hoodie.clustering.async.max.commits
> Config to control frequency of async clustering<br></br>
> **Default Value**: 4 (Optional)<br></br>
-> `Config Param: ASYNC_CLUSTERING_MAX_COMMIT_PROP`<br></br>
+> `Config Param: ASYNC_CLUSTERING_MAX_COMMITS`<br></br>
> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.target.file.max.bytes
-> Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups<br></br>
-> **Default Value**: 1073741824 (Optional)<br></br>
-> `Config Param: CLUSTERING_TARGET_FILE_MAX_BYTES`<br></br>
+> #### hoodie.clustering.inline.max.commits
+> Config to control frequency of clustering planning<br></br>
+> **Default Value**: 4 (Optional)<br></br>
+> `Config Param: INLINE_CLUSTERING_MAX_COMMITS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.daybased.lookback.partitions
-> Number of partitions to list to create ClusteringPlan<br></br>
-> **Default Value**: 2 (Optional)<br></br>
-> `Config Param: CLUSTERING_TARGET_PARTITIONS`<br></br>
+> #### hoodie.clustering.plan.strategy.target.file.max.bytes
+> Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups<br></br>
+> **Default Value**: 1073741824 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_TARGET_FILE_MAX_BYTES`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.max.bytes.per.group
-> Each clustering operation can create multiple output file groups. Total amount of data processed by clustering operation is defined by below two properties (CLUSTERING_MAX_BYTES_PER_GROUP * CLUSTERING_MAX_NUM_GROUPS). Max amount of data to be included in one group<br></br>
-> **Default Value**: 2147483648 (Optional)<br></br>
-> `Config Param: CLUSTERING_MAX_BYTES_PER_GROUP`<br></br>
-> `Since Version: 0.7.0`<br></br>
+> #### hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions
+> Number of partitions to skip from latest when choosing partitions to create ClusteringPlan<br></br>
+> **Default Value**: 0 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
> #### hoodie.clustering.execution.strategy.class
> Config to provide a strategy class (subclass of RunClusteringStrategy) to define how the clustering plan is executed. By default, we sort the file groups in th plan by the specified columns, while meeting the configured target file sizes.<br></br>
> **Default Value**: org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy (Optional)<br></br>
-> `Config Param: CLUSTERING_EXECUTION_STRATEGY_CLASS`<br></br>
+> `Config Param: EXECUTION_STRATEGY_CLASS_NAME`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.max.num.groups
-> Maximum number of groups to create as part of ClusteringPlan. Increasing groups will increase parallelism<br></br>
-> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: CLUSTERING_MAX_NUM_GROUPS`<br></br>
+> #### hoodie.clustering.plan.strategy.max.bytes.per.group
+> Each clustering operation can create multiple output file groups. Total amount of data processed by clustering operation is defined by below two properties (CLUSTERING_MAX_BYTES_PER_GROUP * CLUSTERING_MAX_NUM_GROUPS). Max amount of data to be included in one group<br></br>
+> **Default Value**: 2147483648 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.inline
-> Turn on inline clustering - clustering will be run after each write operation is complete<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INLINE_CLUSTERING_PROP`<br></br>
+> #### hoodie.clustering.plan.strategy.small.file.limit
+> Files smaller than the size specified here are candidates for clustering<br></br>
+> **Default Value**: 629145600 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_SMALL_FILE_LIMIT`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.sort.columns
-> Columns to sort the data by when clustering<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CLUSTERING_SORT_COLUMNS_PROPERTY`<br></br>
+> #### hoodie.clustering.async.enabled
+> Enable running of clustering service, asynchronously as inserts happen on the table.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ASYNC_CLUSTERING_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.inline.max.commits
-> Config to control frequency of clustering planning<br></br>
-> **Default Value**: 4 (Optional)<br></br>
-> `Config Param: INLINE_CLUSTERING_MAX_COMMIT_PROP`<br></br>
+> #### hoodie.clustering.plan.strategy.class
+> Config to provide a strategy class (subclass of ClusteringPlanStrategy) to create clustering plan i.e select what file groups are being clustered. Default strategy, looks at the last N (determined by hoodie.clustering.plan.strategy.daybased.lookback.partitions) day based partitions picks the small file slices within those partitions.<br></br>
+> **Default Value**: org.apache.hudi.client.clustering.plan.strategy.SparkRecentDaysClusteringPlanStrategy (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_CLASS_NAME`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.small.file.limit
-> Files smaller than the size specified here are candidates for clustering<br></br>
-> **Default Value**: 629145600 (Optional)<br></br>
-> `Config Param: CLUSTERING_PLAN_SMALL_FILE_LIMIT`<br></br>
+> #### hoodie.clustering.updates.strategy
+> Determines how to handle updates, deletes to file groups that are under clustering. Default strategy just rejects the update<br></br>
+> **Default Value**: org.apache.hudi.client.clustering.update.strategy.SparkRejectUpdateStrategy (Optional)<br></br>
+> `Config Param: UPDATES_STRATEGY`<br></br>
+> `Since Version: 0.7.0`<br></br>
+
+---
+
+> #### hoodie.clustering.inline
+> Turn on inline clustering - clustering will be run after each write operation is complete<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INLINE_CLUSTERING`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.async.enabled
-> Enable running of clustering service, asynchronously as inserts happen on the table.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: ASYNC_CLUSTERING_ENABLE_OPT_KEY`<br></br>
+> #### hoodie.clustering.plan.strategy.sort.columns
+> Columns to sort the data by when clustering<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: PLAN_STRATEGY_SORT_COLUMNS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.class
-> Config to provide a strategy class (subclass of ClusteringPlanStrategy) to create clustering plan i.e select what file groups are being clustered. Default strategy, looks at the last N (determined by hoodie.clustering.plan.strategy.daybased.lookback.partitions) day based partitions picks the small file slices within those partitions.<br></br>
-> **Default Value**: org.apache.hudi.client.clustering.plan.strategy.SparkRecentDaysClusteringPlanStrategy (Optional)<br></br>
-> `Config Param: CLUSTERING_PLAN_STRATEGY_CLASS`<br></br>
+> #### hoodie.clustering.plan.strategy.daybased.lookback.partitions
+> Number of partitions to list to create ClusteringPlan<br></br>
+> **Default Value**: 2 (Optional)<br></br>
+> `Config Param: DAYBASED_LOOKBACK_PARTITIONS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2619,26 +2763,34 @@ Configurations that control the clustering table service in hudi, which optimize
Configurations used by the Hudi Metadata Table. This table maintains the metadata about a given Hudi table (e.g file listings) to avoid overhead of accessing cloud storage, during queries.
`Config Class`: org.apache.hudi.common.config.HoodieMetadataConfig<br></br>
-> #### hoodie.metadata.enable
-> Enable the internal metadata table which serves table metadata like level file listings<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: METADATA_ENABLE_PROP`<br></br>
+> #### hoodie.metadata.compact.max.delta.commits
+> Controls how often the metadata table is compacted.<br></br>
+> **Default Value**: 24 (Optional)<br></br>
+> `Config Param: COMPACT_NUM_DELTA_COMMITS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.insert.parallelism
-> Parallelism to use when inserting to the metadata table<br></br>
-> **Default Value**: 1 (Optional)<br></br>
-> `Config Param: METADATA_INSERT_PARALLELISM_PROP`<br></br>
+> #### hoodie.assume.date.partitioning
+> Should HoodieWriteClient assume the data is partitioned by dates, i.e three levels from base path. This is a stop-gap to support tables created by versions < 0.3.1. Will be removed eventually<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ASSUME_DATE_PARTITIONING`<br></br>
+> `Since Version: 0.3.0`<br></br>
+
+---
+
+> #### hoodie.metadata.validate
+> Validate contents of metadata table on each access; e.g against the actual listings from lake storage<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: VALIDATE_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.compact.max.delta.commits
-> Controls how often the metadata table is compacted.<br></br>
-> **Default Value**: 24 (Optional)<br></br>
-> `Config Param: METADATA_COMPACT_NUM_DELTA_COMMITS_PROP`<br></br>
+> #### hoodie.metadata.metrics.enable
+> Enable publishing of metrics around metadata table.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: METRICS_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2646,71 +2798,71 @@ Configurations used by the Hudi Metadata Table. This table maintains the metadat
> #### hoodie.metadata.cleaner.commits.retained
> Controls retention/history for metadata table.<br></br>
> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: CLEANER_COMMITS_RETAINED_PROP`<br></br>
+> `Config Param: CLEANER_COMMITS_RETAINED`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.keep.min.commits
-> Controls the archival of the metadata table’s timeline.<br></br>
-> **Default Value**: 20 (Optional)<br></br>
-> `Config Param: MIN_COMMITS_TO_KEEP_PROP`<br></br>
+> #### hoodie.file.listing.parallelism
+> Parallelism to use, when listing the table on lake storage.<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: FILE_LISTING_PARALLELISM_VALUE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.metrics.enable
-> Enable publishing of metrics around metadata table.<br></br>
+> #### hoodie.metadata.enable
+> Enable the internal metadata table which serves table metadata like level file listings<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: METADATA_METRICS_ENABLE_PROP`<br></br>
+> `Config Param: ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.assume.date.partitioning
-> Should HoodieWriteClient assume the data is partitioned by dates, i.e three levels from base path. This is a stop-gap to support tables created by versions < 0.3.1. Will be removed eventually<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HOODIE_ASSUME_DATE_PARTITIONING_PROP`<br></br>
-> `Since Version: 0.3.0`<br></br>
+> #### hoodie.metadata.sync.enable
+> Enable syncing of metadata table from actions on the dataset<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: SYNC_ENABLE`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.metadata.keep.max.commits
-> Controls the archival of the metadata table’s timeline.<br></br>
-> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: MAX_COMMITS_TO_KEEP_PROP`<br></br>
+> #### hoodie.metadata.clean.async
+> Enable asynchronous cleaning for metadata table<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ASYNC_CLEAN_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.dir.filter.regex
-> Directories matching this regex, will be filtered out when initializing metadata table from lake storage for the first time.<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: DIRECTORY_FILTER_REGEX`<br></br>
+> #### hoodie.metadata.keep.max.commits
+> Controls the archival of the metadata table’s timeline.<br></br>
+> **Default Value**: 30 (Optional)<br></br>
+> `Config Param: MAX_COMMITS_TO_KEEP`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.validate
-> Validate contents of metadata table on each access; e.g against the actual listings from lake storage<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: METADATA_VALIDATE_PROP`<br></br>
+> #### hoodie.metadata.insert.parallelism
+> Parallelism to use when inserting to the metadata table<br></br>
+> **Default Value**: 1 (Optional)<br></br>
+> `Config Param: INSERT_PARALLELISM_VALUE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.clean.async
-> Enable asynchronous cleaning for metadata table<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: METADATA_ASYNC_CLEAN_PROP`<br></br>
+> #### hoodie.metadata.dir.filter.regex
+> Directories matching this regex, will be filtered out when initializing metadata table from lake storage for the first time.<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: DIR_FILTER_REGEX`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.file.listing.parallelism
-> Parallelism to use, when listing the table on lake storage.<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: FILE_LISTING_PARALLELISM_PROP`<br></br>
+> #### hoodie.metadata.keep.min.commits
+> Controls the archival of the metadata table’s timeline.<br></br>
+> **Default Value**: 20 (Optional)<br></br>
+> `Config Param: MIN_COMMITS_TO_KEEP`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2723,23 +2875,15 @@ Configurations that control how you want to bootstrap your existing tables for t
> #### hoodie.bootstrap.partitionpath.translator.class
> Translates the partition paths from the bootstrapped data into how is laid out as a Hudi table.<br></br>
> **Default Value**: org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPartitionPathTranslator (Optional)<br></br>
-> `Config Param: BOOTSTRAP_PARTITION_PATH_TRANSLATOR_CLASS`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
-> #### hoodie.bootstrap.keygen.class
-> Key generator implementation to be used for generating keys from the bootstrapped dataset<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BOOTSTRAP_KEYGEN_CLASS`<br></br>
+> `Config Param: PARTITION_PATH_TRANSLATOR_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.mode.selector
-> Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped<br></br>
-> **Default Value**: org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector (Optional)<br></br>
-> `Config Param: BOOTSTRAP_MODE_SELECTOR`<br></br>
+> #### hoodie.bootstrap.full.input.provider
+> Class to use for reading the bootstrap dataset partitions/files, for Bootstrap mode FULL_RECORD<br></br>
+> **Default Value**: org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider (Optional)<br></br>
+> `Config Param: FULL_BOOTSTRAP_INPUT_PROVIDER_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2747,55 +2891,63 @@ Configurations that control how you want to bootstrap your existing tables for t
> #### hoodie.bootstrap.keygen.type
> Type of build-in key generator, currently support SIMPLE, COMPLEX, TIMESTAMP, CUSTOM, NON_PARTITION, GLOBAL_DELETE<br></br>
> **Default Value**: SIMPLE (Optional)<br></br>
-> `Config Param: BOOTSTRAP_KEYGEN_TYPE`<br></br>
+> `Config Param: KEYGEN_TYPE`<br></br>
> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.bootstrap.mode.selector.regex.mode
-> Bootstrap mode to apply for partition paths, that match regex above. METADATA_ONLY will generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will perform a full copy/rewrite of the data as a Hudi table.<br></br>
-> **Default Value**: METADATA_ONLY (Optional)<br></br>
-> `Config Param: BOOTSTRAP_MODE_SELECTOR_REGEX_MODE`<br></br>
+> #### hoodie.bootstrap.keygen.class
+> Key generator implementation to be used for generating keys from the bootstrapped dataset<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: KEYGEN_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.index.class
-> Implementation to use, for mapping a skeleton base file to a boostrap base file.<br></br>
-> **Default Value**: org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex (Optional)<br></br>
-> `Config Param: BOOTSTRAP_INDEX_CLASS_PROP`<br></br>
+> #### hoodie.bootstrap.parallelism
+> Parallelism value to be used to bootstrap data into hudi<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: PARALLELISM_VALUE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.full.input.provider
-> Class to use for reading the bootstrap dataset partitions/files, for Bootstrap mode FULL_RECORD<br></br>
-> **Default Value**: org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider (Optional)<br></br>
-> `Config Param: FULL_BOOTSTRAP_INPUT_PROVIDER`<br></br>
+> #### hoodie.bootstrap.base.path
+> Base path of the dataset that needs to be bootstrapped as a Hudi table<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: BASE_PATH`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.parallelism
-> Parallelism value to be used to bootstrap data into hudi<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: BOOTSTRAP_PARALLELISM`<br></br>
+> #### hoodie.bootstrap.mode.selector.regex
+> Matches each bootstrap dataset partition against this regex and applies the mode below to it.<br></br>
+> **Default Value**: .* (Optional)<br></br>
+> `Config Param: PARTITION_SELECTOR_REGEX_PATTERN`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.mode.selector.regex
-> Matches each bootstrap dataset partition against this regex and applies the mode below to it.<br></br>
-> **Default Value**: .* (Optional)<br></br>
-> `Config Param: BOOTSTRAP_MODE_SELECTOR_REGEX`<br></br>
+> #### hoodie.bootstrap.index.class
+> Implementation to use, for mapping a skeleton base file to a boostrap base file.<br></br>
+> **Default Value**: org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex (Optional)<br></br>
+> `Config Param: INDEX_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.base.path
-> Base path of the dataset that needs to be bootstrapped as a Hudi table<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BOOTSTRAP_BASE_PATH_PROP`<br></br>
+> #### hoodie.bootstrap.mode.selector.regex.mode
+> Bootstrap mode to apply for partition paths, that match regex above. METADATA_ONLY will generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will perform a full copy/rewrite of the data as a Hudi table.<br></br>
+> **Default Value**: METADATA_ONLY (Optional)<br></br>
+> `Config Param: PARTITION_SELECTOR_REGEX_MODE`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.bootstrap.mode.selector
+> Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped<br></br>
+> **Default Value**: org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector (Optional)<br></br>
+> `Config Param: MODE_SELECTOR_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2808,18 +2960,26 @@ These set of configs are used to enable monitoring and reporting of keyHudi stat
Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishes metrics on every commit, clean, rollback etc.
`Config Class`: org.apache.hudi.config.HoodieMetricsDatadogConfig<br></br>
-> #### hoodie.metrics.datadog.metric.tags
-> Datadog metric tags (comma-delimited) to be sent along with metrics data.<br></br>
+> #### hoodie.metrics.datadog.api.key.skip.validation
+> Before sending metrics via Datadog API, whether to skip validating Datadog API key or not. Default to false.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: API_KEY_SKIP_VALIDATION`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.metrics.datadog.api.site
+> Datadog API site: EU or US<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_METRIC_TAGS`<br></br>
+> `Config Param: API_SITE_VALUE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.datadog.api.key.supplier
-> Datadog API key supplier to supply the API key at runtime. This will take effect if hoodie.metrics.datadog.api.key is not set.<br></br>
+> #### hoodie.metrics.datadog.metric.host
+> Datadog metric host to be sent along with metrics data.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_API_KEY_SUPPLIER`<br></br>
+> `Config Param: METRIC_HOST_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2827,7 +2987,7 @@ Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishe
> #### hoodie.metrics.datadog.metric.prefix
> Datadog metric prefix to be prepended to each metric name with a dot as delimiter. For example, if it is set to foo, foo. will be prepended.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_METRIC_PREFIX`<br></br>
+> `Config Param: METRIC_PREFIX_VALUE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2835,39 +2995,31 @@ Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishe
> #### hoodie.metrics.datadog.api.timeout.seconds
> Datadog API timeout in seconds. Default to 3.<br></br>
> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: DATADOG_API_TIMEOUT_SECONDS`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
-> #### hoodie.metrics.datadog.report.period.seconds
-> Datadog reporting period in seconds. Default to 30.<br></br>
-> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: DATADOG_REPORT_PERIOD_SECONDS`<br></br>
+> `Config Param: API_TIMEOUT_IN_SECONDS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.datadog.metric.host
-> Datadog metric host to be sent along with metrics data.<br></br>
+> #### hoodie.metrics.datadog.api.key.supplier
+> Datadog API key supplier to supply the API key at runtime. This will take effect if hoodie.metrics.datadog.api.key is not set.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_METRIC_HOST`<br></br>
+> `Config Param: API_KEY_SUPPLIER`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.datadog.api.key.skip.validation
-> Before sending metrics via Datadog API, whether to skip validating Datadog API key or not. Default to false.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: DATADOG_API_KEY_SKIP_VALIDATION`<br></br>
+> #### hoodie.metrics.datadog.metric.tags
+> Datadog metric tags (comma-delimited) to be sent along with metrics data.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: METRIC_TAG_VALUES`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.datadog.api.site
-> Datadog API site: EU or US<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_API_SITE`<br></br>
+> #### hoodie.metrics.datadog.report.period.seconds
+> Datadog reporting period in seconds. Default to 30.<br></br>
+> **Default Value**: 30 (Optional)<br></br>
+> `Config Param: REPORT_PERIOD_IN_SECONDS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2875,7 +3027,7 @@ Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishe
> #### hoodie.metrics.datadog.api.key
> Datadog API key<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_API_KEY`<br></br>
+> `Config Param: API_KEY`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2885,10 +3037,18 @@ Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishe
Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on every commit, clean, rollback etc.
`Config Class`: org.apache.hudi.config.HoodieMetricsPrometheusConfig<br></br>
-> #### hoodie.metrics.pushgateway.host
-> Hostname of the prometheus push gateway<br></br>
-> **Default Value**: localhost (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_HOST`<br></br>
+> #### hoodie.metrics.pushgateway.random.job.name.suffix
+> <br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: PUSHGATEWAY_RANDOM_JOBNAME_SUFFIX`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.metrics.pushgateway.port
+> Port for the push gateway.<br></br>
+> **Default Value**: 9091 (Optional)<br></br>
+> `Config Param: PUSHGATEWAY_PORT_NUM`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2896,7 +3056,7 @@ Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on e
> #### hoodie.metrics.pushgateway.delete.on.shutdown
> <br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_DELETE_ON_SHUTDOWN`<br></br>
+> `Config Param: PUSHGATEWAY_DELETE_ON_SHUTDOWN_ENABLE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2904,15 +3064,15 @@ Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on e
> #### hoodie.metrics.prometheus.port
> Port for prometheus server.<br></br>
> **Default Value**: 9090 (Optional)<br></br>
-> `Config Param: PROMETHEUS_PORT`<br></br>
+> `Config Param: PROMETHEUS_PORT_NUM`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.pushgateway.random.job.name.suffix
-> <br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_RANDOM_JOB_NAME_SUFFIX`<br></br>
+> #### hoodie.metrics.pushgateway.job.name
+> Name of the push gateway job.<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: PUSHGATEWAY_JOBNAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2920,23 +3080,15 @@ Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on e
> #### hoodie.metrics.pushgateway.report.period.seconds
> Reporting interval in seconds.<br></br>
> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_REPORT_PERIOD_SECONDS`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
-> #### hoodie.metrics.pushgateway.job.name
-> Name of the push gateway job.<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_JOB_NAME`<br></br>
+> `Config Param: PUSHGATEWAY_REPORT_PERIOD_IN_SECONDS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.pushgateway.port
-> Port for the push gateway.<br></br>
-> **Default Value**: 9091 (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_PORT`<br></br>
+> #### hoodie.metrics.pushgateway.host
+> Hostname of the prometheus push gateway<br></br>
+> **Default Value**: localhost (Optional)<br></br>
+> `Config Param: PUSHGATEWAY_HOST_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2946,18 +3098,42 @@ Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on e
Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean, rollback etc. The following sections list the supported reporters.
`Config Class`: org.apache.hudi.config.HoodieMetricsConfig<br></br>
+> #### hoodie.metrics.reporter.type
+> Type of metrics reporter.<br></br>
+> **Default Value**: GRAPHITE (Optional)<br></br>
+> `Config Param: METRICS_REPORTER_TYPE_VALUE`<br></br>
+> `Since Version: 0.5.0`<br></br>
+
+---
+
> #### hoodie.metrics.jmx.host
> Jmx host to connect to<br></br>
> **Default Value**: localhost (Optional)<br></br>
-> `Config Param: JMX_HOST`<br></br>
+> `Config Param: JMX_HOST_NAME`<br></br>
> `Since Version: 0.5.1`<br></br>
---
+> #### hoodie.metrics.reporter.class
+> <br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: METRICS_REPORTER_CLASS_NAME`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.metrics.graphite.port
+> Graphite port to connect to<br></br>
+> **Default Value**: 4756 (Optional)<br></br>
+> `Config Param: GRAPHITE_SERVER_PORT_NUM`<br></br>
+> `Since Version: 0.5.0`<br></br>
+
+---
+
> #### hoodie.metrics.executor.enable
> <br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ENABLE_EXECUTOR_METRICS`<br></br>
+> `Config Param: EXECUTOR_METRICS_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2965,7 +3141,7 @@ Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean
> #### hoodie.metrics.jmx.port
> Jmx port to connect to<br></br>
> **Default Value**: 9889 (Optional)<br></br>
-> `Config Param: JMX_PORT`<br></br>
+> `Config Param: JMX_PORT_NUM`<br></br>
> `Since Version: 0.5.1`<br></br>
---
@@ -2973,7 +3149,7 @@ Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean
> #### hoodie.metrics.graphite.host
> Graphite host to connect to<br></br>
> **Default Value**: localhost (Optional)<br></br>
-> `Config Param: GRAPHITE_SERVER_HOST`<br></br>
+> `Config Param: GRAPHITE_SERVER_HOST_NAME`<br></br>
> `Since Version: 0.5.0`<br></br>
---
@@ -2981,7 +3157,7 @@ Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean
> #### hoodie.metrics.on
> Turn on/off metrics reporting. off by default.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: METRICS_ON`<br></br>
+> `Config Param: TURN_METRICS_ON`<br></br>
> `Since Version: 0.5.0`<br></br>
---
@@ -2989,35 +3165,11 @@ Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean
> #### hoodie.metrics.graphite.metric.prefix
> Standard prefix applied to all metrics. This helps to add datacenter, environment information for e.g<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: GRAPHITE_METRIC_PREFIX`<br></br>
+> `Config Param: GRAPHITE_METRIC_PREFIX_VALUE`<br></br>
> `Since Version: 0.5.1`<br></br>
---
-> #### hoodie.metrics.graphite.port
-> Graphite port to connect to<br></br>
-> **Default Value**: 4756 (Optional)<br></br>
-> `Config Param: GRAPHITE_SERVER_PORT`<br></br>
-> `Since Version: 0.5.0`<br></br>
-
----
-
-> #### hoodie.metrics.reporter.type
-> Type of metrics reporter.<br></br>
-> **Default Value**: GRAPHITE (Optional)<br></br>
-> `Config Param: METRICS_REPORTER_TYPE`<br></br>
-> `Since Version: 0.5.0`<br></br>
-
----
-
-> #### hoodie.metrics.reporter.class
-> <br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: METRICS_REPORTER_CLASS`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
## Record Payload Config {#RECORD_PAYLOAD}
This is the lowest level of customization offered by Hudi. Record payloads define how to produce new values to upsert based on incoming new record and stored old record. Hudi provides default implementations such as OverwriteWithLatestAvroPayload which simply update table with the latest/last-written record. This can be overridden to a custom class extending HoodieRecordPayload class, on both datasource and WriteClient levels.
@@ -3029,14 +3181,14 @@ Payload related configs, that can be leveraged to control merges based on specif
> #### hoodie.payload.event.time.field
> Table column/field name to derive timestamp associated with the records. This canbe useful for e.g, determining the freshness of the table.<br></br>
> **Default Value**: ts (Optional)<br></br>
-> `Config Param: PAYLOAD_EVENT_TIME_FIELD_PROP`<br></br>
+> `Config Param: EVENT_TIME_FIELD`<br></br>
---
> #### hoodie.payload.ordering.field
> Table column/field name to order records that have the same key, before merging and writing to storage.<br></br>
> **Default Value**: ts (Optional)<br></br>
-> `Config Param: PAYLOAD_ORDERING_FIELD_PROP`<br></br>
+> `Config Param: ORDERING_FIELD`<br></br>
---
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index a994238..80f3785 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -374,8 +374,8 @@ module.exports = {
path: 'next',
banner: 'unreleased',
},
- '0.8.0': {
- label: '0.8.0',
+ '0.9.0': {
+ label: '0.9.0',
path: '',
}
},
diff --git a/website/versioned_docs/version-0.9.0/azure_hoodie.md b/website/versioned_docs/version-0.9.0/azure_hoodie.md
new file mode 100644
index 0000000..f28ec60
--- /dev/null
+++ b/website/versioned_docs/version-0.9.0/azure_hoodie.md
@@ -0,0 +1,50 @@
+---
+title: Microsoft Azure
+keywords: [ hudi, hive, azure, spark, presto]
+summary: In this page, we go over how to configure Hudi with Azure filesystem.
+last_modified_at: 2020-05-25T19:00:57-04:00
+---
+In this page, we explain how to use Hudi on Microsoft Azure.
+
+## Disclaimer
+
+This page is maintained by the Hudi community.
+If the information is inaccurate or you have additional information to add.
+Please feel free to create a JIRA ticket. Contribution is highly appreciated.
+
+## Supported Storage System
+
+There are two storage systems support Hudi .
+
+- Azure Blob Storage
+- Azure Data Lake Gen 2
+
+## Verified Combination of Spark and storage system
+
+#### HDInsight Spark2.4 on Azure Data Lake Storage Gen 2
+This combination works out of the box. No extra config needed.
+
+#### Databricks Spark2.4 on Azure Data Lake Storage Gen 2
+- Import Hudi jar to databricks workspace
+
+- Mount the file system to dbutils.
+ ```scala
+ dbutils.fs.mount(
+ source = "abfss://xxx@xxx.dfs.core.windows.net",
+ mountPoint = "/mountpoint",
+ extraConfigs = configs)
+ ```
+- When writing Hudi dataset, use abfss URL
+ ```scala
+ inputDF.write
+ .format("org.apache.hudi")
+ .options(opts)
+ .mode(SaveMode.Append)
+ .save("abfss://<<storage-account>>.dfs.core.windows.net/hudi-tables/customer")
+ ```
+- When reading Hudi dataset, use the mounting point
+ ```scala
+ spark.read
+ .format("org.apache.hudi")
+ .load("/mountpoint/hudi-tables/customer")
+ ```
diff --git a/website/versioned_docs/version-0.9.0/bos_hoodie.md b/website/versioned_docs/version-0.9.0/bos_hoodie.md
new file mode 100644
index 0000000..2a6cde8
--- /dev/null
+++ b/website/versioned_docs/version-0.9.0/bos_hoodie.md
@@ -0,0 +1,57 @@
+---
+title: Baidu Cloud
+keywords: [ hudi, hive, baidu, bos, spark, presto]
+summary: In this page, we go over how to configure Hudi with bos filesystem.
+last_modified_at: 2021-06-09T11:38:24-10:00
+---
+In this page, we explain how to get your Hudi job to store into Baidu BOS.
+
+## Baidu BOS configs
+
+There are two configurations required for Hudi-BOS compatibility:
+
+- Adding Baidu BOS Credentials for Hudi
+- Adding required Jars to classpath
+
+### Baidu BOS Credentials
+
+Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the `fs.defaultFS` with your BOS bucket name, replace `fs.bos.endpoint` with your bos endpoint, replace `fs.bos.access.key` with your bos key, replace `fs.bos.secret.access.key` with your bos secret key. Hudi should be able to read/write from the bucket.
+
+```xml
+<property>
+ <name>fs.defaultFS</name>
+ <value>bos://bucketname/</value>
+</property>
+
+<property>
+ <name>fs.bos.endpoint</name>
+ <value>bos-endpoint-address</value>
+ <description>Baidu bos endpoint to connect to,for example : http://bj.bcebos.com</description>
+</property>
+
+<property>
+ <name>fs.bos.access.key</name>
+ <value>bos-key</value>
+ <description>Baidu access key</description>
+</property>
+
+<property>
+ <name>fs.bos.secret.access.key</name>
+ <value>bos-secret-key</value>
+ <description>Baidu secret key.</description>
+</property>
+
+<property>
+ <name>fs.bos.impl</name>
+ <value>org.apache.hadoop.fs.bos.BaiduBosFileSystem</value>
+</property>
+```
+
+### Baidu bos Libs
+
+Baidu hadoop libraries jars to add to our classpath
+
+- com.baidubce:bce-java-sdk:0.10.165
+- bos-hdfs-sdk-1.0.2-community.jar
+
+You can download the bos-hdfs-sdk jar from [here](https://sdk.bce.baidu.com/console-sdk/bos-hdfs-sdk-1.0.2-community.jar.zip) , and then unzip it.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.9.0/cloud.md b/website/versioned_docs/version-0.9.0/cloud.md
new file mode 100644
index 0000000..57491ec
--- /dev/null
+++ b/website/versioned_docs/version-0.9.0/cloud.md
@@ -0,0 +1,27 @@
+---
+title: Cloud Storage
+keywords: [hudi, aws, gcp, oss, azure, cloud]
+summary: "In this page, we introduce how Hudi work with different Cloud providers."
+toc: true
+last_modified_at: 2019-06-16T21:59:57-04:00
+---
+
+## Talking to Cloud Storage
+
+Immaterial of whether RDD/WriteClient APIs or Datasource is used, the following information helps configure access
+to cloud stores.
+
+ * [AWS S3](/docs/s3_hoodie) <br/>
+ Configurations required for S3 and Hudi co-operability.
+ * [Google Cloud Storage](/docs/gcs_hoodie) <br/>
+ Configurations required for GCS and Hudi co-operability.
+ * [Alibaba Cloud OSS](/docs/oss_hoodie) <br/>
+ Configurations required for OSS and Hudi co-operability.
+ * [Microsoft Azure](/docs/azure_hoodie) <br/>
+ Configurations required for Azure and Hudi co-operability.
+* [Tencent Cloud Object Storage](/docs/cos_hoodie) <br/>
+ Configurations required for COS and Hudi co-operability.
+* [IBM Cloud Object Storage](/docs/ibm_cos_hoodie) <br/>
+ Configurations required for IBM Cloud Object Storage and Hudi co-operability.
+* [Baidu Cloud Object Storage](bos_hoodie) <br/>
+ Configurations required for BOS and Hudi co-operability.
diff --git a/website/versioned_docs/version-0.9.0/comparison.md b/website/versioned_docs/version-0.9.0/comparison.md
new file mode 100644
index 0000000..681b359
--- /dev/null
+++ b/website/versioned_docs/version-0.9.0/comparison.md
@@ -0,0 +1,56 @@
+---
+title: "Comparison"
+keywords: [ apache, hudi, kafka, kudu, hive, hbase, stream processing]
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+
+Apache Hudi fills a big void for processing data on top of DFS, and thus mostly co-exists nicely with these technologies. However,
+it would be useful to understand how Hudi fits into the current big data ecosystem, contrasting it with a few related systems
+and bring out the different tradeoffs these systems have accepted in their design.
+
+## Kudu
+
+[Apache Kudu](https://kudu.apache.org) is a storage system that has similar goals as Hudi, which is to bring real-time analytics on petabytes of data via first
+class support for `upserts`. A key differentiator is that Kudu also attempts to serve as a datastore for OLTP workloads, something that Hudi does not aspire to be.
+Consequently, Kudu does not support incremental pulling (as of early 2017), something Hudi does to enable incremental processing use cases.
+
+
+Kudu diverges from a distributed file system abstraction and HDFS altogether, with its own set of storage servers talking to each other via RAFT.
+Hudi, on the other hand, is designed to work with an underlying Hadoop compatible filesystem (HDFS,S3 or Ceph) and does not have its own fleet of storage servers,
+instead relying on Apache Spark to do the heavy-lifting. Thus, Hudi can be scaled easily, just like other Spark jobs, while Kudu would require hardware
+& operational support, typical to datastores like HBase or Vertica. We have not at this point, done any head to head benchmarks against Kudu (given RTTable is WIP).
+But, if we were to go with results shared by [CERN](https://db-blog.web.cern.ch/blog/zbigniew-baranowski/2017-01-performance-comparison-different-file-formats-and-storage-engines) ,
+we expect Hudi to positioned at something that ingests parquet with superior performance.
+
+
+## Hive Transactions
+
+[Hive Transactions/ACID](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions) is another similar effort, which tries to implement storage like
+`merge-on-read`, on top of ORC file format. Understandably, this feature is heavily tied to Hive and other efforts like [LLAP](https://cwiki.apache.org/confluence/display/Hive/LLAP).
+Hive transactions does not offer the read-optimized storage option or the incremental pulling, that Hudi does. In terms of implementation choices, Hudi leverages
+the full power of a processing framework like Spark, while Hive transactions feature is implemented underneath by Hive tasks/queries kicked off by user or the Hive metastore.
+Based on our production experience, embedding Hudi as a library into existing Spark pipelines was much easier and less operationally heavy, compared with the other approach.
+Hudi is also designed to work with non-hive engines like PrestoDB/Spark and will incorporate file formats other than parquet over time.
+
+## HBase
+
+Even though [HBase](https://hbase.apache.org) is ultimately a key-value store for OLTP workloads, users often tend to associate HBase with analytics given the proximity to Hadoop.
+Given HBase is heavily write-optimized, it supports sub-second upserts out-of-box and Hive-on-HBase lets users query that data. However, in terms of actual performance for analytical workloads,
+hybrid columnar storage formats like Parquet/ORC handily beat HBase, since these workloads are predominantly read-heavy. Hudi bridges this gap between faster data and having
+analytical storage formats. From an operational perspective, arming users with a library that provides faster data, is more scalable, than managing a big farm of HBase region servers,
+just for analytics. Finally, HBase does not support incremental processing primitives like `commit times`, `incremental pull` as first class citizens like Hudi.
+
+## Stream Processing
+
+A popular question, we get is : "How does Hudi relate to stream processing systems?", which we will try to answer here. Simply put, Hudi can integrate with
+batch (`copy-on-write table`) and streaming (`merge-on-read table`) jobs of today, to store the computed results in Hadoop. For Spark apps, this can happen via direct
+integration of Hudi library with Spark/Spark streaming DAGs. In case of Non-Spark processing systems (eg: Flink, Hive), the processing can be done in the respective systems
+and later sent into a Hudi table via a Kafka topic/DFS intermediate file. In more conceptual level, data processing
+pipelines just consist of three components : `source`, `processing`, `sink`, with users ultimately running queries against the sink to use the results of the pipeline.
+Hudi can act as either a source or sink, that stores data on DFS. Applicability of Hudi to a given stream processing pipeline ultimately boils down to suitability
+of PrestoDB/SparkSQL/Hive for your queries.
+
+More advanced use cases revolve around the concepts of [incremental processing](https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop), which effectively
+uses Hudi even inside the `processing` engine to speed up typical batch pipelines. For e.g: Hudi can be used as a state store inside a processing DAG (similar
+to how [rocksDB](https://ci.apache.org/projects/flink/flink-docs-release-1.2/ops/state_backends#the-rocksdbstatebackend) is used by Flink). This is an item on the roadmap
+and will eventually happen as a [Beam Runner](https://issues.apache.org/jira/browse/HUDI-60)
diff --git a/website/versioned_docs/version-0.9.0/concepts.md b/website/versioned_docs/version-0.9.0/concepts.md
new file mode 100644
index 0000000..277484d
--- /dev/null
+++ b/website/versioned_docs/version-0.9.0/concepts.md
@@ -0,0 +1,172 @@
+---
+version: 0.6.0
+title: "Concepts"
+keywords: [ hudi, design, table, queries, timeline]
+summary: "Here we introduce some basic concepts & give a broad technical overview of Hudi"
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+
+Apache Hudi (pronounced “Hudi”) provides the following streaming primitives over hadoop compatible storages
+
+ * Update/Delete Records (how do I change records in a table?)
+ * Change Streams (how do I fetch records that changed?)
+
+In this section, we will discuss key concepts & terminologies that are important to understand, to be able to effectively use these primitives.
+
+## Timeline
+At its core, Hudi maintains a `timeline` of all actions performed on the table at different `instants` of time that helps provide instantaneous views of the table,
+while also efficiently supporting retrieval of data in the order of arrival. A Hudi instant consists of the following components
+
+ * `Instant action` : Type of action performed on the table
+ * `Instant time` : Instant time is typically a timestamp (e.g: 20190117010349), which monotonically increases in the order of action's begin time.
+ * `state` : current state of the instant
+
+Hudi guarantees that the actions performed on the timeline are atomic & timeline consistent based on the instant time.
+
+Key actions performed include
+
+ * `COMMITS` - A commit denotes an **atomic write** of a batch of records into a table.
+ * `CLEANS` - Background activity that gets rid of older versions of files in the table, that are no longer needed.
+ * `DELTA_COMMIT` - A delta commit refers to an **atomic write** of a batch of records into a MergeOnRead type table, where some/all of the data could be just written to delta logs.
+ * `COMPACTION` - Background activity to reconcile differential data structures within Hudi e.g: moving updates from row based log files to columnar formats. Internally, compaction manifests as a special commit on the timeline
+ * `ROLLBACK` - Indicates that a commit/delta commit was unsuccessful & rolled back, removing any partial files produced during such a write
+ * `SAVEPOINT` - Marks certain file groups as "saved", such that cleaner will not delete them. It helps restore the table to a point on the timeline, in case of disaster/data recovery scenarios.
+
+Any given instant can be
+in one of the following states
+
+ * `REQUESTED` - Denotes an action has been scheduled, but has not initiated
+ * `INFLIGHT` - Denotes that the action is currently being performed
+ * `COMPLETED` - Denotes completion of an action on the timeline
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_timeline.png" alt="hudi_timeline.png" />
+</figure>
+
+Example above shows upserts happenings between 10:00 and 10:20 on a Hudi table, roughly every 5 mins, leaving commit metadata on the Hudi timeline, along
+with other background cleaning/compactions. One key observation to make is that the commit time indicates the `arrival time` of the data (10:20AM), while the actual data
+organization reflects the actual time or `event time`, the data was intended for (hourly buckets from 07:00). These are two key concepts when reasoning about tradeoffs between latency and completeness of data.
+
+When there is late arriving data (data intended for 9:00 arriving >1 hr late at 10:20), we can see the upsert producing new data into even older time buckets/folders.
+With the help of the timeline, an incremental query attempting to get all new data that was committed successfully since 10:00 hours, is able to very efficiently consume
+only the changed files without say scanning all the time buckets > 07:00.
+
+## File management
+Hudi organizes a table into a directory structure under a `basepath` on DFS. Table is broken up into partitions, which are folders containing data files for that partition,
+very similar to Hive tables. Each partition is uniquely identified by its `partitionpath`, which is relative to the basepath.
+
+Within each partition, files are organized into `file groups`, uniquely identified by a `file id`. Each file group contains several
+`file slices`, where each slice contains a base file (`*.parquet`) produced at a certain commit/compaction instant time,
+ along with set of log files (`*.log.*`) that contain inserts/updates to the base file since the base file was produced.
+Hudi adopts a MVCC design, where compaction action merges logs and base files to produce new file slices and cleaning action gets rid of
+unused/older file slices to reclaim space on DFS.
+
+## Index
+Hudi provides efficient upserts, by mapping a given hoodie key (record key + partition path) consistently to a file id, via an indexing mechanism.
+This mapping between record key and file group/file id, never changes once the first version of a record has been written to a file. In short, the
+mapped file group contains all versions of a group of records.
+
+## Table Types & Queries
+Hudi table types define how data is indexed & laid out on the DFS and how the above primitives and timeline activities are implemented on top of such organization (i.e how data is written).
+In turn, `query types` define how the underlying data is exposed to the queries (i.e how data is read).
+
+| Table Type | Supported Query types |
+|-------------- |------------------|
+| Copy On Write | Snapshot Queries + Incremental Queries |
+| Merge On Read | Snapshot Queries + Incremental Queries + Read Optimized Queries |
+
+### Table Types
+Hudi supports the following table types.
+
+ - [Copy On Write](#copy-on-write-table) : Stores data using exclusively columnar file formats (e.g parquet). Updates simply version & rewrite the files by performing a synchronous merge during write.
+ - [Merge On Read](#merge-on-read-table) : Stores data using a combination of columnar (e.g parquet) + row based (e.g avro) file formats. Updates are logged to delta files & later compacted to produce new versions of columnar files synchronously or asynchronously.
+
+Following table summarizes the trade-offs between these two table types
+
+| Trade-off | CopyOnWrite | MergeOnRead |
+|-------------- |------------------| ------------------|
+| Data Latency | Higher | Lower |
+| Update cost (I/O) | Higher (rewrite entire parquet) | Lower (append to delta log) |
+| Parquet File Size | Smaller (high update(I/0) cost) | Larger (low update cost) |
+| Write Amplification | Higher | Lower (depending on compaction strategy) |
+
+
+### Query types
+Hudi supports the following query types
+
+ - **Snapshot Queries** : Queries see the latest snapshot of the table as of a given commit or compaction action. In case of merge on read table, it exposes near-real time data(few mins) by merging
+ the base and delta files of the latest file slice on-the-fly. For copy on write table, it provides a drop-in replacement for existing parquet tables, while providing upsert/delete and other write side features.
+ - **Incremental Queries** : Queries only see new data written to the table, since a given commit/compaction. This effectively provides change streams to enable incremental data pipelines.
+ - **Read Optimized Queries** : Queries see the latest snapshot of table as of a given commit/compaction action. Exposes only the base/columnar files in latest file slices and guarantees the
+ same columnar query performance compared to a non-hudi columnar table.
+
+Following table summarizes the trade-offs between the different query types.
+
+| Trade-off | Snapshot | Read Optimized |
+|-------------- |-------------| ------------------|
+| Data Latency | Lower | Higher
+| Query Latency | Higher (merge base / columnar file + row based delta / log files) | Lower (raw base / columnar file performance)
+
+
+## Copy On Write Table
+
+File slices in Copy-On-Write table only contain the base/columnar file and each commit produces new versions of base files.
+In other words, we implicitly compact on every commit, such that only columnar data exists. As a result, the write amplification
+(number of bytes written for 1 byte of incoming data) is much higher, where read amplification is zero.
+This is a much desired property for analytical workloads, which is predominantly read-heavy.
+
+Following illustrates how this works conceptually, when data written into copy-on-write table and two queries running on top of it.
+
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_cow.png" alt="hudi_cow.png" />
+</figure>
+
+
+As data gets written, updates to existing file groups produce a new slice for that file group stamped with the commit instant time,
+while inserts allocate a new file group and write its first slice for that file group. These file slices and their commit instant times are color coded above.
+SQL queries running against such a table (eg: `select count(*)` counting the total records in that partition), first checks the timeline for the latest commit
+and filters all but latest file slices of each file group. As you can see, an old query does not see the current inflight commit's files color coded in pink,
+but a new query starting after the commit picks up the new data. Thus queries are immune to any write failures/partial writes and only run on committed data.
+
+The intention of copy on write table, is to fundamentally improve how tables are managed today through
+
+ - First class support for atomically updating data at file-level, instead of rewriting whole tables/partitions
+ - Ability to incremental consume changes, as opposed to wasteful scans or fumbling with heuristics
+ - Tight control of file sizes to keep query performance excellent (small files hurt query performance considerably).
+
+
+## Merge On Read Table
+
+Merge on read table is a superset of copy on write, in the sense it still supports read optimized queries of the table by exposing only the base/columnar files in latest file slices.
+Additionally, it stores incoming upserts for each file group, onto a row based delta log, to support snapshot queries by applying the delta log,
+onto the latest version of each file id on-the-fly during query time. Thus, this table type attempts to balance read and write amplification intelligently, to provide near real-time data.
+The most significant change here, would be to the compactor, which now carefully chooses which delta log files need to be compacted onto
+their columnar base file, to keep the query performance in check (larger delta log files would incur longer merge times with merge data on query side)
+
+Following illustrates how the table works, and shows two types of queries - snapshot query and read optimized query.
+
+<figure>
+ <img className="docimage" src="/assets/images/hudi_mor.png" alt="hudi_mor.png" />
+</figure>
+
+There are lot of interesting things happening in this example, which bring out the subtleties in the approach.
+
+ - We now have commits every 1 minute or so, something we could not do in the other table type.
+ - Within each file id group, now there is an delta log file, which holds incoming updates to records in the base columnar files. In the example, the delta log files hold
+ all the data from 10:05 to 10:10. The base columnar files are still versioned with the commit, as before.
+ Thus, if one were to simply look at base files alone, then the table layout looks exactly like a copy on write table.
+ - A periodic compaction process reconciles these changes from the delta log and produces a new version of base file, just like what happened at 10:05 in the example.
+ - There are two ways of querying the same underlying table: Read Optimized query and Snapshot query, depending on whether we chose query performance or freshness of data.
+ - The semantics around when data from a commit is available to a query changes in a subtle way for a read optimized query. Note, that such a query
+ running at 10:10, wont see data after 10:05 above, while a snapshot query always sees the freshest data.
+ - When we trigger compaction & what it decides to compact hold all the key to solving these hard problems. By implementing a compacting
+ strategy, where we aggressively compact the latest partitions compared to older partitions, we could ensure the read optimized queries see data
+ published within X minutes in a consistent fashion.
+
+The intention of merge on read table is to enable near real-time processing directly on top of DFS, as opposed to copying
+data out to specialized systems, which may not be able to handle the data volume. There are also a few secondary side benefits to
+this table such as reduced write amplification by avoiding synchronous merge of data, i.e, the amount of data written per 1 bytes of data in a batch
+
+
diff --git a/website/versioned_docs/version-0.9.0/concurrency_control.md b/website/versioned_docs/version-0.9.0/concurrency_control.md
new file mode 100644
index 0000000..96ff6eb
--- /dev/null
+++ b/website/versioned_docs/version-0.9.0/concurrency_control.md
@@ -0,0 +1,149 @@
+---
+title: "Concurrency Control"
+summary: In this page, we will discuss how to perform concurrent writes to Hudi Tables.
+toc: true
+last_modified_at: 2021-03-19T15:59:57-04:00
+---
+
+In this section, we will cover Hudi's concurrency model and describe ways to ingest data into a Hudi Table from multiple writers; using the [DeltaStreamer](#deltastreamer) tool as well as
+using the [Hudi datasource](#datasource-writer).
+
+## Supported Concurrency Controls
+
+- **MVCC** : Hudi table services such as compaction, cleaning, clustering leverage Multi Version Concurrency Control to provide snapshot isolation
+between multiple table service writers and readers. Additionally, using MVCC, Hudi provides snapshot isolation between an ingestion writer and multiple concurrent readers.
+ With this model, Hudi supports running any number of table service jobs concurrently, without any concurrency conflict.
+ This is made possible by ensuring that scheduling plans of such table services always happens in a single writer mode to ensure no conflict and avoids race conditions.
+
+- **[NEW] OPTIMISTIC CONCURRENCY** : Write operations such as the ones described above (UPSERT, INSERT) etc, leverage optimistic concurrency control to enable multiple ingestion writers to
+the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits (or writers) happening to the same table, if they do not have writes to overlapping files being changed, both writers are allowed to succeed.
+ This feature is currently *experimental* and requires either Zookeeper or HiveMetastore to acquire locks.
+
+It may be helpful to understand the different guarantees provided by [write operations](/docs/writing_data#write-operations) via Hudi datasource or the delta streamer.
+
+## Single Writer Guarantees
+
+ - *UPSERT Guarantee*: The target table will NEVER show duplicates.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order.
+
+## Multi Writer Guarantees
+
+With multiple writers using OCC, some of the above guarantees change as follows
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out of order due to multiple writer jobs finishing at different times.
+
+## Enabling Multi Writing
+
+The following properties are needed to be set properly to turn on optimistic concurrency control.
+
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.cleaner.policy.failed.writes=LAZY
+hoodie.write.lock.provider=<lock-provider-classname>
+```
+
+There are 2 different server based lock providers that require different configuration to be set.
+
+**`Zookeeper`** based lock provider
+
+```
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
+hoodie.write.lock.zookeeper.url
+hoodie.write.lock.zookeeper.port
+hoodie.write.lock.zookeeper.lock_key
+hoodie.write.lock.zookeeper.base_path
+```
+
+**`HiveMetastore`** based lock provider
+
+```
+hoodie.write.lock.provider=org.apache.hudi.hive.HiveMetastoreBasedLockProvider
+hoodie.write.lock.hivemetastore.database
+hoodie.write.lock.hivemetastore.table
+```
+
+`The HiveMetastore URI's are picked up from the hadoop configuration file loaded during runtime.`
+
+## Datasource Writer
+
+The `hudi-spark` module offers the DataSource API to write (and read) a Spark DataFrame into a Hudi table.
+
+Following is an example of how to use optimistic_concurrency_control via spark datasource
+
+```java
+inputDF.write.format("hudi")
+ .options(getQuickstartWriteConfigs)
+ .option(PRECOMBINE_FIELD_OPT_KEY, "ts")
+ .option("hoodie.cleaner.policy.failed.writes", "LAZY")
+ .option("hoodie.write.concurrency.mode", "optimistic_concurrency_control")
+ .option("hoodie.write.lock.zookeeper.url", "zookeeper")
+ .option("hoodie.write.lock.zookeeper.port", "2181")
+ .option("hoodie.write.lock.zookeeper.lock_key", "test_table")
+ .option("hoodie.write.lock.zookeeper.base_path", "/test")
+ .option(RECORDKEY_FIELD_OPT_KEY, "uuid")
+ .option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath")
+ .option(TABLE_NAME, tableName)
+ .mode(Overwrite)
+ .save(basePath)
+```
+
+## DeltaStreamer
+
+The `HoodieDeltaStreamer` utility (part of hudi-utilities-bundle) provides ways to ingest from different sources such as DFS or Kafka, with the following capabilities.
+
+Using optimistic_concurrency_control via delta streamer requires adding the above configs to the properties file that can be passed to the
+job. For example below, adding the configs to kafka-source.properties file and passing them to deltastreamer will enable optimistic concurrency.
+A deltastreamer job can then be triggered as follows:
+
+```java
+[hoodie]$ spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer `ls packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` \
+ --props file://${PWD}/hudi-utilities/src/test/resources/delta-streamer-config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --target-base-path file:\/\/\/tmp/hudi-deltastreamer-op \
+ --target-table uber.impressions \
+ --op BULK_INSERT
+```
+
+## Best Practices when using Optimistic Concurrency Control
+
+Concurrent Writing to Hudi tables requires acquiring a lock with either Zookeeper or HiveMetastore. Due to several reasons you might want to configure retries to allow your application to acquire the lock.
+1. Network connectivity or excessive load on servers increasing time for lock acquisition resulting in timeouts
+2. Running a large number of concurrent jobs that are writing to the same hudi table can result in contention during lock acquisition can cause timeouts
+3. In some scenarios of conflict resolution, Hudi commit operations might take upto 10's of seconds while the lock is being held. This can result in timeouts for other jobs waiting to acquire a lock.
+
+Set the correct native lock provider client retries. NOTE that sometimes these settings are set on the server once and all clients inherit the same configs. Please check your settings before enabling optimistic concurrency.
+
+```
+hoodie.write.lock.wait_time_ms
+hoodie.write.lock.num_retries
+```
+
+Set the correct hudi client retries for Zookeeper & HiveMetastore. This is useful in cases when native client retry settings cannot be changed. Please note that these retries will happen in addition to any native client retries that you may have set.
+
+```
+hoodie.write.lock.client.wait_time_ms
+hoodie.write.lock.client.num_retries
+```
+
+*Setting the right values for these depends on a case by case basis; some defaults have been provided for general cases.*
+
+## Disabling Multi Writing
+
+Remove the following settings that were used to enable multi-writer or override with default values.
+
+```
+hoodie.write.concurrency.mode=single_writer
+hoodie.cleaner.policy.failed.writes=EAGER
+```
+
+## Caveats
+
+If you are using the `WriteClient` API, please note that multiple writes to the table need to be initiated from 2 different instances of the write client.
+It is NOT recommended to use the same instance of the write client to perform multi writing.
\ No newline at end of file
diff --git a/website/docs/configurations.md b/website/versioned_docs/version-0.9.0/configurations.md
similarity index 82%
copy from website/docs/configurations.md
copy to website/versioned_docs/version-0.9.0/configurations.md
index a8307db..31e5f9f 100644
--- a/website/docs/configurations.md
+++ b/website/versioned_docs/version-0.9.0/configurations.md
@@ -4,7 +4,7 @@ keywords: [ configurations, default, flink options, spark, configs, parameters ]
permalink: /docs/configurations.html
summary: This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
toc: true
-last_modified_at: 2021-08-09T17:00:50.653032
+last_modified_at: 2021-08-26T22:04:13.167
---
This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
@@ -24,6 +24,13 @@ Options useful for reading tables via `read.format.option(...)`
`Config Class`: org.apache.hudi.DataSourceOptions.scala<br></br>
+> #### hoodie.datasource.read.incr.path.glob
+> For the use-cases like users only want to incremental pull from certain partitions instead of the full table. This option allows using glob pattern to directly filter on path.<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: INCR_PATH_GLOB`<br></br>
+
+---
+
> #### hoodie.file.index.enable
> Enables use of the spark file index implementation for Hudi, that speeds up listing of large tables.<br></br>
> **Default Value**: true (Optional)<br></br>
@@ -31,24 +38,17 @@ Options useful for reading tables via `read.format.option(...)`
---
-> #### hoodie.datasource.merge.type
-> For Snapshot query on merge on read table, control whether we invoke the record payload implementation to merge (payload_combine) or skip merging altogetherskip_merge<br></br>
-> **Default Value**: payload_combine (Optional)<br></br>
-> `Config Param: REALTIME_MERGE_OPT_KEY`<br></br>
-
----
-
-> #### hoodie.datasource.read.incr.path.glob
-> For the use-cases like users only want to incremental pull from certain partitions instead of the full table. This option allows using glob pattern to directly filter on path.<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: INCR_PATH_GLOB_OPT_KEY`<br></br>
+> #### hoodie.datasource.read.paths
+> Comma separated list of file paths to read within a Hudi table.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: READ_PATHS`<br></br>
---
-> #### hoodie.datasource.query.type
-> Whether data needs to be read, in incremental mode (new data since an instantTime) (or) Read Optimized mode (obtain latest view, based on base files) (or) Snapshot mode (obtain latest view, by merging base and (if any) log files)<br></br>
-> **Default Value**: snapshot (Optional)<br></br>
-> `Config Param: QUERY_TYPE_OPT_KEY`<br></br>
+> #### hoodie.datasource.read.end.instanttime
+> Instant time to limit incrementally fetched data to. New data written with an instant_time <= END_INSTANTTIME are fetched out.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: END_INSTANTTIME`<br></br>
---
@@ -59,38 +59,45 @@ Options useful for reading tables via `read.format.option(...)`
---
-> #### hoodie.datasource.read.end.instanttime
-> Instant time to limit incrementally fetched data to. New data written with an instant_time <= END_INSTANTTIME are fetched out.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: END_INSTANTTIME_OPT_KEY`<br></br>
+> #### hoodie.datasource.read.incr.filters
+> For use-cases like DeltaStreamer which reads from Hoodie Incremental table and applies opaque map functions, filters appearing late in the sequence of transformations cannot be automatically pushed down. This option allows setting filters directly on Hoodie Source.<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: PUSH_DOWN_INCR_FILTERS`<br></br>
+
+---
+
+> #### hoodie.datasource.merge.type
+> For Snapshot query on merge on read table, control whether we invoke the record payload implementation to merge (payload_combine) or skip merging altogetherskip_merge<br></br>
+> **Default Value**: payload_combine (Optional)<br></br>
+> `Config Param: REALTIME_MERGE`<br></br>
---
> #### hoodie.datasource.read.begin.instanttime
> Instant time to start incrementally pulling data from. The instanttime here need not necessarily correspond to an instant on the timeline. New data written with an instant_time > BEGIN_INSTANTTIME are fetched out. For e.g: ‘20170901080000’ will get all new data written after Sep 1, 2017 08:00AM.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BEGIN_INSTANTTIME_OPT_KEY`<br></br>
+> `Config Param: BEGIN_INSTANTTIME`<br></br>
---
-> #### hoodie.datasource.read.schema.use.end.instanttime
-> Uses end instant schema when incrementally fetched data to. Default: users latest instant schema.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME_OPT_KEY`<br></br>
+> #### as.of.instant
+> The query instant for time travel. Without specified this option, we query the latest snapshot.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: TIME_TRAVEL_AS_OF_INSTANT`<br></br>
---
-> #### hoodie.datasource.read.incr.filters
-> For use-cases like DeltaStreamer which reads from Hoodie Incremental table and applies opaque map functions, filters appearing late in the sequence of transformations cannot be automatically pushed down. This option allows setting filters directly on Hoodie Source.<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: PUSH_DOWN_INCR_FILTERS_OPT_KEY`<br></br>
+> #### hoodie.datasource.query.type
+> Whether data needs to be read, in incremental mode (new data since an instantTime) (or) Read Optimized mode (obtain latest view, based on base files) (or) Snapshot mode (obtain latest view, by merging base and (if any) log files)<br></br>
+> **Default Value**: snapshot (Optional)<br></br>
+> `Config Param: QUERY_TYPE`<br></br>
---
-> #### hoodie.datasource.read.paths
-> Comma separated list of file paths to read within a Hudi table.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: READ_PATHS_OPT_KEY`<br></br>
+> #### hoodie.datasource.read.schema.use.end.instanttime
+> Uses end instant schema when incrementally fetched data to. Default: users latest instant schema.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME`<br></br>
---
@@ -114,311 +121,354 @@ Options useful for writing tables via `write.format.option(...)`
`Config Class`: org.apache.hudi.DataSourceOptions.scala<br></br>
-> #### hoodie.datasource.hive_sync.database
-> database to sync to<br></br>
-> **Default Value**: default (Optional)<br></br>
-> `Config Param: HIVE_DATABASE_OPT_KEY`<br></br>
+> #### hoodie.datasource.clustering.async.enable
+> Enable asynchronous clustering. Disabled by default.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ASYNC_CLUSTERING_ENABLE`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.datasource.write.precombine.field
-> Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br></br>
-> **Default Value**: ts (Optional)<br></br>
-> `Config Param: PRECOMBINE_FIELD_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.operation
+> Whether to do upsert, insert or bulkinsert for the write operation. Use bulkinsert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br></br>
+> **Default Value**: upsert (Optional)<br></br>
+> `Config Param: OPERATION`<br></br>
---
-> #### hoodie.datasource.hive_sync.username
-> hive user name to use<br></br>
-> **Default Value**: hive (Optional)<br></br>
-> `Config Param: HIVE_USER_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.reconcile.schema
+> When a new batch of write has records with old schema, but latest table schema got evolved, this config will upgrade the records to leverage latest table schema(default values will be injected to missing fields). If not, the write batch would fail.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: RECONCILE_SCHEMA`<br></br>
---
-> #### hoodie.datasource.hive_sync.table_properties
-> Additional properties to store with table.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_TABLE_PROPERTIES`<br></br>
+> #### hoodie.datasource.write.recordkey.field
+> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
+Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using
+the dot notation eg: `a.b.c`<br></br>
+> **Default Value**: uuid (Optional)<br></br>
+> `Config Param: RECORDKEY_FIELD`<br></br>
---
-> #### hoodie.datasource.hive_sync.batch_num
-> The number of partitions one batch when synchronous partitions to hive.<br></br>
-> **Default Value**: 1000 (Optional)<br></br>
-> `Config Param: HIVE_BATCH_SYNC_PARTITION_NUM`<br></br>
+> #### hoodie.datasource.hive_sync.skip_ro_suffix
+> Skip the _ro suffix for Read optimized table, when registering<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE`<br></br>
---
-> #### hoodie.datasource.write.keygenerator.class
-> Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: KEYGENERATOR_CLASS_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.partitionpath.urlencode
+> Should we url encode the partition path value, before creating the folder structure.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: URL_ENCODE_PARTITIONING`<br></br>
---
-> #### hoodie.datasource.hive_sync.base_file_format
-> Base file format for the sync.<br></br>
-> **Default Value**: PARQUET (Optional)<br></br>
-> `Config Param: HIVE_BASE_FILE_FORMAT_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.partition_extractor_class
+> <br></br>
+> **Default Value**: org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor (Optional)<br></br>
+> `Config Param: HIVE_PARTITION_EXTRACTOR_CLASS`<br></br>
---
-> #### hoodie.datasource.write.table.type
-> The table type for the underlying data, for this write. This can’t change between writes.<br></br>
-> **Default Value**: COPY_ON_WRITE (Optional)<br></br>
-> `Config Param: TABLE_TYPE_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.serde_properties
+> <br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_TABLE_SERDE_PROPERTIES`<br></br>
---
-> #### hoodie.datasource.write.operation
-> Whether to do upsert, insert or bulkinsert for the write operation. Use bulkinsert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br></br>
-> **Default Value**: upsert (Optional)<br></br>
-> `Config Param: OPERATION_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.password
+> hive password to use<br></br>
+> **Default Value**: hive (Optional)<br></br>
+> `Config Param: HIVE_PASS`<br></br>
---
-> #### hoodie.datasource.write.streaming.retry.interval.ms
-> Config to indicate how long (by millisecond) before a retry should issued for failed microbatch<br></br>
-> **Default Value**: 2000 (Optional)<br></br>
-> `Config Param: STREAMING_RETRY_INTERVAL_MS_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.support_timestamp
+> ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE`<br></br>
---
-> #### hoodie.datasource.write.hive_style_partitioning
-> Flag to indicate whether to use Hive style partitioning.
-If set true, the names of partition folders follow <partition_column_name>=<partition_value> format.
-By default false (the names of partition folders are only partition values)<br></br>
+> #### hoodie.datasource.hive_sync.create_managed_table
+> Whether to sync the table as managed table.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_STYLE_PARTITIONING_OPT_KEY`<br></br>
+> `Config Param: HIVE_CREATE_MANAGED_TABLE`<br></br>
---
-> #### hoodie.datasource.hive_sync.serde_properties
-> <br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_TABLE_SERDE_PROPERTIES`<br></br>
+> #### hoodie.datasource.clustering.inline.enable
+> Enable inline clustering. Disabled by default.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INLINE_CLUSTERING_ENABLE`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.datasource.write.payload.class
-> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: PAYLOAD_CLASS_OPT_KEY`<br></br>
+> #### hoodie.datasource.compaction.async.enable
+> Controls whether async compaction should be turned on for MOR table writing.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: ASYNC_COMPACT_ENABLE`<br></br>
---
-> #### hoodie.datasource.hive_sync.assume_date_partitioning
-> Assume partitioning is yyyy/mm/dd<br></br>
+> #### hoodie.datasource.meta.sync.enable
+> <br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_ASSUME_DATE_PARTITION_OPT_KEY`<br></br>
+> `Config Param: META_SYNC_ENABLED`<br></br>
---
-> #### hoodie.meta.sync.client.tool.class
-> Sync tool class name used to sync to metastore. Defaults to Hive.<br></br>
-> **Default Value**: org.apache.hudi.hive.HiveSyncTool (Optional)<br></br>
-> `Config Param: META_SYNC_CLIENT_TOOL_CLASS`<br></br>
+> #### hoodie.datasource.write.streaming.ignore.failed.batch
+> Config to indicate whether to ignore any non exception error (e.g. writestatus error) within a streaming microbatch<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: STREAMING_IGNORE_FAILED_BATCH`<br></br>
---
-> #### hoodie.datasource.hive_sync.password
-> hive password to use<br></br>
+> #### hoodie.datasource.write.precombine.field
+> Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br></br>
+> **Default Value**: ts (Optional)<br></br>
+> `Config Param: PRECOMBINE_FIELD`<br></br>
+
+---
+
+> #### hoodie.datasource.hive_sync.username
+> hive user name to use<br></br>
> **Default Value**: hive (Optional)<br></br>
-> `Config Param: HIVE_PASS_OPT_KEY`<br></br>
+> `Config Param: HIVE_USER`<br></br>
---
-> #### hoodie.datasource.hive_sync.create_managed_table
-> Whether to sync the table as managed table.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_CREATE_MANAGED_TABLE`<br></br>
+> #### hoodie.datasource.write.partitionpath.field
+> Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value ontained by invoking .toString()<br></br>
+> **Default Value**: partitionpath (Optional)<br></br>
+> `Config Param: PARTITIONPATH_FIELD`<br></br>
+
+---
+
+> #### hoodie.datasource.write.streaming.retry.count
+> Config to indicate how many times streaming job should retry for a failed micro batch.<br></br>
+> **Default Value**: 3 (Optional)<br></br>
+> `Config Param: STREAMING_RETRY_CNT`<br></br>
---
> #### hoodie.datasource.hive_sync.partition_fields
> field in the table to use for determining hive partition columns.<br></br>
> **Default Value**: (Optional)<br></br>
-> `Config Param: HIVE_PARTITION_FIELDS_OPT_KEY`<br></br>
+> `Config Param: HIVE_PARTITION_FIELDS`<br></br>
---
-> #### hoodie.datasource.write.recordkey.field
-> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
-Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using
-the dot notation eg: `a.b.c`<br></br>
-> **Default Value**: uuid (Optional)<br></br>
-> `Config Param: RECORDKEY_FIELD_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.sync_as_datasource
+> <br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE`<br></br>
---
-> #### hoodie.datasource.write.commitmeta.key.prefix
-> Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata. This is useful to store checkpointing information, in a consistent way with the hudi timeline<br></br>
-> **Default Value**: _ (Optional)<br></br>
-> `Config Param: COMMIT_METADATA_KEYPREFIX_OPT_KEY`<br></br>
+> #### hoodie.sql.insert.mode
+> Insert mode when insert data to pk-table. The optional modes are: upsert, strict and non-strict.For upsert mode, insert statement do the upsert operation for the pk-table which will update the duplicate record.For strict mode, insert statement will keep the primary key uniqueness constraint which do not allow duplicate record.While for non-strict mode, hudi just do the insert operation for the pk-table.<br></br>
+> **Default Value**: upsert (Optional)<br></br>
+> `Config Param: SQL_INSERT_MODE`<br></br>
---
-> #### hoodie.datasource.hive_sync.table
-> table to sync to<br></br>
-> **Default Value**: unknown (Optional)<br></br>
-> `Config Param: HIVE_TABLE_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.use_jdbc
+> Use JDBC when hive synchronization is enabled<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: HIVE_USE_JDBC`<br></br>
+> `Deprecated Version: 0.9.0`<br></br>
---
-> #### hoodie.datasource.hive_sync.auto_create_database
-> Auto create hive database if does not exists<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: HIVE_AUTO_CREATE_DATABASE_OPT_KEY`<br></br>
+> #### hoodie.meta.sync.client.tool.class
+> Sync tool class name used to sync to metastore. Defaults to Hive.<br></br>
+> **Default Value**: org.apache.hudi.hive.HiveSyncTool (Optional)<br></br>
+> `Config Param: META_SYNC_CLIENT_TOOL_CLASS_NAME`<br></br>
---
-> #### hoodie.datasource.write.table.name
-> Table name for the datasource write. Also used to register the table into meta stores.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: TABLE_NAME_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.keygenerator.class
+> Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`<br></br>
+> **Default Value**: org.apache.hudi.keygen.SimpleKeyGenerator (Optional)<br></br>
+> `Config Param: KEYGENERATOR_CLASS_NAME`<br></br>
---
-> #### hoodie.datasource.write.partitionpath.urlencode
-> Should we url encode the partition path value, before creating the folder structure.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: URL_ENCODE_PARTITIONING_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.payload.class
+> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: PAYLOAD_CLASS_NAME`<br></br>
---
-> #### hoodie.datasource.hive_sync.support_timestamp
-> ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_SUPPORT_TIMESTAMP`<br></br>
+> #### hoodie.datasource.hive_sync.table_properties
+> Additional properties to store with table.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_TABLE_PROPERTIES`<br></br>
---
> #### hoodie.datasource.hive_sync.jdbcurl
> Hive metastore url<br></br>
> **Default Value**: jdbc:hive2://localhost:10000 (Optional)<br></br>
-> `Config Param: HIVE_URL_OPT_KEY`<br></br>
+> `Config Param: HIVE_URL`<br></br>
---
-> #### hoodie.datasource.compaction.async.enable
-> Controls whether async compaction should be turned on for MOR table writing.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: ASYNC_COMPACT_ENABLE_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.batch_num
+> The number of partitions one batch when synchronous partitions to hive.<br></br>
+> **Default Value**: 1000 (Optional)<br></br>
+> `Config Param: HIVE_BATCH_SYNC_PARTITION_NUM`<br></br>
---
-> #### hoodie.datasource.write.streaming.retry.count
-> Config to indicate how many times streaming job should retry for a failed micro batch.<br></br>
-> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: STREAMING_RETRY_CNT_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.assume_date_partitioning
+> Assume partitioning is yyyy/mm/dd<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: HIVE_ASSUME_DATE_PARTITION`<br></br>
---
-> #### hoodie.datasource.hive_sync.ignore_exceptions
-> <br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_IGNORE_EXCEPTIONS_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.auto_create_database
+> Auto create hive database if does not exists<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: HIVE_AUTO_CREATE_DATABASE`<br></br>
---
-> #### hoodie.datasource.write.insert.drop.duplicates
-> If set to true, filters out all duplicate records from incoming dataframe, during insert operations.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INSERT_DROP_DUPS_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.database
+> database to sync to<br></br>
+> **Default Value**: default (Optional)<br></br>
+> `Config Param: HIVE_DATABASE`<br></br>
---
-> #### hoodie.datasource.hive_sync.partition_extractor_class
-> <br></br>
-> **Default Value**: org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor (Optional)<br></br>
-> `Config Param: HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.streaming.retry.interval.ms
+> Config to indicate how long (by millisecond) before a retry should issued for failed microbatch<br></br>
+> **Default Value**: 2000 (Optional)<br></br>
+> `Config Param: STREAMING_RETRY_INTERVAL_MS`<br></br>
---
-> #### hoodie.datasource.hive_sync.use_pre_apache_input_format
-> <br></br>
+> #### hoodie.sql.bulk.insert.enable
+> When set to true, the sql insert statement will use bulk insert.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMAT_OPT_KEY`<br></br>
+> `Config Param: SQL_ENABLE_BULK_INSERT`<br></br>
---
-> #### hoodie.datasource.write.partitionpath.field
-> Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value ontained by invoking .toString()<br></br>
-> **Default Value**: partitionpath (Optional)<br></br>
-> `Config Param: PARTITIONPATH_FIELD_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.commitmeta.key.prefix
+> Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata. This is useful to store checkpointing information, in a consistent way with the hudi timeline<br></br>
+> **Default Value**: _ (Optional)<br></br>
+> `Config Param: COMMIT_METADATA_KEYPREFIX`<br></br>
---
-> #### hoodie.datasource.hive_sync.sync_as_datasource
-> <br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE`<br></br>
+> #### hoodie.datasource.write.drop.partition.columns
+> When set to true, will not write the partition columns into hudi. By default, false.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: DROP_PARTITION_COLUMNS`<br></br>
---
-> #### hoodie.deltastreamer.source.kafka.value.deserializer.class
-> This class is used by kafka client to deserialize the records<br></br>
-> **Default Value**: io.confluent.kafka.serializers.KafkaAvroDeserializer (Optional)<br></br>
-> `Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br></br>
-> `Since Version: 0.9.0`<br></br>
+> #### hoodie.datasource.hive_sync.enable
+> When set to true, register/sync the table to Apache Hive metastore<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: HIVE_SYNC_ENABLED`<br></br>
---
-> #### hoodie.datasource.hive_sync.skip_ro_suffix
-> Skip the _ro suffix for Read optimized table, when registering<br></br>
+> #### hoodie.datasource.hive_sync.table
+> table to sync to<br></br>
+> **Default Value**: unknown (Optional)<br></br>
+> `Config Param: HIVE_TABLE`<br></br>
+
+---
+
+> #### hoodie.datasource.hive_sync.ignore_exceptions
+> <br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_SKIP_RO_SUFFIX`<br></br>
+> `Config Param: HIVE_IGNORE_EXCEPTIONS`<br></br>
---
-> #### hoodie.datasource.meta.sync.enable
+> #### hoodie.datasource.hive_sync.use_pre_apache_input_format
> <br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: META_SYNC_ENABLED_OPT_KEY`<br></br>
+> `Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMAT`<br></br>
+
+---
+
+> #### hoodie.datasource.write.table.type
+> The table type for the underlying data, for this write. This can’t change between writes.<br></br>
+> **Default Value**: COPY_ON_WRITE (Optional)<br></br>
+> `Config Param: TABLE_TYPE`<br></br>
---
> #### hoodie.datasource.write.row.writer.enable
> When set to true, will perform write operations directly using the spark native `Row` representation, avoiding any additional conversion costs.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: ENABLE_ROW_WRITER`<br></br>
+
+---
+
+> #### hoodie.datasource.write.hive_style_partitioning
+> Flag to indicate whether to use Hive style partitioning.
+If set true, the names of partition folders follow <partition_column_name>=<partition_value> format.
+By default false (the names of partition folders are only partition values)<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: ENABLE_ROW_WRITER_OPT_KEY`<br></br>
+> `Config Param: HIVE_STYLE_PARTITIONING`<br></br>
---
-> #### hoodie.datasource.write.streaming.ignore.failed.batch
-> Config to indicate whether to ignore any non exception error (e.g. writestatus error) within a streaming microbatch<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: STREAMING_IGNORE_FAILED_BATCH_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.mode
+> Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_SYNC_MODE`<br></br>
---
-> #### hoodie.datasource.clustering.async.enable
-> Enable asynchronous clustering. Disabled by default.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: ASYNC_CLUSTERING_ENABLE_OPT_KEY`<br></br>
-> `Since Version: 0.9.0`<br></br>
+> #### hoodie.datasource.write.table.name
+> Table name for the datasource write. Also used to register the table into meta stores.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: TABLE_NAME`<br></br>
---
-> #### hoodie.datasource.clustering.inline.enable
-> Enable inline clustering. Disabled by default.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INLINE_CLUSTERING_ENABLE_OPT_KEY`<br></br>
+> #### hoodie.datasource.hive_sync.base_file_format
+> Base file format for the sync.<br></br>
+> **Default Value**: PARQUET (Optional)<br></br>
+> `Config Param: HIVE_BASE_FILE_FORMAT`<br></br>
+
+---
+
+> #### hoodie.deltastreamer.source.kafka.value.deserializer.class
+> This class is used by kafka client to deserialize the records<br></br>
+> **Default Value**: io.confluent.kafka.serializers.KafkaAvroDeserializer (Optional)<br></br>
+> `Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br></br>
> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.datasource.hive_sync.enable
-> When set to true, register/sync the table to Apache Hive metastore<br></br>
+> #### hoodie.datasource.write.insert.drop.duplicates
+> If set to true, filters out all duplicate records from incoming dataframe, during insert operations.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_SYNC_ENABLED_OPT_KEY`<br></br>
+> `Config Param: INSERT_DROP_DUPS`<br></br>
---
-> #### hoodie.datasource.hive_sync.use_jdbc
-> Use JDBC when hive synchronization is enabled<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: HIVE_USE_JDBC_OPT_KEY`<br></br>
+> #### hoodie.datasource.write.partitions.to.delete
+> Comma separated list of partitions to delete<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: PARTITIONS_TO_DELETE`<br></br>
---
@@ -430,6 +480,13 @@ These configs control the Hudi Flink SQL source/sink connectors, providing abili
Flink jobs using the SQL can be configured through the options in WITH clause. The actual datasource level configs are listed below.
`Config Class`: org.apache.hudi.configuration.FlinkOptions<br></br>
+> #### write.bulk_insert.sort_by_partition
+> Whether to sort the inputs by partition path for bulk insert tasks, default true<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: WRITE_BULK_INSERT_SORT_BY_PARTITION`<br></br>
+
+---
+
> #### read.streaming.enabled
> Whether to read as streaming source, default false<br></br>
> **Default Value**: false (Optional)<br></br>
@@ -484,6 +541,13 @@ Disabled by default for backward compatibility.<br></br>
---
+> #### metadata.compaction.delta_commits
+> Max delta commits for metadata table to trigger compaction, default 24<br></br>
+> **Default Value**: 24 (Optional)<br></br>
+> `Config Param: METADATA_COMPACTION_DELTA_COMMITS`<br></br>
+
+---
+
> #### hive_sync.assume_date_partitioning
> Assume partitioning is yyyy/mm/dd, default false<br></br>
> **Default Value**: false (Optional)<br></br>
@@ -491,6 +555,13 @@ Disabled by default for backward compatibility.<br></br>
---
+> #### write.bulk_insert.shuffle_by_partition
+> Whether to shuffle the inputs by partition path for bulk insert tasks, default true<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: WRITE_BULK_INSERT_SHUFFLE_BY_PARTITION`<br></br>
+
+---
+
> #### hive_sync.table
> Table name for hive sync, default 'unknown'<br></br>
> **Default Value**: unknown (Optional)<br></br>
@@ -498,6 +569,14 @@ Disabled by default for backward compatibility.<br></br>
---
+> #### write.payload.class
+> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting.
+This will render any value set for the option in-effective<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: PAYLOAD_CLASS_NAME`<br></br>
+
+---
+
> #### compaction.tasks
> Parallelism of tasks that do actual compaction, default is 10<br></br>
> **Default Value**: 10 (Optional)<br></br>
@@ -521,13 +600,6 @@ By default false (the names of partition folders are only partition values)<br><
---
-> #### hive_sync.partition_extractor_class
-> Tool to extract the partition value from HDFS path, default 'SlashEncodedDayPartitionValueExtractor'<br></br>
-> **Default Value**: org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor (Optional)<br></br>
-> `Config Param: HIVE_SYNC_PARTITION_EXTRACTOR_CLASS`<br></br>
-
----
-
> #### hive_sync.auto_create_db
> Auto create hive database if it does not exists, default true<br></br>
> **Default Value**: true (Optional)<br></br>
@@ -542,6 +614,13 @@ By default false (the names of partition folders are only partition values)<br><
---
+> #### write.sort.memory
+> Sort memory in MB, default 128MB<br></br>
+> **Default Value**: 128 (Optional)<br></br>
+> `Config Param: WRITE_SORT_MEMORY`<br></br>
+
+---
+
> #### hive_sync.enable
> Asynchronously sync Hive meta to HMS, default false<br></br>
> **Default Value**: false (Optional)<br></br>
@@ -549,6 +628,17 @@ By default false (the names of partition folders are only partition values)<br><
---
+> #### changelog.enabled
+> Whether to keep all the intermediate changes, we try to keep all the changes of a record when enabled:
+1). The sink accept the UPDATE_BEFORE message;
+2). The source try to emit every changes of a record.
+The semantics is best effort because the compaction job would finally merge all changes of a record into one.
+ default false to have UPSERT semantics<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: CHANGELOG_ENABLED`<br></br>
+
+---
+
> #### read.streaming.check-interval
> Check interval for streaming read of SECOND, default 1 minute<br></br>
> **Default Value**: 60 (Optional)<br></br>
@@ -574,6 +664,13 @@ By default 3<br></br>
---
+> #### metadata.enabled
+> Enable the internal metadata table which serves table metadata like level file listings, default false<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: METADATA_ENABLED`<br></br>
+
+---
+
> #### read.tasks
> Parallelism of tasks that do actual read, default is 4<br></br>
> **Default Value**: 4 (Optional)<br></br>
@@ -645,6 +742,13 @@ By default 2000 and it will be doubled by every retry<br></br>
---
+> #### hive_sync.partition_extractor_class
+> Tool to extract the partition value from HDFS path, default 'SlashEncodedDayPartitionValueExtractor'<br></br>
+> **Default Value**: org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor (Optional)<br></br>
+> `Config Param: HIVE_SYNC_PARTITION_EXTRACTOR_CLASS_NAME`<br></br>
+
+---
+
> #### write.batch.size
> Batch buffer size in MB to flush data into the underneath filesystem, default 64MB<br></br>
> **Default Value**: 64.0 (Optional)<br></br>
@@ -659,10 +763,10 @@ By default 2000 and it will be doubled by every retry<br></br>
---
-> #### sink.shuffle-by-partition.enable
-> The option to enable shuffle data by dynamic partition fields in sink phase, this can greatly reduce the number of file for filesystem sink but may lead data skew.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: SINK_SHUFFLE_BY_PARTITION`<br></br>
+> #### hoodie.datasource.write.keygenerator.class
+> Key generator class, that implements will extract the key out of incoming record<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: KEYGEN_CLASS_NAME`<br></br>
---
@@ -682,6 +786,13 @@ By default insert will accept duplicates, to gain extra performance<br></br>
---
+> #### write.insert.deduplicate
+> Whether to deduplicate for INSERT operation, if disabled, writes the base files directly, default true<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: INSERT_DEDUP`<br></br>
+
+---
+
> #### index.partition.regex
> Whether to load partitions in state if partition path matching， default *<br></br>
> **Default Value**: .* (Optional)<br></br>
@@ -756,14 +867,6 @@ waits for the instant commit success, only for internal use<br></br>
---
-> #### write.payload.class
-> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting.
-This will render any value set for the option in-effective<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: PAYLOAD_CLASS`<br></br>
-
----
-
> #### write.operation
> The write operation, that this write should do<br></br>
> **Default Value**: upsert (Optional)<br></br>
@@ -906,13 +1009,6 @@ it flushes the max size data bucket to avoid OOM, default 1GB<br></br>
---
-> #### hoodie.datasource.write.keygenerator.class
-> Key generator class, that implements will extract the key out of incoming record<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: KEYGEN_CLASS`<br></br>
-
----
-
> #### hoodie.datasource.write.recordkey.field
> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br></br>
@@ -921,13 +1017,6 @@ Actual value will be obtained by invoking .toString() on the field value. Nested
---
-> #### write.bulk_insert.partition.sorted
-> Whether the bulk insert write task input records are already sorted by the partition path<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: WRITE_BULK_INSERT_PARTITION_SORTED`<br></br>
-
----
-
> #### compaction.delta_seconds
> Max delta seconds time needed to trigger compaction, default 1 hour<br></br>
> **Default Value**: 3600 (Optional)<br></br>
@@ -967,7 +1056,7 @@ The consistency guard related config options, to help talk to eventually consist
> #### hoodie.optimistic.consistency.guard.sleep_time_ms
> Amount of time (in ms), to wait after which we assume storage is consistent.<br></br>
> **Default Value**: 500 (Optional)<br></br>
-> `Config Param: OPTIMISTIC_CONSISTENCY_GUARD_SLEEP_TIME_MS_PROP`<br></br>
+> `Config Param: OPTIMISTIC_CONSISTENCY_GUARD_SLEEP_TIME_MS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -975,25 +1064,24 @@ The consistency guard related config options, to help talk to eventually consist
> #### hoodie.consistency.check.max_interval_ms
> Maximum amount of time (in ms), to wait for consistency checking.<br></br>
> **Default Value**: 20000 (Optional)<br></br>
-> `Config Param: MAX_CONSISTENCY_CHECK_INTERVAL_MS_PROP`<br></br>
+> `Config Param: MAX_CHECK_INTERVAL_MS`<br></br>
> `Since Version: 0.5.0`<br></br>
> `Deprecated Version: 0.7.0`<br></br>
---
-> #### hoodie.consistency.check.enabled
-> Enabled to handle S3 eventual consistency issue. This property is no longer required since S3 is now strongly consistent. Will be removed in the future releases.<br></br>
+> #### _hoodie.optimistic.consistency.guard.enable
+> Enable consistency guard, which optimistically assumes consistency is achieved after a certain time period.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: CONSISTENCY_CHECK_ENABLED_PROP`<br></br>
-> `Since Version: 0.5.0`<br></br>
-> `Deprecated Version: 0.7.0`<br></br>
+> `Config Param: OPTIMISTIC_CONSISTENCY_GUARD_ENABLE`<br></br>
+> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.consistency.check.initial_interval_ms
-> Amount of time (in ms) to wait, before checking for consistency after an operation on storage.<br></br>
-> **Default Value**: 400 (Optional)<br></br>
-> `Config Param: INITIAL_CONSISTENCY_CHECK_INTERVAL_MS_PROP`<br></br>
+> #### hoodie.consistency.check.enabled
+> Enabled to handle S3 eventual consistency issue. This property is no longer required since S3 is now strongly consistent. Will be removed in the future releases.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ENABLE`<br></br>
> `Since Version: 0.5.0`<br></br>
> `Deprecated Version: 0.7.0`<br></br>
@@ -1002,17 +1090,18 @@ The consistency guard related config options, to help talk to eventually consist
> #### hoodie.consistency.check.max_checks
> Maximum number of consistency checks to perform, with exponential backoff.<br></br>
> **Default Value**: 6 (Optional)<br></br>
-> `Config Param: MAX_CONSISTENCY_CHECKS_PROP`<br></br>
+> `Config Param: MAX_CHECKS`<br></br>
> `Since Version: 0.5.0`<br></br>
> `Deprecated Version: 0.7.0`<br></br>
---
-> #### _hoodie.optimistic.consistency.guard.enable
-> Enable consistency guard, which optimistically assumes consistency is achieved after a certain time period.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: ENABLE_OPTIMISTIC_CONSISTENCY_GUARD_PROP`<br></br>
-> `Since Version: 0.6.0`<br></br>
+> #### hoodie.consistency.check.initial_interval_ms
+> Amount of time (in ms) to wait, before checking for consistency after an operation on storage.<br></br>
+> **Default Value**: 400 (Optional)<br></br>
+> `Config Param: INITIAL_CHECK_INTERVAL_MS`<br></br>
+> `Since Version: 0.5.0`<br></br>
+> `Deprecated Version: 0.7.0`<br></br>
---
@@ -1021,333 +1110,365 @@ The consistency guard related config options, to help talk to eventually consist
Configurations that control write behavior on Hudi tables. These can be directly passed down from even higher level frameworks (e.g Spark datasources, Flink sink) and utilities (e.g DeltaStreamer).
`Config Class`: org.apache.hudi.config.HoodieWriteConfig<br></br>
-> #### hoodie.embed.timeline.server.reuse.enabled
-> Controls whether the timeline server instance should be cached and reused across the JVM (across task lifecycles)to avoid startup costs. This should rarely be changed.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_REUSE_ENABLED`<br></br>
+> #### hoodie.combine.before.upsert
+> When upserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage. This should be turned off only if you are absolutely certain that there are no duplicates incoming, otherwise it can lead to duplicate keys and violate the uniqueness guarantees.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: COMBINE_BEFORE_UPSERT`<br></br>
---
-> #### hoodie.datasource.write.keygenerator.class
-> Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: KEYGENERATOR_CLASS_PROP`<br></br>
+> #### hoodie.write.markers.type
+> Marker type to use. Two modes are supported: - DIRECT: individual marker file corresponding to each data file is directly created by the writer. - TIMELINE_SERVER_BASED: marker operations are all handled at the timeline service which serves as a proxy. New marker entries are batch processed and stored in a limited number of underlying files for efficiency.<br></br>
+> **Default Value**: DIRECT (Optional)<br></br>
+> `Config Param: MARKERS_TYPE`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.bulkinsert.shuffle.parallelism
-> For large initial imports using bulk_insert operation, controls the parallelism to use for sort modes or custom partitioning donebefore writing records to the table.<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: BULKINSERT_PARALLELISM`<br></br>
+> #### hoodie.consistency.check.max_interval_ms
+> Max time to wait between successive attempts at performing consistency checks<br></br>
+> **Default Value**: 300000 (Optional)<br></br>
+> `Config Param: MAX_CONSISTENCY_CHECK_INTERVAL_MS`<br></br>
---
-> #### hoodie.rollback.parallelism
-> Parallelism for rollback of commits. Rollbacks perform delete of files or logging delete blocks to file groups on storage in parallel.<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: ROLLBACK_PARALLELISM`<br></br>
+> #### hoodie.embed.timeline.server.port
+> Port at which the timeline server listens for requests. When running embedded in each writer, it picks a free port and communicates to all the executors. This should rarely be changed.<br></br>
+> **Default Value**: 0 (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_PORT_NUM`<br></br>
---
-> #### hoodie.write.schema
-> The specified write schema. In most case, we do not need set this parameter, but for the case the write schema is not equal to the specified table schema, we can specify the write schema by this parameter. Used by MergeIntoHoodieTableCommand<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: WRITE_SCHEMA_PROP`<br></br>
+> #### hoodie.write.meta.key.prefixes
+> Comma separated metadata key prefixes to override from latest commit during overlapping commits via multi writing<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: WRITE_META_KEY_PREFIXES`<br></br>
---
-> #### hoodie.timeline.layout.version
-> Controls the layout of the timeline. Version 0 relied on renames, Version 1 (default) models the timeline as an immutable log relying only on atomic writes for object storage.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: TIMELINE_LAYOUT_VERSION`<br></br>
-> `Since Version: 0.5.1`<br></br>
+> #### hoodie.table.base.file.format
+> <br></br>
+> **Default Value**: PARQUET (Optional)<br></br>
+> `Config Param: BASE_FILE_FORMAT`<br></br>
---
> #### hoodie.avro.schema.validate
> Validate the schema used for the write against the latest schema, for backwards compatibility.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: AVRO_SCHEMA_VALIDATE`<br></br>
+> `Config Param: AVRO_SCHEMA_VALIDATE_ENABLE`<br></br>
---
-> #### hoodie.bulkinsert.sort.mode
-> Sorting modes to use for sorting records for bulk insert. This is user when user hoodie.bulkinsert.user.defined.partitioner.classis not configured. Available values are - GLOBAL_SORT: this ensures best file sizes, with lowest memory overhead at cost of sorting. PARTITION_SORT: Strikes a balance by only sorting within a partition, still keeping the memory overhead of writing lowest and best effort file sizing. NONE: No sorting. Fastest and matches `spark.write.parquet()` in terms of num [...]
-> **Default Value**: GLOBAL_SORT (Optional)<br></br>
-> `Config Param: BULKINSERT_SORT_MODE`<br></br>
+> #### hoodie.write.buffer.limit.bytes
+> Size of in-memory buffer used for parallelizing network reads and lake storage writes.<br></br>
+> **Default Value**: 4194304 (Optional)<br></br>
+> `Config Param: WRITE_BUFFER_LIMIT_BYTES_VALUE`<br></br>
---
-> #### hoodie.upsert.shuffle.parallelism
-> Parallelism to use for upsert operation on the table. Upserts can shuffle data to perform index lookups, file sizing, bin packing records optimallyinto file groups.<br></br>
+> #### hoodie.insert.shuffle.parallelism
+> Parallelism for inserting records into the table. Inserts can shuffle data before writing to tune file sizes and optimize the storage layout.<br></br>
> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: UPSERT_PARALLELISM`<br></br>
+> `Config Param: INSERT_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.embed.timeline.server.gzip
-> Controls whether gzip compression is used, for large responses from the timeline server, to improve latency.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_COMPRESS_OUTPUT`<br></br>
+> #### hoodie.embed.timeline.server.async
+> Controls whether or not, the requests to the timeline server are processed in asynchronous fashion, potentially improving throughput.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_USE_ASYNC_ENABLE`<br></br>
---
-> #### hoodie.fail.on.timeline.archiving
-> Timeline archiving removes older instants from the timeline, after each write operation, to minimize metadata overhead. Controls whether or not, the write should be failed as well, if such archiving fails.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: FAIL_ON_TIMELINE_ARCHIVING_ENABLED_PROP`<br></br>
+> #### hoodie.rollback.parallelism
+> Parallelism for rollback of commits. Rollbacks perform delete of files or logging delete blocks to file groups on storage in parallel.<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: ROLLBACK_PARALLELISM_VALUE`<br></br>
---
> #### hoodie.write.status.storage.level
> Write status objects hold metadata about a write (stats, errors), that is not yet committed to storage. This controls the how that information is cached for inspection by clients. We rarely expect this to be changed.<br></br>
> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
-> `Config Param: WRITE_STATUS_STORAGE_LEVEL`<br></br>
+> `Config Param: WRITE_STATUS_STORAGE_LEVEL_VALUE`<br></br>
+
+---
+
+> #### hoodie.writestatus.class
+> Subclass of org.apache.hudi.client.WriteStatus to be used to collect information about a write. Can be overridden to collection additional metrics/statistics about the data if needed.<br></br>
+> **Default Value**: org.apache.hudi.client.WriteStatus (Optional)<br></br>
+> `Config Param: WRITE_STATUS_CLASS_NAME`<br></br>
---
-> #### hoodie.table.base.file.format
-> <br></br>
-> **Default Value**: PARQUET (Optional)<br></br>
-> `Config Param: BASE_FILE_FORMAT`<br></br>
+> #### hoodie.base.path
+> Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: BASE_PATH`<br></br>
---
-> #### _.hoodie.allow.multi.write.on.same.instant
-> <br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: ALLOW_MULTI_WRITE_ON_SAME_INSTANT`<br></br>
+> #### hoodie.allow.empty.commit
+> Whether to allow generation of empty commits, even if no data was written in the commit. It's useful in cases where extra metadata needs to be published regardless e.g tracking source offsets when ingesting data<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: ALLOW_EMPTY_COMMIT`<br></br>
---
-> #### hoodie.embed.timeline.server.port
-> Port at which the timeline server listens for requests. When running embedded in each writer, it picks a free port and communicates to all the executors. This should rarely be changed.<br></br>
-> **Default Value**: 0 (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_PORT`<br></br>
+> #### hoodie.bulkinsert.user.defined.partitioner.class
+> If specified, this class will be used to re-partition records before they are bulk inserted. This can be used to sort, pack, cluster data optimally for common query patterns.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: BULKINSERT_USER_DEFINED_PARTITIONER_CLASS_NAME`<br></br>
---
-> #### hoodie.consistency.check.max_interval_ms
-> Max time to wait between successive attempts at performing consistency checks<br></br>
-> **Default Value**: 300000 (Optional)<br></br>
-> `Config Param: MAX_CONSISTENCY_CHECK_INTERVAL_MS_PROP`<br></br>
+> #### hoodie.table.name
+> Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: TBL_NAME`<br></br>
---
-> #### hoodie.write.concurrency.mode
-> Enable different concurrency modes. Options are SINGLE_WRITER: Only one active writer to the table. Maximizes throughputOPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table and exactly one of them succeed if a conflict (writes affect the same file group) is detected.<br></br>
-> **Default Value**: SINGLE_WRITER (Optional)<br></br>
-> `Config Param: WRITE_CONCURRENCY_MODE_PROP`<br></br>
+> #### hoodie.combine.before.delete
+> During delete operations, controls whether we should combine deletes (and potentially also upserts) before writing to storage.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: COMBINE_BEFORE_DELETE`<br></br>
---
-> #### hoodie.delete.shuffle.parallelism
-> Parallelism used for “delete” operation. Delete operations also performs shuffles, similar to upsert operation.<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: DELETE_PARALLELISM`<br></br>
+> #### hoodie.embed.timeline.server.threads
+> Number of threads to serve requests in the timeline server. By default, auto configured based on the number of underlying cores.<br></br>
+> **Default Value**: -1 (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_NUM_SERVER_THREADS`<br></br>
---
-> #### hoodie.embed.timeline.server
-> When true, spins up an instance of the timeline server (meta server that serves cached file listings, statistics),running on each writer's driver process, accepting requests during the write from executors.<br></br>
+> #### hoodie.fail.on.timeline.archiving
+> Timeline archiving removes older instants from the timeline, after each write operation, to minimize metadata overhead. Controls whether or not, the write should be failed as well, if such archiving fails.<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_ENABLED`<br></br>
+> `Config Param: FAIL_ON_TIMELINE_ARCHIVING_ENABLE`<br></br>
---
-> #### hoodie.base.path
-> Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br></br>
+> #### hoodie.datasource.write.keygenerator.class
+> Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BASE_PATH_PROP`<br></br>
+> `Config Param: KEYGENERATOR_CLASS_NAME`<br></br>
---
-> #### hoodie.avro.schema
-> Schema string representing the current write schema of the table. Hudi passes this to implementations of HoodieRecordPayload to convert incoming records to avro. This is also used as the write schema evolving records during an update.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: AVRO_SCHEMA`<br></br>
+> #### hoodie.combine.before.insert
+> When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: COMBINE_BEFORE_INSERT`<br></br>
---
-> #### hoodie.allow.empty.commit
-> Whether to allow generation of empty commits, even if no data was written in the commit. It's useful in cases where extra metadata needs to be published regardless e.g tracking source offsets when ingesting data<br></br>
+> #### hoodie.embed.timeline.server.gzip
+> Controls whether gzip compression is used, for large responses from the timeline server, to improve latency.<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: ALLOW_EMPTY_COMMIT`<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_COMPRESS_ENABLE`<br></br>
---
-> #### hoodie.table.name
-> Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: TABLE_NAME`<br></br>
+> #### hoodie.markers.timeline_server_based.batch.interval_ms
+> The batch interval in milliseconds for marker creation batch processing<br></br>
+> **Default Value**: 50 (Optional)<br></br>
+> `Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_INTERVAL_MS`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.embed.timeline.server.async
-> Controls whether or not, the requests to the timeline server are processed in asynchronous fashion, potentially improving throughput.<br></br>
+> #### hoodie.markers.timeline_server_based.batch.num_threads
+> Number of threads to use for batch processing marker creation requests at the timeline server<br></br>
+> **Default Value**: 20 (Optional)<br></br>
+> `Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_NUM_THREADS`<br></br>
+> `Since Version: 0.9.0`<br></br>
+
+---
+
+> #### _.hoodie.allow.multi.write.on.same.instant
+> <br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_USE_ASYNC`<br></br>
+> `Config Param: ALLOW_MULTI_WRITE_ON_SAME_INSTANT_ENABLE`<br></br>
---
-> #### hoodie.combine.before.upsert
-> When upserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage. This should be turned off only if you are absolutely certain that there are no duplicates incoming, otherwise it can lead to duplicate keys and violate the uniqueness guarantees.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: COMBINE_BEFORE_UPSERT_PROP`<br></br>
+> #### hoodie.datasource.write.payload.class
+> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: WRITE_PAYLOAD_CLASS_NAME`<br></br>
---
-> #### hoodie.embed.timeline.server.threads
-> Number of threads to serve requests in the timeline server. By default, auto configured based on the number of underlying cores.<br></br>
-> **Default Value**: -1 (Optional)<br></br>
-> `Config Param: EMBEDDED_TIMELINE_SERVER_THREADS`<br></br>
+> #### hoodie.bulkinsert.shuffle.parallelism
+> For large initial imports using bulk_insert operation, controls the parallelism to use for sort modes or custom partitioning donebefore writing records to the table.<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: BULKINSERT_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.consistency.check.max_checks
-> Maximum number of checks, for consistency of written data.<br></br>
-> **Default Value**: 7 (Optional)<br></br>
-> `Config Param: MAX_CONSISTENCY_CHECKS_PROP`<br></br>
+> #### hoodie.delete.shuffle.parallelism
+> Parallelism used for “delete” operation. Delete operations also performs shuffles, similar to upsert operation.<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: DELETE_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.combine.before.delete
-> During delete operations, controls whether we should combine deletes (and potentially also upserts) before writing to storage.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: COMBINE_BEFORE_DELETE_PROP`<br></br>
+> #### hoodie.consistency.check.max_checks
+> Maximum number of checks, for consistency of written data.<br></br>
+> **Default Value**: 7 (Optional)<br></br>
+> `Config Param: MAX_CONSISTENCY_CHECKS`<br></br>
---
> #### hoodie.datasource.write.keygenerator.type
> Easily configure one the built-in key generators, instead of specifying the key generator class.Currently supports SIMPLE, COMPLEX, TIMESTAMP, CUSTOM, NON_PARTITION, GLOBAL_DELETE<br></br>
> **Default Value**: SIMPLE (Optional)<br></br>
-> `Config Param: KEYGENERATOR_TYPE_PROP`<br></br>
+> `Config Param: KEYGENERATOR_TYPE`<br></br>
---
-> #### hoodie.write.buffer.limit.bytes
-> Size of in-memory buffer used for parallelizing network reads and lake storage writes.<br></br>
-> **Default Value**: 4194304 (Optional)<br></br>
-> `Config Param: WRITE_BUFFER_LIMIT_BYTES`<br></br>
+> #### hoodie.merge.allow.duplicate.on.inserts
+> When enabled, we allow duplicate keys even if inserts are routed to merge with an existing file (for ensuring file sizing). This is only relevant for insert operation, since upsert, delete operations will ensure unique key constraints are maintained.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE`<br></br>
---
-> #### hoodie.bulkinsert.user.defined.partitioner.class
-> If specified, this class will be used to re-partition records before they are bulk inserted. This can be used to sort, pack, cluster data optimally for common query patterns.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BULKINSERT_USER_DEFINED_PARTITIONER_CLASS`<br></br>
+> #### hoodie.embed.timeline.server.reuse.enabled
+> Controls whether the timeline server instance should be cached and reused across the JVM (across task lifecycles)to avoid startup costs. This should rarely be changed.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_REUSE_ENABLED`<br></br>
---
-> #### hoodie.client.heartbeat.interval_in_ms
-> Writers perform heartbeats to indicate liveness. Controls how often (in ms), such heartbeats are registered to lake storage.<br></br>
-> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: CLIENT_HEARTBEAT_INTERVAL_IN_MS_PROP`<br></br>
+> #### hoodie.datasource.write.precombine.field
+> Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br></br>
+> **Default Value**: ts (Optional)<br></br>
+> `Config Param: PRECOMBINE_FIELD_NAME`<br></br>
---
-> #### hoodie.avro.schema.external.transformation
-> When enabled, records in older schema are rewritten into newer schema during upsert,delete and background compaction,clustering operations.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION`<br></br>
+> #### hoodie.bulkinsert.sort.mode
+> Sorting modes to use for sorting records for bulk insert. This is user when user hoodie.bulkinsert.user.defined.partitioner.classis not configured. Available values are - GLOBAL_SORT: this ensures best file sizes, with lowest memory overhead at cost of sorting. PARTITION_SORT: Strikes a balance by only sorting within a partition, still keeping the memory overhead of writing lowest and best effort file sizing. NONE: No sorting. Fastest and matches `spark.write.parquet()` in terms of num [...]
+> **Default Value**: GLOBAL_SORT (Optional)<br></br>
+> `Config Param: BULK_INSERT_SORT_MODE`<br></br>
---
-> #### hoodie.client.heartbeat.tolerable.misses
-> Number of heartbeat misses, before a writer is deemed not alive and all pending writes are aborted.<br></br>
-> **Default Value**: 2 (Optional)<br></br>
-> `Config Param: CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES_PROP`<br></br>
+> #### hoodie.avro.schema
+> Schema string representing the current write schema of the table. Hudi passes this to implementations of HoodieRecordPayload to convert incoming records to avro. This is also used as the write schema evolving records during an update.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: AVRO_SCHEMA_STRING`<br></br>
---
> #### hoodie.auto.commit
> Controls whether a write operation should auto commit. This can be turned off to perform inspection of the uncommitted write before deciding to commit.<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: HOODIE_AUTO_COMMIT_PROP`<br></br>
+> `Config Param: AUTO_COMMIT_ENABLE`<br></br>
---
-> #### hoodie.combine.before.insert
-> When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: COMBINE_BEFORE_INSERT_PROP`<br></br>
+> #### hoodie.embed.timeline.server
+> When true, spins up an instance of the timeline server (meta server that serves cached file listings, statistics),running on each writer's driver process, accepting requests during the write from executors.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: EMBEDDED_TIMELINE_SERVER_ENABLE`<br></br>
---
-> #### hoodie.writestatus.class
-> Subclass of org.apache.hudi.client.WriteStatus to be used to collect information about a write. Can be overridden to collection additional metrics/statistics about the data if needed.<br></br>
-> **Default Value**: org.apache.hudi.client.WriteStatus (Optional)<br></br>
-> `Config Param: HOODIE_WRITE_STATUS_CLASS_PROP`<br></br>
+> #### hoodie.timeline.layout.version
+> Controls the layout of the timeline. Version 0 relied on renames, Version 1 (default) models the timeline as an immutable log relying only on atomic writes for object storage.<br></br>
+> **Default Value**: 1 (Optional)<br></br>
+> `Config Param: TIMELINE_LAYOUT_VERSION_NUM`<br></br>
+> `Since Version: 0.5.1`<br></br>
---
-> #### hoodie.markers.delete.parallelism
-> Determines the parallelism for deleting marker files, which are used to track all files (valid or invalid/partial) written during a write operation. Increase this value if delays are observed, with large batch writes.<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: MARKERS_DELETE_PARALLELISM`<br></br>
+> #### hoodie.upsert.shuffle.parallelism
+> Parallelism to use for upsert operation on the table. Upserts can shuffle data to perform index lookups, file sizing, bin packing records optimallyinto file groups.<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: UPSERT_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.consistency.check.initial_interval_ms
-> Initial time between successive attempts to ensure written data's metadata is consistent on storage. Grows with exponential backoff after the initial value.<br></br>
-> **Default Value**: 2000 (Optional)<br></br>
-> `Config Param: INITIAL_CONSISTENCY_CHECK_INTERVAL_MS_PROP`<br></br>
+> #### hoodie.write.schema
+> The specified write schema. In most case, we do not need set this parameter, but for the case the write schema is not equal to the specified table schema, we can specify the write schema by this parameter. Used by MergeIntoHoodieTableCommand<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: WRITE_SCHEMA`<br></br>
---
-> #### hoodie.merge.data.validation.enabled
-> When enabled, data validation checks are performed during merges to ensure expected number of records after merge operation.<br></br>
+> #### hoodie.rollback.using.markers
+> Enables a more efficient mechanism for rollbacks based on the marker files generated during the writes. Turned off by default.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: MERGE_DATA_VALIDATION_CHECK_ENABLED`<br></br>
+> `Config Param: ROLLBACK_USING_MARKERS_ENABLE`<br></br>
---
-> #### hoodie.datasource.write.payload.class
-> Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: WRITE_PAYLOAD_CLASS`<br></br>
+> #### hoodie.merge.data.validation.enabled
+> When enabled, data validation checks are performed during merges to ensure expected number of records after merge operation.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: MERGE_DATA_VALIDATION_CHECK_ENABLE`<br></br>
---
-> #### hoodie.insert.shuffle.parallelism
-> Parallelism for inserting records into the table. Inserts can shuffle data before writing to tune file sizes and optimize the storage layout.<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: INSERT_PARALLELISM`<br></br>
+> #### hoodie.client.heartbeat.tolerable.misses
+> Number of heartbeat misses, before a writer is deemed not alive and all pending writes are aborted.<br></br>
+> **Default Value**: 2 (Optional)<br></br>
+> `Config Param: CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES`<br></br>
---
-> #### hoodie.datasource.write.precombine.field
-> Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br></br>
-> **Default Value**: ts (Optional)<br></br>
-> `Config Param: PRECOMBINE_FIELD_PROP`<br></br>
+> #### hoodie.write.concurrency.mode
+> Enable different concurrency modes. Options are SINGLE_WRITER: Only one active writer to the table. Maximizes throughputOPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table and exactly one of them succeed if a conflict (writes affect the same file group) is detected.<br></br>
+> **Default Value**: SINGLE_WRITER (Optional)<br></br>
+> `Config Param: WRITE_CONCURRENCY_MODE`<br></br>
---
-> #### hoodie.write.meta.key.prefixes
-> Comma separated metadata key prefixes to override from latest commit during overlapping commits via multi writing<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: WRITE_META_KEY_PREFIXES_PROP`<br></br>
+> #### hoodie.markers.delete.parallelism
+> Determines the parallelism for deleting marker files, which are used to track all files (valid or invalid/partial) written during a write operation. Increase this value if delays are observed, with large batch writes.<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: MARKERS_DELETE_PARALLELISM_VALUE`<br></br>
---
> #### hoodie.finalize.write.parallelism
> Parallelism for the write finalization internal operation, which involves removing any partially written files from lake storage, before committing the write. Reduce this value, if the high number of tasks incur delays for smaller tables or low latency writes.<br></br>
> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: FINALIZE_WRITE_PARALLELISM`<br></br>
+> `Config Param: FINALIZE_WRITE_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.merge.allow.duplicate.on.inserts
-> When enabled, we allow duplicate keys even if inserts are routed to merge with an existing file (for ensuring file sizing). This is only relevant for insert operation, since upsert, delete operations will ensure unique key constraints are maintained.<br></br>
+> #### hoodie.client.heartbeat.interval_in_ms
+> Writers perform heartbeats to indicate liveness. Controls how often (in ms), such heartbeats are registered to lake storage.<br></br>
+> **Default Value**: 60000 (Optional)<br></br>
+> `Config Param: CLIENT_HEARTBEAT_INTERVAL_IN_MS`<br></br>
+
+---
+
+> #### hoodie.allow.operation.metadata.field
+> Whether to include '_hoodie_operation' in the metadata fields. Once enabled, all the changes of a record are persisted to the delta log directly without merge<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: MERGE_ALLOW_DUPLICATE_ON_INSERTS`<br></br>
+> `Config Param: ALLOW_OPERATION_METADATA_FIELD`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.rollback.using.markers
-> Enables a more efficient mechanism for rollbacks based on the marker files generated during the writes. Turned off by default.<br></br>
+> #### hoodie.consistency.check.initial_interval_ms
+> Initial time between successive attempts to ensure written data's metadata is consistent on storage. Grows with exponential backoff after the initial value.<br></br>
+> **Default Value**: 2000 (Optional)<br></br>
+> `Config Param: INITIAL_CONSISTENCY_CHECK_INTERVAL_MS`<br></br>
+
+---
+
+> #### hoodie.avro.schema.external.transformation
+> When enabled, records in older schema are rewritten into newer schema during upsert,delete and background compaction,clustering operations.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: ROLLBACK_USING_MARKERS`<br></br>
+> `Config Param: AVRO_EXTERNAL_SCHEMA_TRANSFORMATION_ENABLE`<br></br>
---
@@ -1359,16 +1480,7 @@ Hudi maintains keys (record key + partition path) for uniquely identifying a par
> #### hoodie.datasource.write.partitionpath.urlencode
> Should we url encode the partition path value, before creating the folder structure.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: URL_ENCODE_PARTITIONING_OPT_KEY`<br></br>
-
----
-
-> #### hoodie.datasource.write.recordkey.field
-> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
-Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using
-the dot notation eg: `a.b.c`<br></br>
-> **Default Value**: uuid (Optional)<br></br>
-> `Config Param: RECORDKEY_FIELD_OPT_KEY`<br></br>
+> `Config Param: URL_ENCODE_PARTITIONING`<br></br>
---
@@ -1377,14 +1489,23 @@ the dot notation eg: `a.b.c`<br></br>
If set true, the names of partition folders follow <partition_column_name>=<partition_value> format.
By default false (the names of partition folders are only partition values)<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HIVE_STYLE_PARTITIONING_OPT_KEY`<br></br>
+> `Config Param: HIVE_STYLE_PARTITIONING_ENABLE`<br></br>
---
> #### hoodie.datasource.write.partitionpath.field
> Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value ontained by invoking .toString()<br></br>
> **Default Value**: partitionpath (Optional)<br></br>
-> `Config Param: PARTITIONPATH_FIELD_OPT_KEY`<br></br>
+> `Config Param: PARTITIONPATH_FIELD_NAME`<br></br>
+
+---
+
+> #### hoodie.datasource.write.recordkey.field
+> Record key field. Value to be used as the `recordKey` component of `HoodieKey`.
+Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using
+the dot notation eg: `a.b.c`<br></br>
+> **Default Value**: uuid (Optional)<br></br>
+> `Config Param: RECORDKEY_FIELD_NAME`<br></br>
---
@@ -1393,38 +1514,52 @@ By default false (the names of partition folders are only partition values)<br><
Configurations that control indexing behavior (when HBase based indexing is enabled), which tags incoming records as either inserts or updates to older records.
`Config Class`: org.apache.hudi.config.HoodieHBaseIndexConfig<br></br>
-> #### hoodie.index.hbase.qps.fraction
-> Property to set the fraction of the global share of QPS that should be allocated to this job. Let's say there are 3 jobs which have input size in terms of number of rows required for HbaseIndexing as x, 2x, 3x respectively. Then this fraction for the jobs would be (0.17) 1/6, 0.33 (2/6) and 0.5 (3/6) respectively. Default is 50%, which means a total of 2 jobs can run using HbaseIndex without overwhelming Region Servers.<br></br>
-> **Default Value**: 0.5 (Optional)<br></br>
-> `Config Param: HBASE_QPS_FRACTION_PROP`<br></br>
+> #### hoodie.index.hbase.zkport
+> Only applies if index type is HBASE. HBase ZK Quorum port to connect to<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: ZKPORT`<br></br>
---
-> #### hoodie.index.hbase.zknode.path
-> Only applies if index type is HBASE. This is the root znode that will contain all the znodes created/used by HBase<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_ZK_ZNODEPARENT`<br></br>
+> #### hoodie.hbase.index.update.partition.path
+> Only applies if index type is HBASE. When an already existing record is upserted to a new partition compared to whats in storage, this config when set, will delete old record in old paritition and will insert it as new record in new partition.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: UPDATE_PARTITION_PATH_ENABLE`<br></br>
---
-> #### hoodie.index.hbase.zkpath.qps_root
-> chroot in zookeeper, to use for all qps allocation co-ordination.<br></br>
-> **Default Value**: /QPS_ROOT (Optional)<br></br>
-> `Config Param: HBASE_ZK_PATH_QPS_ROOT`<br></br>
+> #### hoodie.index.hbase.qps.allocator.class
+> Property to set which implementation of HBase QPS resource allocator to be used, whichcontrols the batching rate dynamically.<br></br>
+> **Default Value**: org.apache.hudi.index.hbase.DefaultHBaseQPSResourceAllocator (Optional)<br></br>
+> `Config Param: QPS_ALLOCATOR_CLASS_NAME`<br></br>
---
-> #### hoodie.index.hbase.put.batch.size
-> Controls the batch size for performing puts against HBase. Batching improves throughput, by saving round trips.<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: HBASE_PUT_BATCH_SIZE_PROP`<br></br>
+> #### hoodie.index.hbase.put.batch.size.autocompute
+> Property to set to enable auto computation of put batch size<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: PUT_BATCH_SIZE_AUTO_COMPUTE`<br></br>
---
-> #### hoodie.index.hbase.dynamic_qps
-> Property to decide if HBASE_QPS_FRACTION_PROP is dynamically calculated based on write volume.<br></br>
+> #### hoodie.index.hbase.rollback.sync
+> When set to true, the rollback method will delete the last failed task index. The default value is false. Because deleting the index will add extra load on the Hbase cluster for each rollback<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: HOODIE_INDEX_COMPUTE_QPS_DYNAMICALLY`<br></br>
+> `Config Param: ROLLBACK_SYNC_ENABLE`<br></br>
+
+---
+
+> #### hoodie.index.hbase.get.batch.size
+> Controls the batch size for performing gets against HBase. Batching improves throughput, by saving round trips.<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: GET_BATCH_SIZE`<br></br>
+
+---
+
+> #### hoodie.index.hbase.zkpath.qps_root
+> chroot in zookeeper, to use for all qps allocation co-ordination.<br></br>
+> **Default Value**: /QPS_ROOT (Optional)<br></br>
+> `Config Param: ZKPATH_QPS_ROOT`<br></br>
---
@@ -1434,112 +1569,98 @@ Configurations that control indexing behavior (when HBase based indexing is enab
value based on global indexing throughput needs and most importantly, how much the HBase installation in use is
able to tolerate without Region Servers going down.<br></br>
> **Default Value**: 1000 (Optional)<br></br>
-> `Config Param: HBASE_MAX_QPS_PER_REGION_SERVER_PROP`<br></br>
-
----
-
-> #### hoodie.index.hbase.zk.session_timeout_ms
-> Session timeout value to use for Zookeeper failure detection, for the HBase client.Lower this value, if you want to fail faster.<br></br>
-> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: HOODIE_INDEX_HBASE_ZK_SESSION_TIMEOUT_MS`<br></br>
+> `Config Param: MAX_QPS_PER_REGION_SERVER`<br></br>
---
-> #### hoodie.index.hbase.sleep.ms.for.get.batch
-> <br></br>
+> #### hoodie.index.hbase.max.qps.fraction
+> Maximum for HBASE_QPS_FRACTION_PROP to stabilize skewed write workloads<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_SLEEP_MS_GET_BATCH_PROP`<br></br>
+> `Config Param: MAX_QPS_FRACTION`<br></br>
---
> #### hoodie.index.hbase.min.qps.fraction
> Minimum for HBASE_QPS_FRACTION_PROP to stabilize skewed write workloads<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_MIN_QPS_FRACTION_PROP`<br></br>
+> `Config Param: MIN_QPS_FRACTION`<br></br>
---
-> #### hoodie.index.hbase.desired_puts_time_in_secs
-> <br></br>
-> **Default Value**: 600 (Optional)<br></br>
-> `Config Param: HOODIE_INDEX_DESIRED_PUTS_TIME_IN_SECS`<br></br>
+> #### hoodie.index.hbase.zk.connection_timeout_ms
+> Timeout to use for establishing connection with zookeeper, from HBase client.<br></br>
+> **Default Value**: 15000 (Optional)<br></br>
+> `Config Param: ZK_CONNECTION_TIMEOUT_MS`<br></br>
---
-> #### hoodie.hbase.index.update.partition.path
-> Only applies if index type is HBASE. When an already existing record is upserted to a new partition compared to whats in storage, this config when set, will delete old record in old paritition and will insert it as new record in new partition.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HBASE_INDEX_UPDATE_PARTITION_PATH`<br></br>
+> #### hoodie.index.hbase.table
+> Only applies if index type is HBASE. HBase Table name to use as the index. Hudi stores the row_key and [partition_path, fileID, commitTime] mapping in the table<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: TABLENAME`<br></br>
---
-> #### hoodie.index.hbase.qps.allocator.class
-> Property to set which implementation of HBase QPS resource allocator to be used, whichcontrols the batching rate dynamically.<br></br>
-> **Default Value**: org.apache.hudi.index.hbase.DefaultHBaseQPSResourceAllocator (Optional)<br></br>
-> `Config Param: HBASE_INDEX_QPS_ALLOCATOR_CLASS`<br></br>
+> #### hoodie.index.hbase.dynamic_qps
+> Property to decide if HBASE_QPS_FRACTION_PROP is dynamically calculated based on write volume.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: COMPUTE_QPS_DYNAMICALLY`<br></br>
---
-> #### hoodie.index.hbase.sleep.ms.for.put.batch
-> <br></br>
+> #### hoodie.index.hbase.zknode.path
+> Only applies if index type is HBASE. This is the root znode that will contain all the znodes created/used by HBase<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_SLEEP_MS_PUT_BATCH_PROP`<br></br>
-
----
-
-> #### hoodie.index.hbase.rollback.sync
-> When set to true, the rollback method will delete the last failed task index. The default value is false. Because deleting the index will add extra load on the Hbase cluster for each rollback<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HBASE_INDEX_ROLLBACK_SYNC`<br></br>
+> `Config Param: ZK_NODE_PATH`<br></br>
---
> #### hoodie.index.hbase.zkquorum
> Only applies if index type is HBASE. HBase ZK Quorum url to connect to<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_ZKQUORUM_PROP`<br></br>
+> `Config Param: ZKQUORUM`<br></br>
---
-> #### hoodie.index.hbase.get.batch.size
-> Controls the batch size for performing gets against HBase. Batching improves throughput, by saving round trips.<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: HBASE_GET_BATCH_SIZE_PROP`<br></br>
+> #### hoodie.index.hbase.qps.fraction
+> Property to set the fraction of the global share of QPS that should be allocated to this job. Let's say there are 3 jobs which have input size in terms of number of rows required for HbaseIndexing as x, 2x, 3x respectively. Then this fraction for the jobs would be (0.17) 1/6, 0.33 (2/6) and 0.5 (3/6) respectively. Default is 50%, which means a total of 2 jobs can run using HbaseIndex without overwhelming Region Servers.<br></br>
+> **Default Value**: 0.5 (Optional)<br></br>
+> `Config Param: QPS_FRACTION`<br></br>
---
-> #### hoodie.index.hbase.zk.connection_timeout_ms
-> Timeout to use for establishing connection with zookeeper, from HBase client.<br></br>
-> **Default Value**: 15000 (Optional)<br></br>
-> `Config Param: HOODIE_INDEX_HBASE_ZK_CONNECTION_TIMEOUT_MS`<br></br>
+> #### hoodie.index.hbase.zk.session_timeout_ms
+> Session timeout value to use for Zookeeper failure detection, for the HBase client.Lower this value, if you want to fail faster.<br></br>
+> **Default Value**: 60000 (Optional)<br></br>
+> `Config Param: ZK_SESSION_TIMEOUT_MS`<br></br>
---
-> #### hoodie.index.hbase.put.batch.size.autocompute
-> Property to set to enable auto computation of put batch size<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HBASE_PUT_BATCH_SIZE_AUTO_COMPUTE_PROP`<br></br>
+> #### hoodie.index.hbase.put.batch.size
+> Controls the batch size for performing puts against HBase. Batching improves throughput, by saving round trips.<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: PUT_BATCH_SIZE`<br></br>
---
-> #### hoodie.index.hbase.zkport
-> Only applies if index type is HBASE. HBase ZK Quorum port to connect to<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_ZKPORT_PROP`<br></br>
+> #### hoodie.index.hbase.desired_puts_time_in_secs
+> <br></br>
+> **Default Value**: 600 (Optional)<br></br>
+> `Config Param: DESIRED_PUTS_TIME_IN_SECONDS`<br></br>
---
-> #### hoodie.index.hbase.max.qps.fraction
-> Maximum for HBASE_QPS_FRACTION_PROP to stabilize skewed write workloads<br></br>
+> #### hoodie.index.hbase.sleep.ms.for.put.batch
+> <br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_MAX_QPS_FRACTION_PROP`<br></br>
+> `Config Param: SLEEP_MS_FOR_PUT_BATCH`<br></br>
---
-> #### hoodie.index.hbase.table
-> Only applies if index type is HBASE. HBase Table name to use as the index. Hudi stores the row_key and [partition_path, fileID, commitTime] mapping in the table<br></br>
+> #### hoodie.index.hbase.sleep.ms.for.get.batch
+> <br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HBASE_TABLENAME_PROP`<br></br>
+> `Config Param: SLEEP_MS_FOR_GET_BATCH`<br></br>
---
@@ -1548,18 +1669,10 @@ Configurations that control indexing behavior (when HBase based indexing is enab
Controls callback behavior into HTTP endpoints, to push notifications on commits on hudi tables.
`Config Class`: org.apache.hudi.config.HoodieWriteCommitCallbackConfig<br></br>
-> #### hoodie.write.commit.callback.http.api.key
-> Http callback API key. hudi_write_commit_http_callback by default<br></br>
-> **Default Value**: hudi_write_commit_http_callback (Optional)<br></br>
-> `Config Param: CALLBACK_HTTP_API_KEY`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
> #### hoodie.write.commit.callback.on
> Turn commit callback on/off. off by default.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: CALLBACK_ON`<br></br>
+> `Config Param: TURN_CALLBACK_ON`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -1567,7 +1680,7 @@ Controls callback behavior into HTTP endpoints, to push notifications on commit
> #### hoodie.write.commit.callback.http.url
> Callback host to be sent along with callback messages<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CALLBACK_HTTP_URL_PROP`<br></br>
+> `Config Param: CALLBACK_HTTP_URL`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -1575,7 +1688,7 @@ Controls callback behavior into HTTP endpoints, to push notifications on commit
> #### hoodie.write.commit.callback.http.timeout.seconds
> Callback timeout in seconds. 3 by default<br></br>
> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: CALLBACK_HTTP_TIMEOUT_SECONDS`<br></br>
+> `Config Param: CALLBACK_HTTP_TIMEOUT_IN_SECONDS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -1583,7 +1696,15 @@ Controls callback behavior into HTTP endpoints, to push notifications on commit
> #### hoodie.write.commit.callback.class
> Full path of callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default<br></br>
> **Default Value**: org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback (Optional)<br></br>
-> `Config Param: CALLBACK_CLASS_PROP`<br></br>
+> `Config Param: CALLBACK_CLASS_NAME`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.write.commit.callback.http.api.key
+> Http callback API key. hudi_write_commit_http_callback by default<br></br>
+> **Default Value**: hudi_write_commit_http_callback (Optional)<br></br>
+> `Config Param: CALLBACK_HTTP_API_KEY_VALUE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -1593,26 +1714,18 @@ Controls callback behavior into HTTP endpoints, to push notifications on commit
Controls notifications sent to Kafka, on events happening to a hudi table.
`Config Class`: org.apache.hudi.utilities.callback.kafka.HoodieWriteCommitKafkaCallbackConfig<br></br>
-> #### hoodie.write.commit.callback.kafka.acks
-> kafka acks level, all by default to ensure strong durability.<br></br>
-> **Default Value**: all (Optional)<br></br>
-> `Config Param: CALLBACK_KAFKA_ACKS`<br></br>
-> `Since Version: 0.7.0`<br></br>
-
----
-
-> #### hoodie.write.commit.callback.kafka.bootstrap.servers
-> Bootstrap servers of kafka cluster, to be used for publishing commit metadata.<br></br>
+> #### hoodie.write.commit.callback.kafka.topic
+> Kafka topic name to publish timeline activity into.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CALLBACK_KAFKA_BOOTSTRAP_SERVERS`<br></br>
+> `Config Param: TOPIC`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.write.commit.callback.kafka.topic
-> Kafka topic name to publish timeline activity into.<br></br>
+> #### hoodie.write.commit.callback.kafka.partition
+> It may be desirable to serialize all changes into a single Kafka partition for providing strict ordering. By default, Kafka messages are keyed by table name, which guarantees ordering at the table level, but not globally (or when new partitions are added)<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CALLBACK_KAFKA_TOPIC`<br></br>
+> `Config Param: PARTITION`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -1620,15 +1733,23 @@ Controls notifications sent to Kafka, on events happening to a hudi table.
> #### hoodie.write.commit.callback.kafka.retries
> Times to retry the produce. 3 by default<br></br>
> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: CALLBACK_KAFKA_RETRIES`<br></br>
+> `Config Param: RETRIES`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.write.commit.callback.kafka.partition
-> It may be desirable to serialize all changes into a single Kafka partition for providing strict ordering. By default, Kafka messages are keyed by table name, which guarantees ordering at the table level, but not globally (or when new partitions are added)<br></br>
+> #### hoodie.write.commit.callback.kafka.acks
+> kafka acks level, all by default to ensure strong durability.<br></br>
+> **Default Value**: all (Optional)<br></br>
+> `Config Param: ACKS`<br></br>
+> `Since Version: 0.7.0`<br></br>
+
+---
+
+> #### hoodie.write.commit.callback.kafka.bootstrap.servers
+> Bootstrap servers of kafka cluster, to be used for publishing commit metadata.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CALLBACK_KAFKA_PARTITION`<br></br>
+> `Config Param: BOOTSTRAP_SERVERS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -1638,34 +1759,26 @@ Controls notifications sent to Kafka, on events happening to a hudi table.
Configs that control locking mechanisms required for concurrency control between writers to a Hudi table. Concurrency between Hudi's own table services are auto managed internally.
`Config Class`: org.apache.hudi.config.HoodieLockConfig<br></br>
-> #### hoodie.write.lock.zookeeper.url
-> Zookeeper URL to connect to.<br></br>
+> #### hoodie.write.lock.zookeeper.base_path
+> The base path on Zookeeper under which to create lock related ZNodes. This should be same for all concurrent writers to the same table<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ZK_CONNECT_URL_PROP`<br></br>
-> `Since Version: 0.8.0`<br></br>
-
----
-
-> #### hoodie.write.lock.max_wait_time_ms_between_retry
-> Maximum amount of time to wait between retries by lock provider client. This bounds the maximum delay from the exponential backoff. Currently used by ZK based lock provider only.<br></br>
-> **Default Value**: 5000 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_RETRY_MAX_WAIT_TIME_IN_MILLIS_PROP`<br></br>
+> `Config Param: ZK_BASE_PATH`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.filesystem.path
-> For DFS based lock providers, path to store the locks under.<br></br>
+> #### hoodie.write.lock.zookeeper.lock_key
+> Key name under base_path at which to create a ZNode and acquire lock. Final path on zk will look like base_path/lock_key. We recommend setting this to the table name<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: FILESYSTEM_LOCK_PATH_PROP`<br></br>
+> `Config Param: ZK_LOCK_KEY`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.hivemetastore.uris
-> For Hive based lock provider, the Hive metastore URI to acquire locks against.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_METASTORE_URI_PROP`<br></br>
+> #### hoodie.write.lock.client.num_retries
+> Maximum number of times to retry to acquire lock additionally from the lock manager.<br></br>
+> **Default Value**: 0 (Optional)<br></br>
+> `Config Param: LOCK_ACQUIRE_CLIENT_NUM_RETRIES`<br></br>
> `Since Version: 0.8.0`<br></br>
---
@@ -1673,7 +1786,15 @@ Configs that control locking mechanisms required for concurrency control betwee
> #### hoodie.write.lock.wait_time_ms_between_retry
> Initial amount of time to wait between retries to acquire locks, subsequent retries will exponentially backoff.<br></br>
> **Default Value**: 5000 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP`<br></br>
+> `Config Param: LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS`<br></br>
+> `Since Version: 0.8.0`<br></br>
+
+---
+
+> #### hoodie.write.lock.num_retries
+> Maximum number of times to retry lock acquire, at each lock provider<br></br>
+> **Default Value**: 3 (Optional)<br></br>
+> `Config Param: LOCK_ACQUIRE_NUM_RETRIES`<br></br>
> `Since Version: 0.8.0`<br></br>
---
@@ -1681,103 +1802,103 @@ Configs that control locking mechanisms required for concurrency control betwee
> #### hoodie.write.lock.wait_time_ms
> Timeout in ms, to wait on an individual lock acquire() call, at the lock provider.<br></br>
> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP`<br></br>
+> `Config Param: LOCK_ACQUIRE_WAIT_TIMEOUT_MS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.conflict.resolution.strategy
-> Lock provider class name, this should be subclass of org.apache.hudi.client.transaction.ConflictResolutionStrategy<br></br>
-> **Default Value**: org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy (Optional)<br></br>
-> `Config Param: WRITE_CONFLICT_RESOLUTION_STRATEGY_CLASS_PROP`<br></br>
+> #### hoodie.write.lock.zookeeper.connection_timeout_ms
+> Timeout in ms, to wait for establishing connection with Zookeeper.<br></br>
+> **Default Value**: 15000 (Optional)<br></br>
+> `Config Param: ZK_CONNECTION_TIMEOUT_MS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.num_retries
-> Maximum number of times to retry lock acquire, at each lock provider<br></br>
-> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_NUM_RETRIES_PROP`<br></br>
+> #### hoodie.write.lock.zookeeper.port
+> Zookeeper port to connect to.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: ZK_PORT`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.connection_timeout_ms
-> Timeout in ms, to wait for establishing connection with Zookeeper.<br></br>
-> **Default Value**: 15000 (Optional)<br></br>
-> `Config Param: ZK_CONNECTION_TIMEOUT_MS_PROP`<br></br>
+> #### hoodie.write.lock.hivemetastore.table
+> For Hive based lock provider, the Hive table to acquire lock against<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_TABLE_NAME`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.client.num_retries
-> Maximum number of times to retry to acquire lock additionally from the lock manager.<br></br>
-> **Default Value**: 0 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_CLIENT_NUM_RETRIES_PROP`<br></br>
+> #### hoodie.write.lock.zookeeper.url
+> Zookeeper URL to connect to.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: ZK_CONNECT_URL`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.hivemetastore.database
-> For Hive based lock provider, the Hive database to acquire lock against<br></br>
+> #### hoodie.write.lock.filesystem.path
+> For DFS based lock providers, path to store the locks under.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_DATABASE_NAME_PROP`<br></br>
+> `Config Param: FILESYSTEM_LOCK_PATH`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.port
-> Zookeeper port to connect to.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ZK_PORT_PROP`<br></br>
+> #### hoodie.write.lock.provider
+> Lock provider class name, user can provide their own implementation of LockProvider which should be subclass of org.apache.hudi.common.lock.LockProvider<br></br>
+> **Default Value**: org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider (Optional)<br></br>
+> `Config Param: LOCK_PROVIDER_CLASS_NAME`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.client.wait_time_ms_between_retry
-> Amount of time to wait between retries on the lock provider by the lock manager<br></br>
-> **Default Value**: 10000 (Optional)<br></br>
-> `Config Param: LOCK_ACQUIRE_CLIENT_RETRY_WAIT_TIME_IN_MILLIS_PROP`<br></br>
+> #### hoodie.write.lock.zookeeper.session_timeout_ms
+> Timeout in ms, to wait after losing connection to ZooKeeper, before the session is expired<br></br>
+> **Default Value**: 60000 (Optional)<br></br>
+> `Config Param: ZK_SESSION_TIMEOUT_MS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.lock_key
-> Key name under base_path at which to create a ZNode and acquire lock. Final path on zk will look like base_path/lock_key. We recommend setting this to the table name<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ZK_LOCK_KEY_PROP`<br></br>
+> #### hoodie.write.lock.conflict.resolution.strategy
+> Lock provider class name, this should be subclass of org.apache.hudi.client.transaction.ConflictResolutionStrategy<br></br>
+> **Default Value**: org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy (Optional)<br></br>
+> `Config Param: WRITE_CONFLICT_RESOLUTION_STRATEGY_CLASS_NAME`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.base_path
-> The base path on Zookeeper under which to create lock related ZNodes. This should be same for all concurrent writers to the same table<br></br>
+> #### hoodie.write.lock.hivemetastore.database
+> For Hive based lock provider, the Hive database to acquire lock against<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ZK_BASE_PATH_PROP`<br></br>
+> `Config Param: HIVE_DATABASE_NAME`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.zookeeper.session_timeout_ms
-> Timeout in ms, to wait after losing connection to ZooKeeper, before the session is expired<br></br>
-> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: ZK_SESSION_TIMEOUT_MS_PROP`<br></br>
+> #### hoodie.write.lock.hivemetastore.uris
+> For Hive based lock provider, the Hive metastore URI to acquire locks against.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: HIVE_METASTORE_URI`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.hivemetastore.table
-> For Hive based lock provider, the Hive table to acquire lock against<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HIVE_TABLE_NAME_PROP`<br></br>
+> #### hoodie.write.lock.max_wait_time_ms_between_retry
+> Maximum amount of time to wait between retries by lock provider client. This bounds the maximum delay from the exponential backoff. Currently used by ZK based lock provider only.<br></br>
+> **Default Value**: 5000 (Optional)<br></br>
+> `Config Param: LOCK_ACQUIRE_RETRY_MAX_WAIT_TIME_IN_MILLIS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
-> #### hoodie.write.lock.provider
-> Lock provider class name, user can provide their own implementation of LockProvider which should be subclass of org.apache.hudi.common.lock.LockProvider<br></br>
-> **Default Value**: org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider (Optional)<br></br>
-> `Config Param: LOCK_PROVIDER_CLASS_PROP`<br></br>
+> #### hoodie.write.lock.client.wait_time_ms_between_retry
+> Amount of time to wait between retries on the lock provider by the lock manager<br></br>
+> **Default Value**: 10000 (Optional)<br></br>
+> `Config Param: LOCK_ACQUIRE_CLIENT_RETRY_WAIT_TIME_IN_MILLIS`<br></br>
> `Since Version: 0.8.0`<br></br>
---
@@ -1787,192 +1908,192 @@ Configs that control locking mechanisms required for concurrency control betwee
Configurations that control compaction (merging of log files onto a new base files) as well as cleaning (reclamation of older/unused file groups/slices).
`Config Class`: org.apache.hudi.config.HoodieCompactionConfig<br></br>
-> #### hoodie.compact.inline.trigger.strategy
-> Controls how compaction scheduling is triggered, by time or num delta commits or combination of both. Valid options: NUM_COMMITS,TIME_ELAPSED,NUM_AND_TIME,NUM_OR_TIME<br></br>
-> **Default Value**: NUM_COMMITS (Optional)<br></br>
-> `Config Param: INLINE_COMPACT_TRIGGER_STRATEGY_PROP`<br></br>
+> #### hoodie.compaction.payload.class
+> This needs to be same as class used during insert/upserts. Just like writing, compaction also uses the record payload class to merge records in the log against each other, merge again with the base file and produce the final record to be written after compaction.<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: PAYLOAD_CLASS_NAME`<br></br>
---
-> #### hoodie.cleaner.fileversions.retained
-> When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning.<br></br>
-> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: CLEANER_FILE_VERSIONS_RETAINED_PROP`<br></br>
+> #### hoodie.copyonwrite.record.size.estimate
+> The average record size. If not explicitly specified, hudi will compute the record size estimate compute dynamically based on commit metadata. This is critical in computing the insert parallelism and bin-packing inserts into small files.<br></br>
+> **Default Value**: 1024 (Optional)<br></br>
+> `Config Param: COPY_ON_WRITE_RECORD_SIZE_ESTIMATE`<br></br>
---
-> #### hoodie.cleaner.policy.failed.writes
-> Cleaning policy for failed writes to be used. Hudi will delete any files written by failed writes to re-claim space. Choose to perform this rollback of failed writes eagerly before every writer starts (only supported for single writer) or lazily by the cleaner (required for multi-writers)<br></br>
-> **Default Value**: EAGER (Optional)<br></br>
-> `Config Param: FAILED_WRITES_CLEANER_POLICY_PROP`<br></br>
+> #### hoodie.cleaner.policy
+> Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. By default, cleaner spares the file slices written by the last N commits, determined by hoodie.cleaner.commits.retained Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query execution time<br></br>
+> **Default Value**: KEEP_LATEST_COMMITS (Optional)<br></br>
+> `Config Param: CLEANER_POLICY`<br></br>
---
-> #### hoodie.parquet.small.file.limit
-> During upsert operation, we opportunistically expand existing small files on storage, instead of writing new files, to keep number of files to an optimum. This config sets the file size limit below which a file on storage becomes a candidate to be selected as such a `small file`. By default, treat any file <= 100MB as a small file.<br></br>
-> **Default Value**: 104857600 (Optional)<br></br>
-> `Config Param: PARQUET_SMALL_FILE_LIMIT_BYTES`<br></br>
+> #### hoodie.compact.inline.max.delta.seconds
+> Number of elapsed seconds after the last compaction, before scheduling a new one.<br></br>
+> **Default Value**: 3600 (Optional)<br></br>
+> `Config Param: INLINE_COMPACT_TIME_DELTA_SECONDS`<br></br>
---
-> #### hoodie.keep.max.commits
-> Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows.This config controls the maximum number of instants to retain in the active timeline. <br></br>
-> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: MAX_COMMITS_TO_KEEP_PROP`<br></br>
+> #### hoodie.cleaner.delete.bootstrap.base.file
+> When set to true, cleaner also deletes the bootstrap base file when it's skeleton base file is cleaned. Turn this to true, if you want to ensure the bootstrap dataset storage is reclaimed over time, as the table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap base files are also physically deleted, to comply with data privacy enforcement processes.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: CLEANER_BOOTSTRAP_BASE_FILE_ENABLE`<br></br>
---
-> #### hoodie.compaction.lazy.block.read
-> When merging the delta log files, this config helps to choose whether the log blocks should be read lazily or not. Choose true to use lazy block reading (low memory usage, but incurs seeks to each block header) or false for immediate block read (higher memory usage)<br></br>
+> #### hoodie.cleaner.commits.retained
+> Number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br></br>
+> **Default Value**: 10 (Optional)<br></br>
+> `Config Param: CLEANER_COMMITS_RETAINED`<br></br>
+
+---
+
+> #### hoodie.cleaner.policy.failed.writes
+> Cleaning policy for failed writes to be used. Hudi will delete any files written by failed writes to re-claim space. Choose to perform this rollback of failed writes eagerly before every writer starts (only supported for single writer) or lazily by the cleaner (required for multi-writers)<br></br>
+> **Default Value**: EAGER (Optional)<br></br>
+> `Config Param: FAILED_WRITES_CLEANER_POLICY`<br></br>
+
+---
+
+> #### hoodie.clean.async
+> Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: COMPACTION_LAZY_BLOCK_READ_ENABLED_PROP`<br></br>
+> `Config Param: ASYNC_CLEAN`<br></br>
---
> #### hoodie.clean.automatic
> When enabled, the cleaner table service is invoked immediately after each commit, to delete older file slices. It's recommended to enable this, to ensure metadata and data storage growth is bounded.<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: AUTO_CLEAN_PROP`<br></br>
+> `Config Param: AUTO_CLEAN`<br></br>
---
> #### hoodie.commits.archival.batch
> Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size.<br></br>
> **Default Value**: 10 (Optional)<br></br>
-> `Config Param: COMMITS_ARCHIVAL_BATCH_SIZE_PROP`<br></br>
+> `Config Param: COMMITS_ARCHIVAL_BATCH_SIZE`<br></br>
+
+---
+
+> #### hoodie.compaction.reverse.log.read
+> HoodieLogFormatReader reads a logfile in the forward direction starting from pos=0 to pos=file_length. If this config is set to true, the reader reads the logfile in reverse direction, from pos=file_length to pos=0<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: COMPACTION_REVERSE_LOG_READ_ENABLE`<br></br>
---
-> #### hoodie.cleaner.parallelism
-> Parallelism for the cleaning operation. Increase this if cleaning becomes slow.<br></br>
-> **Default Value**: 200 (Optional)<br></br>
-> `Config Param: CLEANER_PARALLELISM`<br></br>
+> #### hoodie.cleaner.fileversions.retained
+> When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning.<br></br>
+> **Default Value**: 3 (Optional)<br></br>
+> `Config Param: CLEANER_FILE_VERSIONS_RETAINED`<br></br>
---
-> #### hoodie.cleaner.incremental.mode
-> When enabled, the plans for each cleaner service run is computed incrementally off the events in the timeline, since the last cleaner run. This is much more efficient than obtaining listings for the full table for each planning (even with a metadata table).<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: CLEANER_INCREMENTAL_MODE`<br></br>
+> #### hoodie.compact.inline
+> When set to true, compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INLINE_COMPACT`<br></br>
---
-> #### hoodie.cleaner.delete.bootstrap.base.file
-> When set to true, cleaner also deletes the bootstrap base file when it's skeleton base file is cleaned. Turn this to true, if you want to ensure the bootstrap dataset storage is reclaimed over time, as the table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap base files are also physically deleted, to comply with data privacy enforcement processes.<br></br>
+> #### hoodie.compaction.lazy.block.read
+> When merging the delta log files, this config helps to choose whether the log blocks should be read lazily or not. Choose true to use lazy block reading (low memory usage, but incurs seeks to each block header) or false for immediate block read (higher memory usage)<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: CLEANER_BOOTSTRAP_BASE_FILE_ENABLED`<br></br>
+> `Config Param: COMPACTION_LAZY_BLOCK_READ_ENABLE`<br></br>
---
-> #### hoodie.copyonwrite.insert.split.size
-> Number of inserts assigned for each partition/bucket for writing. We based the default on writing out 100MB files, with at least 1kb records (100K records per file), and over provision to 500K. As long as auto-tuning of splits is turned on, this only affects the first write, where there is no history to learn record sizes from.<br></br>
-> **Default Value**: 500000 (Optional)<br></br>
-> `Config Param: COPY_ON_WRITE_TABLE_INSERT_SPLIT_SIZE`<br></br>
+> #### hoodie.parquet.small.file.limit
+> During upsert operation, we opportunistically expand existing small files on storage, instead of writing new files, to keep number of files to an optimum. This config sets the file size limit below which a file on storage becomes a candidate to be selected as such a `small file`. By default, treat any file <= 100MB as a small file.<br></br>
+> **Default Value**: 104857600 (Optional)<br></br>
+> `Config Param: PARQUET_SMALL_FILE_LIMIT`<br></br>
---
> #### hoodie.compaction.strategy
> Compaction strategy decides which file groups are picked up for compaction during each compaction run. By default. Hudi picks the log file with most accumulated unmerged data<br></br>
> **Default Value**: org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy (Optional)<br></br>
-> `Config Param: COMPACTION_STRATEGY_PROP`<br></br>
+> `Config Param: COMPACTION_STRATEGY`<br></br>
---
> #### hoodie.compaction.target.io
> Amount of MBs to spend during compaction run for the LogFileSizeBasedCompactionStrategy. This value helps bound ingestion latency while compaction is run inline mode.<br></br>
> **Default Value**: 512000 (Optional)<br></br>
-> `Config Param: TARGET_IO_PER_COMPACTION_IN_MB_PROP`<br></br>
-
----
-
-> #### hoodie.compaction.payload.class
-> This needs to be same as class used during insert/upserts. Just like writing, compaction also uses the record payload class to merge records in the log against each other, merge again with the base file and produce the final record to be written after compaction.<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: PAYLOAD_CLASS_PROP`<br></br>
-
----
-
-> #### hoodie.cleaner.commits.retained
-> Number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br></br>
-> **Default Value**: 10 (Optional)<br></br>
-> `Config Param: CLEANER_COMMITS_RETAINED_PROP`<br></br>
+> `Config Param: TARGET_IO_PER_COMPACTION_IN_MB`<br></br>
---
-> #### hoodie.record.size.estimation.threshold
-> We use the previous commits' metadata to calculate the estimated record size and use it to bin pack records into partitions. If the previous commit is too small to make an accurate estimation, Hudi will search commits in the reverse order, until we find a commit that has totalBytesWritten larger than (PARQUET_SMALL_FILE_LIMIT_BYTES * this_threshold)<br></br>
-> **Default Value**: 1.0 (Optional)<br></br>
-> `Config Param: RECORD_SIZE_ESTIMATION_THRESHOLD_PROP`<br></br>
+> #### hoodie.copyonwrite.insert.auto.split
+> Config to control whether we control insert split sizes automatically based on average record sizes. It's recommended to keep this turned on, since hand tuning is otherwise extremely cumbersome.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: COPY_ON_WRITE_AUTO_SPLIT_INSERTS`<br></br>
---
-> #### hoodie.compaction.daybased.target.partitions
-> Used by org.apache.hudi.io.compact.strategy.DayBasedCompactionStrategy to denote the number of latest partitions to compact during a compaction run.<br></br>
-> **Default Value**: 10 (Optional)<br></br>
-> `Config Param: TARGET_PARTITIONS_PER_DAYBASED_COMPACTION_PROP`<br></br>
+> #### hoodie.compact.inline.max.delta.commits
+> Number of delta commits after the last compaction, before scheduling of a new compaction is attempted.<br></br>
+> **Default Value**: 5 (Optional)<br></br>
+> `Config Param: INLINE_COMPACT_NUM_DELTA_COMMITS`<br></br>
---
> #### hoodie.keep.min.commits
> Similar to hoodie.keep.max.commits, but controls the minimum number ofinstants to retain in the active timeline.<br></br>
> **Default Value**: 20 (Optional)<br></br>
-> `Config Param: MIN_COMMITS_TO_KEEP_PROP`<br></br>
-
----
-
-> #### hoodie.compaction.reverse.log.read
-> HoodieLogFormatReader reads a logfile in the forward direction starting from pos=0 to pos=file_length. If this config is set to true, the reader reads the logfile in reverse direction, from pos=file_length to pos=0<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: COMPACTION_REVERSE_LOG_READ_ENABLED_PROP`<br></br>
+> `Config Param: MIN_COMMITS_TO_KEEP`<br></br>
---
-> #### hoodie.compact.inline
-> When set to true, compaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INLINE_COMPACT_PROP`<br></br>
+> #### hoodie.cleaner.parallelism
+> Parallelism for the cleaning operation. Increase this if cleaning becomes slow.<br></br>
+> **Default Value**: 200 (Optional)<br></br>
+> `Config Param: CLEANER_PARALLELISM_VALUE`<br></br>
---
-> #### hoodie.clean.async
-> Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: ASYNC_CLEAN_PROP`<br></br>
+> #### hoodie.cleaner.incremental.mode
+> When enabled, the plans for each cleaner service run is computed incrementally off the events in the timeline, since the last cleaner run. This is much more efficient than obtaining listings for the full table for each planning (even with a metadata table).<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: CLEANER_INCREMENTAL_MODE_ENABLE`<br></br>
---
-> #### hoodie.copyonwrite.insert.auto.split
-> Config to control whether we control insert split sizes automatically based on average record sizes. It's recommended to keep this turned on, since hand tuning is otherwise extremely cumbersome.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: COPY_ON_WRITE_TABLE_AUTO_SPLIT_INSERTS`<br></br>
+> #### hoodie.record.size.estimation.threshold
+> We use the previous commits' metadata to calculate the estimated record size and use it to bin pack records into partitions. If the previous commit is too small to make an accurate estimation, Hudi will search commits in the reverse order, until we find a commit that has totalBytesWritten larger than (PARQUET_SMALL_FILE_LIMIT_BYTES * this_threshold)<br></br>
+> **Default Value**: 1.0 (Optional)<br></br>
+> `Config Param: RECORD_SIZE_ESTIMATION_THRESHOLD`<br></br>
---
-> #### hoodie.copyonwrite.record.size.estimate
-> The average record size. If not explicitly specified, hudi will compute the record size estimate compute dynamically based on commit metadata. This is critical in computing the insert parallelism and bin-packing inserts into small files.<br></br>
-> **Default Value**: 1024 (Optional)<br></br>
-> `Config Param: COPY_ON_WRITE_TABLE_RECORD_SIZE_ESTIMATE`<br></br>
+> #### hoodie.compact.inline.trigger.strategy
+> Controls how compaction scheduling is triggered, by time or num delta commits or combination of both. Valid options: NUM_COMMITS,TIME_ELAPSED,NUM_AND_TIME,NUM_OR_TIME<br></br>
+> **Default Value**: NUM_COMMITS (Optional)<br></br>
+> `Config Param: INLINE_COMPACT_TRIGGER_STRATEGY`<br></br>
---
-> #### hoodie.compact.inline.max.delta.commits
-> Number of delta commits after the last compaction, before scheduling of a new compaction is attempted.<br></br>
-> **Default Value**: 5 (Optional)<br></br>
-> `Config Param: INLINE_COMPACT_NUM_DELTA_COMMITS_PROP`<br></br>
+> #### hoodie.keep.max.commits
+> Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows.This config controls the maximum number of instants to retain in the active timeline. <br></br>
+> **Default Value**: 30 (Optional)<br></br>
+> `Config Param: MAX_COMMITS_TO_KEEP`<br></br>
---
-> #### hoodie.compact.inline.max.delta.seconds
-> Number of elapsed seconds after the last compaction, before scheduling a new one.<br></br>
-> **Default Value**: 3600 (Optional)<br></br>
-> `Config Param: INLINE_COMPACT_TIME_DELTA_SECONDS_PROP`<br></br>
+> #### hoodie.copyonwrite.insert.split.size
+> Number of inserts assigned for each partition/bucket for writing. We based the default on writing out 100MB files, with at least 1kb records (100K records per file), and over provision to 500K. As long as auto-tuning of splits is turned on, this only affects the first write, where there is no history to learn record sizes from.<br></br>
+> **Default Value**: 500000 (Optional)<br></br>
+> `Config Param: COPY_ON_WRITE_INSERT_SPLIT_SIZE`<br></br>
---
-> #### hoodie.cleaner.policy
-> Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. By default, cleaner spares the file slices written by the last N commits, determined by hoodie.cleaner.commits.retained Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query execution time<br></br>
-> **Default Value**: KEEP_LATEST_COMMITS (Optional)<br></br>
-> `Config Param: CLEANER_POLICY_PROP`<br></br>
+> #### hoodie.compaction.daybased.target.partitions
+> Used by org.apache.hudi.io.compact.strategy.DayBasedCompactionStrategy to denote the number of latest partitions to compact during a compaction run.<br></br>
+> **Default Value**: 10 (Optional)<br></br>
+> `Config Param: TARGET_PARTITIONS_PER_DAYBASED_COMPACTION`<br></br>
---
@@ -1984,98 +2105,98 @@ Configurations that control how file metadata is stored by Hudi, for transaction
> #### hoodie.filesystem.view.spillable.replaced.mem.fraction
> Fraction of the file system view memory, to be used for holding replace commit related metadata.<br></br>
> **Default Value**: 0.01 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_REPLACED_MEM_FRACTION`<br></br>
+> `Config Param: SPILLABLE_REPLACED_MEM_FRACTION`<br></br>
---
-> #### hoodie.filesystem.view.incr.timeline.sync.enable
-> Controls whether or not, the file system view is incrementally updated as new actions are performed on the timeline.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_INCREMENTAL_SYNC_MODE`<br></br>
+> #### hoodie.filesystem.view.spillable.dir
+> Path on local storage to use, when file system view is held in a spillable map.<br></br>
+> **Default Value**: /tmp/ (Optional)<br></br>
+> `Config Param: SPILLABLE_DIR`<br></br>
---
-> #### hoodie.filesystem.view.secondary.type
-> Specifies the secondary form of storage for file system view, if the primary (e.g timeline server) is unavailable.<br></br>
-> **Default Value**: MEMORY (Optional)<br></br>
-> `Config Param: FILESYSTEM_SECONDARY_VIEW_STORAGE_TYPE`<br></br>
+> #### hoodie.filesystem.remote.backup.view.enable
+> Config to control whether backup needs to be configured if clients were not able to reach timeline service.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: REMOTE_BACKUP_VIEW_ENABLE`<br></br>
---
> #### hoodie.filesystem.view.spillable.compaction.mem.fraction
> Fraction of the file system view memory, to be used for holding compaction related metadata.<br></br>
> **Default Value**: 0.8 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_PENDING_COMPACTION_MEM_FRACTION`<br></br>
+> `Config Param: SPILLABLE_COMPACTION_MEM_FRACTION`<br></br>
---
> #### hoodie.filesystem.view.spillable.mem
> Amount of memory to be used for holding file system view, before spilling to disk.<br></br>
> **Default Value**: 104857600 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_SPILLABLE_MEM`<br></br>
+> `Config Param: SPILLABLE_MEMORY`<br></br>
---
-> #### hoodie.filesystem.view.spillable.clustering.mem.fraction
-> Fraction of the file system view memory, to be used for holding clustering related metadata.<br></br>
-> **Default Value**: 0.01 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_PENDING_CLUSTERING_MEM_FRACTION`<br></br>
+> #### hoodie.filesystem.view.secondary.type
+> Specifies the secondary form of storage for file system view, if the primary (e.g timeline server) is unavailable.<br></br>
+> **Default Value**: MEMORY (Optional)<br></br>
+> `Config Param: SECONDARY_VIEW_TYPE`<br></br>
---
-> #### hoodie.filesystem.view.spillable.dir
-> Path on local storage to use, when file system view is held in a spillable map.<br></br>
-> **Default Value**: /tmp/view_map/ (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_SPILLABLE_DIR`<br></br>
+> #### hoodie.filesystem.view.remote.host
+> We expect this to be rarely hand configured.<br></br>
+> **Default Value**: localhost (Optional)<br></br>
+> `Config Param: REMOTE_HOST_NAME`<br></br>
---
-> #### hoodie.filesystem.view.remote.timeout.secs
-> Timeout in seconds, to wait for API requests against a remote file system view. e.g timeline server.<br></br>
-> **Default Value**: 300 (Optional)<br></br>
-> `Config Param: FILESYSTEM_REMOTE_TIMELINE_CLIENT_TIMEOUT_SECS`<br></br>
+> #### hoodie.filesystem.view.type
+> File system view provides APIs for viewing the files on the underlying lake storage, as file groups and file slices. This config controls how such a view is held. Options include MEMORY,SPILLABLE_DISK,EMBEDDED_KV_STORE,REMOTE_ONLY,REMOTE_FIRST which provide different trade offs for memory usage and API request performance.<br></br>
+> **Default Value**: MEMORY (Optional)<br></br>
+> `Config Param: VIEW_TYPE`<br></br>
---
-> #### hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction
-> Fraction of the file system view memory, to be used for holding mapping to bootstrap base files.<br></br>
-> **Default Value**: 0.05 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_BOOTSTRAP_BASE_FILE_FRACTION`<br></br>
+> #### hoodie.filesystem.view.remote.timeout.secs
+> Timeout in seconds, to wait for API requests against a remote file system view. e.g timeline server.<br></br>
+> **Default Value**: 300 (Optional)<br></br>
+> `Config Param: REMOTE_TIMEOUT_SECS`<br></br>
---
> #### hoodie.filesystem.view.remote.port
> Port to serve file system view queries, when remote. We expect this to be rarely hand configured.<br></br>
> **Default Value**: 26754 (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_REMOTE_PORT`<br></br>
+> `Config Param: REMOTE_PORT_NUM`<br></br>
---
-> #### hoodie.filesystem.view.type
-> File system view provides APIs for viewing the files on the underlying lake storage, as file groups and file slices. This config controls how such a view is held. Options include MEMORY,SPILLABLE_DISK,EMBEDDED_KV_STORE,REMOTE_ONLY,REMOTE_FIRST which provide different trade offs for memory usage and API request performance.<br></br>
-> **Default Value**: MEMORY (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_STORAGE_TYPE`<br></br>
+> #### hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction
+> Fraction of the file system view memory, to be used for holding mapping to bootstrap base files.<br></br>
+> **Default Value**: 0.05 (Optional)<br></br>
+> `Config Param: BOOTSTRAP_BASE_FILE_MEM_FRACTION`<br></br>
---
-> #### hoodie.filesystem.view.remote.host
-> We expect this to be rarely hand configured.<br></br>
-> **Default Value**: localhost (Optional)<br></br>
-> `Config Param: FILESYSTEM_VIEW_REMOTE_HOST`<br></br>
+> #### hoodie.filesystem.view.spillable.clustering.mem.fraction
+> Fraction of the file system view memory, to be used for holding clustering related metadata.<br></br>
+> **Default Value**: 0.01 (Optional)<br></br>
+> `Config Param: SPILLABLE_CLUSTERING_MEM_FRACTION`<br></br>
---
> #### hoodie.filesystem.view.rocksdb.base.path
> Path on local storage to use, when storing file system view in embedded kv store/rocksdb.<br></br>
> **Default Value**: /tmp/hoodie_timeline_rocksdb (Optional)<br></br>
-> `Config Param: ROCKSDB_BASE_PATH_PROP`<br></br>
+> `Config Param: ROCKSDB_BASE_PATH`<br></br>
---
-> #### hoodie.filesystem.remote.backup.view.enable
-> Config to control whether backup needs to be configured if clients were not able to reach timeline service.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: REMOTE_BACKUP_VIEW_HANDLER_ENABLE`<br></br>
+> #### hoodie.filesystem.view.incr.timeline.sync.enable
+> Controls whether or not, the file system view is incrementally updated as new actions are performed on the timeline.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INCREMENTAL_TIMELINE_SYNC_ENABLE`<br></br>
---
@@ -2084,122 +2205,122 @@ Configurations that control how file metadata is stored by Hudi, for transaction
Configurations that persist across writes and read on a Hudi table like base, log file formats, table name, creation schema, table version layouts. Configurations are loaded from hoodie.properties, these properties are usually set during initializing a path as hoodie base path and rarely changes during the lifetime of the table. Writers/Queries' configurations are validated against these each time for compatibility.
`Config Class`: org.apache.hudi.common.table.HoodieTableConfig<br></br>
-> #### hoodie.table.recordkey.fields
-> Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br></br>
+> #### hoodie.bootstrap.index.enable
+> Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_RECORDKEY_FIELDS`<br></br>
+> `Config Param: BOOTSTRAP_INDEX_ENABLE`<br></br>
---
-> #### hoodie.archivelog.folder
-> path under the meta folder, to store archived timeline instants at.<br></br>
-> **Default Value**: archived (Optional)<br></br>
-> `Config Param: HOODIE_ARCHIVELOG_FOLDER_PROP`<br></br>
+> #### hoodie.table.precombine.field
+> Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: PRECOMBINE_FIELD`<br></br>
---
-> #### hoodie.table.create.schema
-> Schema used when creating the table, for the first time.<br></br>
+> #### hoodie.table.partition.fields
+> Fields used to partition the table. Concatenated values of these fields are used as the partition path, by invoking toString()<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_CREATE_SCHEMA`<br></br>
+> `Config Param: PARTITION_FIELDS`<br></br>
---
-> #### hoodie.table.type
-> The table type for the underlying data, for this write. This can’t change between writes.<br></br>
-> **Default Value**: COPY_ON_WRITE (Optional)<br></br>
-> `Config Param: HOODIE_TABLE_TYPE_PROP`<br></br>
+> #### hoodie.populate.meta.fields
+> When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: POPULATE_META_FIELDS`<br></br>
---
-> #### hoodie.table.base.file.format
-> Base file format to store all the base file data.<br></br>
-> **Default Value**: PARQUET (Optional)<br></br>
-> `Config Param: HOODIE_BASE_FILE_FORMAT_PROP`<br></br>
+> #### hoodie.compaction.payload.class
+> Payload class to use for performing compactions, i.e merge delta logs with current base file and then produce a new base file.<br></br>
+> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
+> `Config Param: PAYLOAD_CLASS_NAME`<br></br>
---
-> #### hoodie.table.keygenerator.class
-> Key Generator class property for the hoodie table<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_KEY_GENERATOR_CLASS`<br></br>
+> #### hoodie.archivelog.folder
+> path under the meta folder, to store archived timeline instants at.<br></br>
+> **Default Value**: archived (Optional)<br></br>
+> `Config Param: ARCHIVELOG_FOLDER`<br></br>
---
-> #### hoodie.bootstrap.base.path
-> Base path of the dataset that needs to be bootstrapped as a Hudi table<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_BOOTSTRAP_BASE_PATH_PROP`<br></br>
+> #### hoodie.bootstrap.index.class
+> Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br></br>
+> **Default Value**: org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex (Optional)<br></br>
+> `Config Param: BOOTSTRAP_INDEX_CLASS_NAME`<br></br>
---
-> #### hoodie.table.name
-> Table name that will be used for registering with Hive. Needs to be same across runs.<br></br>
+> #### hoodie.table.type
+> The table type for the underlying data, for this write. This can’t change between writes.<br></br>
+> **Default Value**: COPY_ON_WRITE (Optional)<br></br>
+> `Config Param: TYPE`<br></br>
+
+---
+
+> #### hoodie.table.keygenerator.class
+> Key Generator class property for the hoodie table<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_NAME_PROP`<br></br>
+> `Config Param: KEY_GENERATOR_CLASS_NAME`<br></br>
---
> #### hoodie.table.version
> Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br></br>
> **Default Value**: ZERO (Optional)<br></br>
-> `Config Param: HOODIE_TABLE_VERSION_PROP`<br></br>
-
----
-
-> #### hoodie.table.precombine.field
-> Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_PRECOMBINE_FIELD_PROP`<br></br>
+> `Config Param: VERSION`<br></br>
---
-> #### hoodie.bootstrap.index.class
-> Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br></br>
-> **Default Value**: org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex (Optional)<br></br>
-> `Config Param: HOODIE_BOOTSTRAP_INDEX_CLASS_PROP`<br></br>
+> #### hoodie.table.base.file.format
+> Base file format to store all the base file data.<br></br>
+> **Default Value**: PARQUET (Optional)<br></br>
+> `Config Param: BASE_FILE_FORMAT`<br></br>
---
-> #### hoodie.table.log.file.format
-> Log format used for the delta logs.<br></br>
-> **Default Value**: HOODIE_LOG (Optional)<br></br>
-> `Config Param: HOODIE_LOG_FILE_FORMAT_PROP`<br></br>
+> #### hoodie.bootstrap.base.path
+> Base path of the dataset that needs to be bootstrapped as a Hudi table<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: BOOTSTRAP_BASE_PATH`<br></br>
---
-> #### hoodie.bootstrap.index.enable
-> Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined.<br></br>
+> #### hoodie.table.create.schema
+> Schema used when creating the table, for the first time.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_BOOTSTRAP_INDEX_ENABLE_PROP`<br></br>
+> `Config Param: CREATE_SCHEMA`<br></br>
---
> #### hoodie.timeline.layout.version
> Version of timeline used, by the table.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TIMELINE_LAYOUT_VERSION_PROP`<br></br>
+> `Config Param: TIMELINE_LAYOUT_VERSION`<br></br>
---
-> #### hoodie.table.partition.fields
-> Fields used to partition the table. Concatenated values of these fields are used as the partition path, by invoking toString()<br></br>
+> #### hoodie.table.name
+> Table name that will be used for registering with Hive. Needs to be same across runs.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: HOODIE_TABLE_PARTITION_FIELDS_PROP`<br></br>
+> `Config Param: NAME`<br></br>
---
-> #### hoodie.compaction.payload.class
-> Payload class to use for performing compactions, i.e merge delta logs with current base file and then produce a new base file.<br></br>
-> **Default Value**: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload (Optional)<br></br>
-> `Config Param: HOODIE_PAYLOAD_CLASS_PROP`<br></br>
+> #### hoodie.table.recordkey.fields
+> Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: RECORDKEY_FIELDS`<br></br>
---
-> #### hoodie.populate.meta.fields
-> When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: HOODIE_POPULATE_META_FIELDS`<br></br>
+> #### hoodie.table.log.file.format
+> Log format used for the delta logs.<br></br>
+> **Default Value**: HOODIE_LOG (Optional)<br></br>
+> `Config Param: LOG_FILE_FORMAT`<br></br>
---
@@ -2211,220 +2332,199 @@ Controls memory usage for compaction and merges, performed internally by Hudi.
> #### hoodie.memory.merge.fraction
> This fraction is multiplied with the user memory fraction (1 - spark.memory.fraction) to get a final fraction of heap space to use during merge<br></br>
> **Default Value**: 0.6 (Optional)<br></br>
-> `Config Param: MAX_MEMORY_FRACTION_FOR_MERGE_PROP`<br></br>
-
----
-
-> #### hoodie.memory.compaction.max.size
-> Maximum amount of memory used for compaction operations, before spilling to local storage.<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: MAX_MEMORY_FOR_COMPACTION_PROP`<br></br>
-
----
-
-> #### hoodie.memory.spillable.map.path
-> Default file path prefix for spillable map<br></br>
-> **Default Value**: /tmp/ (Optional)<br></br>
-> `Config Param: SPILLABLE_MAP_BASE_PATH_PROP`<br></br>
+> `Config Param: MAX_MEMORY_FRACTION_FOR_MERGE`<br></br>
---
-> #### hoodie.memory.compaction.fraction
-> HoodieCompactedLogScanner reads logblocks, converts records to HoodieRecords and then merges these log blocks and records. At any point, the number of entries in a log block can be less than or equal to the number of entries in the corresponding parquet file. This can lead to OOM in the Scanner. Hence, a spillable map helps alleviate the memory pressure. Use this config to set the max allowable inMemory footprint of the spillable map<br></br>
-> **Default Value**: 0.6 (Optional)<br></br>
-> `Config Param: MAX_MEMORY_FRACTION_FOR_COMPACTION_PROP`<br></br>
+> #### hoodie.memory.dfs.buffer.max.size
+> Property to control the max memory for dfs input stream buffer size<br></br>
+> **Default Value**: 16777216 (Optional)<br></br>
+> `Config Param: MAX_DFS_STREAM_BUFFER_SIZE`<br></br>
---
> #### hoodie.memory.writestatus.failure.fraction
> Property to control how what fraction of the failed record, exceptions we report back to driver. Default is 10%. If set to 100%, with lot of failures, this can cause memory pressure, cause OOMs and mask actual data errors.<br></br>
> **Default Value**: 0.1 (Optional)<br></br>
-> `Config Param: WRITESTATUS_FAILURE_FRACTION_PROP`<br></br>
+> `Config Param: WRITESTATUS_FAILURE_FRACTION`<br></br>
+
+---
+
+> #### hoodie.memory.compaction.fraction
+> HoodieCompactedLogScanner reads logblocks, converts records to HoodieRecords and then merges these log blocks and records. At any point, the number of entries in a log block can be less than or equal to the number of entries in the corresponding parquet file. This can lead to OOM in the Scanner. Hence, a spillable map helps alleviate the memory pressure. Use this config to set the max allowable inMemory footprint of the spillable map<br></br>
+> **Default Value**: 0.6 (Optional)<br></br>
+> `Config Param: MAX_MEMORY_FRACTION_FOR_COMPACTION`<br></br>
---
> #### hoodie.memory.merge.max.size
> Maximum amount of memory used for merge operations, before spilling to local storage.<br></br>
> **Default Value**: 1073741824 (Optional)<br></br>
-> `Config Param: MAX_MEMORY_FOR_MERGE_PROP`<br></br>
+> `Config Param: MAX_MEMORY_FOR_MERGE`<br></br>
---
-> #### hoodie.memory.dfs.buffer.max.size
-> Property to control the max memory for dfs input stream buffer size<br></br>
-> **Default Value**: 16777216 (Optional)<br></br>
-> `Config Param: MAX_DFS_STREAM_BUFFER_SIZE_PROP`<br></br>
+> #### hoodie.memory.spillable.map.path
+> Default file path prefix for spillable map<br></br>
+> **Default Value**: /tmp/ (Optional)<br></br>
+> `Config Param: SPILLABLE_MAP_BASE_PATH`<br></br>
---
-### Index Configs {#Index-Configs}
-
-Configurations that control indexing behavior, which tags incoming records as either inserts or updates to older records.
-
-`Config Class`: org.apache.hudi.config.HoodieIndexConfig<br></br>
-> #### hoodie.index.type
-> Type of index to use. Default is Bloom filter. Possible options are [BLOOM | GLOBAL_BLOOM |SIMPLE | GLOBAL_SIMPLE | INMEMORY | HBASE]. Bloom filters removes the dependency on a external system and is stored in the footer of the Parquet Data Files<br></br>
+> #### hoodie.memory.compaction.max.size
+> Maximum amount of memory used for compaction operations, before spilling to local storage.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: INDEX_TYPE_PROP`<br></br>
+> `Config Param: MAX_MEMORY_FOR_COMPACTION`<br></br>
---
-> #### hoodie.bloom.index.use.treebased.filter
-> Only applies if index type is BLOOM. When true, interval tree based file pruning optimization is enabled. This mode speeds-up file-pruning based on key ranges when compared with the brute-force mode<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_TREE_BASED_FILTER_PROP`<br></br>
+### Index Configs {#Index-Configs}
----
+Configurations that control indexing behavior, which tags incoming records as either inserts or updates to older records.
+`Config Class`: org.apache.hudi.config.HoodieIndexConfig<br></br>
> #### hoodie.index.bloom.num_entries
> Only applies if index type is BLOOM. This is the number of entries to be stored in the bloom filter. The rationale for the default: Assume the maxParquetFileSize is 128MB and averageRecordSize is 1kb and hence we approx a total of 130K records in a file. The default (60000) is roughly half of this approximation. Warning: Setting this very low, will generate a lot of false positives and index lookup will have to scan a lot more files than it has to and setting this to a very high number [...]
> **Default Value**: 60000 (Optional)<br></br>
-> `Config Param: BLOOM_FILTER_NUM_ENTRIES`<br></br>
+> `Config Param: BLOOM_FILTER_NUM_ENTRIES_VALUE`<br></br>
---
-> #### hoodie.bloom.index.bucketized.checking
-> Only applies if index type is BLOOM. When true, bucketized bloom filtering is enabled. This reduces skew seen in sort based bloom index lookup<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_BUCKETIZED_CHECKING_PROP`<br></br>
+> #### hoodie.bloom.index.keys.per.bucket
+> Only applies if bloomIndexBucketizedChecking is enabled and index type is bloom. This configuration controls the “bucket” size which tracks the number of record-key checks made against a single file and is the unit of work allocated to each partition performing bloom filter lookup. A higher value would amortize the fixed cost of reading a bloom filter to memory.<br></br>
+> **Default Value**: 10000000 (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_KEYS_PER_BUCKET`<br></br>
---
-> #### hoodie.bloom.index.update.partition.path
-> Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_UPDATE_PARTITION_PATH`<br></br>
+> #### hoodie.simple.index.input.storage.level
+> Only applies when #simpleIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br></br>
+> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
+> `Config Param: SIMPLE_INDEX_INPUT_STORAGE_LEVEL_VALUE`<br></br>
---
-> #### hoodie.bloom.index.parallelism
-> Only applies if index type is BLOOM. This is the amount of parallelism for index lookup, which involves a shuffle. By default, this is auto computed based on input workload characteristics.<br></br>
-> **Default Value**: 0 (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_PARALLELISM_PROP`<br></br>
+> #### hoodie.simple.index.parallelism
+> Only applies if index type is SIMPLE. This is the amount of parallelism for index lookup, which involves a Spark Shuffle<br></br>
+> **Default Value**: 50 (Optional)<br></br>
+> `Config Param: SIMPLE_INDEX_PARALLELISM`<br></br>
---
-> #### hoodie.bloom.index.input.storage.level
-> Only applies when #bloomIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br></br>
-> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_INPUT_STORAGE_LEVEL`<br></br>
+> #### hoodie.global.simple.index.parallelism
+> Only applies if index type is GLOBAL_SIMPLE. This is the amount of parallelism for index lookup, which involves a Spark Shuffle<br></br>
+> **Default Value**: 100 (Optional)<br></br>
+> `Config Param: GLOBAL_SIMPLE_INDEX_PARALLELISM`<br></br>
---
-> #### hoodie.bloom.index.keys.per.bucket
-> Only applies if bloomIndexBucketizedChecking is enabled and index type is bloom. This configuration controls the “bucket” size which tracks the number of record-key checks made against a single file and is the unit of work allocated to each partition performing bloom filter lookup. A higher value would amortize the fixed cost of reading a bloom filter to memory.<br></br>
-> **Default Value**: 10000000 (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_KEYS_PER_BUCKET_PROP`<br></br>
+> #### hoodie.simple.index.update.partition.path
+> Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: true description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition since version: version is not defined deprecated after: version i [...]
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: SIMPLE_INDEX_UPDATE_PARTITION_PATH_ENABLE`<br></br>
---
-> #### hoodie.simple.index.update.partition.path
-> Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: false description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition since version: version is not defined deprecated after: version [...]
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: SIMPLE_INDEX_UPDATE_PARTITION_PATH`<br></br>
+> #### hoodie.bloom.index.bucketized.checking
+> Only applies if index type is BLOOM. When true, bucketized bloom filtering is enabled. This reduces skew seen in sort based bloom index lookup<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_BUCKETIZED_CHECKING`<br></br>
---
-> #### hoodie.simple.index.input.storage.level
-> Only applies when #simpleIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br></br>
-> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
-> `Config Param: SIMPLE_INDEX_INPUT_STORAGE_LEVEL`<br></br>
+> #### hoodie.index.type
+> Type of index to use. Default is Bloom filter. Possible options are [BLOOM | GLOBAL_BLOOM |SIMPLE | GLOBAL_SIMPLE | INMEMORY | HBASE]. Bloom filters removes the dependency on a external system and is stored in the footer of the Parquet Data Files<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: INDEX_TYPE`<br></br>
---
> #### hoodie.index.bloom.fpp
> Only applies if index type is BLOOM. Error rate allowed given the number of entries. This is used to calculate how many bits should be assigned for the bloom filter and the number of hash functions. This is usually set very low (default: 0.000000001), we like to tradeoff disk space for lower false positives. If the number of entries added to bloom filter exceeds the configured value (hoodie.index.bloom.num_entries), then this fpp may not be honored.<br></br>
> **Default Value**: 0.000000001 (Optional)<br></br>
-> `Config Param: BLOOM_FILTER_FPP`<br></br>
+> `Config Param: BLOOM_FILTER_FPP_VALUE`<br></br>
---
-> #### hoodie.bloom.index.filter.dynamic.max.entries
-> The threshold for the maximum number of keys to record in a dynamic Bloom filter row. Only applies if filter type is BloomFilterTypeCode.DYNAMIC_V0.<br></br>
-> **Default Value**: 100000 (Optional)<br></br>
-> `Config Param: HOODIE_BLOOM_INDEX_FILTER_DYNAMIC_MAX_ENTRIES`<br></br>
+> #### hoodie.bloom.index.update.partition.path
+> Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_UPDATE_PARTITION_PATH_ENABLE`<br></br>
---
-> #### hoodie.index.class
-> Full path of user-defined index class and must be a subclass of HoodieIndex class. It will take precedence over the hoodie.index.type configuration if specified<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: INDEX_CLASS_PROP`<br></br>
+> #### hoodie.bloom.index.use.caching
+> Only applies if index type is BLOOM.When true, the input RDD will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_USE_CACHING`<br></br>
---
-> #### hoodie.bloom.index.filter.type
-> Filter type used. Default is BloomFilterTypeCode.SIMPLE. Available values are [BloomFilterTypeCode.SIMPLE , BloomFilterTypeCode.DYNAMIC_V0]. Dynamic bloom filters auto size themselves based on number of keys.<br></br>
-> **Default Value**: SIMPLE (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_FILTER_TYPE`<br></br>
+> #### hoodie.bloom.index.input.storage.level
+> Only applies when #bloomIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br></br>
+> **Default Value**: MEMORY_AND_DISK_SER (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_INPUT_STORAGE_LEVEL_VALUE`<br></br>
---
-> #### hoodie.global.simple.index.parallelism
-> Only applies if index type is GLOBAL_SIMPLE. This is the amount of parallelism for index lookup, which involves a Spark Shuffle<br></br>
-> **Default Value**: 100 (Optional)<br></br>
-> `Config Param: GLOBAL_SIMPLE_INDEX_PARALLELISM_PROP`<br></br>
+> #### hoodie.bloom.index.use.treebased.filter
+> Only applies if index type is BLOOM. When true, interval tree based file pruning optimization is enabled. This mode speeds-up file-pruning based on key ranges when compared with the brute-force mode<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_TREE_BASED_FILTER`<br></br>
---
-> #### hoodie.bloom.index.use.caching
-> Only applies if index type is BLOOM.When true, the input RDD will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_USE_CACHING_PROP`<br></br>
+> #### hoodie.bloom.index.parallelism
+> Only applies if index type is BLOOM. This is the amount of parallelism for index lookup, which involves a shuffle. By default, this is auto computed based on input workload characteristics.<br></br>
+> **Default Value**: 0 (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_PARALLELISM`<br></br>
---
-> #### hoodie.bloom.index.prune.by.ranges
-> Only applies if index type is BLOOM. When true, range information from files to leveraged speed up index lookups. Particularly helpful, if the key has a monotonously increasing prefix, such as timestamp. If the record key is completely random, it is better to turn this off, since range pruning will only add extra overhead to the index lookup.<br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: BLOOM_INDEX_PRUNE_BY_RANGES_PROP`<br></br>
+> #### hoodie.bloom.index.filter.dynamic.max.entries
+> The threshold for the maximum number of keys to record in a dynamic Bloom filter row. Only applies if filter type is BloomFilterTypeCode.DYNAMIC_V0.<br></br>
+> **Default Value**: 100000 (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_FILTER_DYNAMIC_MAX_ENTRIES`<br></br>
---
> #### hoodie.simple.index.use.caching
> Only applies if index type is SIMPLE. When true, the incoming writes will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: SIMPLE_INDEX_USE_CACHING_PROP`<br></br>
+> `Config Param: SIMPLE_INDEX_USE_CACHING`<br></br>
---
-> #### hoodie.simple.index.parallelism
-> Only applies if index type is SIMPLE. This is the amount of parallelism for index lookup, which involves a Spark Shuffle<br></br>
-> **Default Value**: 50 (Optional)<br></br>
-> `Config Param: SIMPLE_INDEX_PARALLELISM_PROP`<br></br>
+> #### hoodie.bloom.index.prune.by.ranges
+> Only applies if index type is BLOOM. When true, range information from files to leveraged speed up index lookups. Particularly helpful, if the key has a monotonously increasing prefix, such as timestamp. If the record key is completely random, it is better to turn this off, since range pruning will only add extra overhead to the index lookup.<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: BLOOM_INDEX_PRUNE_BY_RANGES`<br></br>
---
-### Storage Configs {#Storage-Configs}
+> #### hoodie.bloom.index.filter.type
+> Filter type used. Default is BloomFilterTypeCode.DYNAMIC_V0. Available values are [BloomFilterTypeCode.SIMPLE , BloomFilterTypeCode.DYNAMIC_V0]. Dynamic bloom filters auto size themselves based on number of keys.<br></br>
+> **Default Value**: DYNAMIC_V0 (Optional)<br></br>
+> `Config Param: BLOOM_FILTER_TYPE`<br></br>
-Configurations that control aspects around writing, sizing, reading base and log files.
+---
-`Config Class`: org.apache.hudi.config.HoodieStorageConfig<br></br>
-> #### hoodie.parquet.compression.ratio
-> Expected compression of parquet data used by Hudi, when it tries to size new parquet files. Increase this value, if bulk_insert is producing smaller than expected sized files<br></br>
-> **Default Value**: 0.1 (Optional)<br></br>
-> `Config Param: PARQUET_COMPRESSION_RATIO`<br></br>
+> #### hoodie.index.class
+> Full path of user-defined index class and must be a subclass of HoodieIndex class. It will take precedence over the hoodie.index.type configuration if specified<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: INDEX_CLASS_NAME`<br></br>
---
-> #### hoodie.hfile.max.file.size
-> Target file size for HFile base files.<br></br>
-> **Default Value**: 125829120 (Optional)<br></br>
-> `Config Param: HFILE_FILE_MAX_BYTES`<br></br>
+### Storage Configs {#Storage-Configs}
----
+Configurations that control aspects around writing, sizing, reading base and log files.
+`Config Class`: org.apache.hudi.config.HoodieStorageConfig<br></br>
> #### hoodie.logfile.data.block.max.size
> LogFile Data block max size. This is the maximum size allowed for a single data block to be appended to a log file. This helps to make sure the data appended to the log file is broken up into sizable blocks to prevent from OOM errors. This size should be greater than the JVM memory.<br></br>
> **Default Value**: 268435456 (Optional)<br></br>
-> `Config Param: LOGFILE_DATA_BLOCK_SIZE_MAX_BYTES`<br></br>
-
----
-
-> #### hoodie.parquet.block.size
-> Parquet RowGroup size. It's recommended to make this large enough that scan costs can be amortized by packing enough column values into a single row group.<br></br>
-> **Default Value**: 125829120 (Optional)<br></br>
-> `Config Param: PARQUET_BLOCK_SIZE_BYTES`<br></br>
+> `Config Param: LOGFILE_DATA_BLOCK_MAX_SIZE`<br></br>
---
@@ -2442,66 +2542,87 @@ Configurations that control aspects around writing, sizing, reading base and log
---
-> #### hoodie.orc.max.file.size
-> Target file size for ORC base files.<br></br>
-> **Default Value**: 125829120 (Optional)<br></br>
-> `Config Param: ORC_FILE_MAX_BYTES`<br></br>
+> #### hoodie.orc.compression.codec
+> Compression codec to use for ORC base files.<br></br>
+> **Default Value**: ZLIB (Optional)<br></br>
+> `Config Param: ORC_COMPRESSION_CODEC_NAME`<br></br>
---
-> #### hoodie.hfile.compression.algorithm
-> Compression codec to use for hfile base files.<br></br>
-> **Default Value**: GZ (Optional)<br></br>
-> `Config Param: HFILE_COMPRESSION_ALGORITHM`<br></br>
+> #### hoodie.parquet.max.file.size
+> Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br></br>
+> **Default Value**: 125829120 (Optional)<br></br>
+> `Config Param: PARQUET_MAX_FILE_SIZE`<br></br>
---
-> #### hoodie.parquet.max.file.size
-> Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br></br>
+> #### hoodie.hfile.max.file.size
+> Target file size for HFile base files.<br></br>
> **Default Value**: 125829120 (Optional)<br></br>
-> `Config Param: PARQUET_FILE_MAX_BYTES`<br></br>
+> `Config Param: HFILE_MAX_FILE_SIZE`<br></br>
---
-> #### hoodie.orc.compression.codec
-> Compression codec to use for ORC base files.<br></br>
-> **Default Value**: ZLIB (Optional)<br></br>
-> `Config Param: ORC_COMPRESSION_CODEC`<br></br>
+> #### hoodie.parquet.block.size
+> Parquet RowGroup size. It's recommended to make this large enough that scan costs can be amortized by packing enough column values into a single row group.<br></br>
+> **Default Value**: 125829120 (Optional)<br></br>
+> `Config Param: PARQUET_BLOCK_SIZE`<br></br>
---
> #### hoodie.logfile.max.size
> LogFile max size. This is the maximum size allowed for a log file before it is rolled over to the next version.<br></br>
> **Default Value**: 1073741824 (Optional)<br></br>
-> `Config Param: LOGFILE_SIZE_MAX_BYTES`<br></br>
+> `Config Param: LOGFILE_MAX_SIZE`<br></br>
---
-> #### hoodie.parquet.compression.codec
-> Compression Codec for parquet files<br></br>
-> **Default Value**: gzip (Optional)<br></br>
-> `Config Param: PARQUET_COMPRESSION_CODEC`<br></br>
+> #### hoodie.hfile.block.size
+> Lower values increase the size of metadata tracked within HFile, but can offer potentially faster lookup times.<br></br>
+> **Default Value**: 1048576 (Optional)<br></br>
+> `Config Param: HFILE_BLOCK_SIZE`<br></br>
+
+---
+
+> #### hoodie.parquet.page.size
+> Parquet page size. Page is the unit of read within a parquet file. Within a block, pages are compressed separately.<br></br>
+> **Default Value**: 1048576 (Optional)<br></br>
+> `Config Param: PARQUET_PAGE_SIZE`<br></br>
+
+---
+
+> #### hoodie.hfile.compression.algorithm
+> Compression codec to use for hfile base files.<br></br>
+> **Default Value**: GZ (Optional)<br></br>
+> `Config Param: HFILE_COMPRESSION_ALGORITHM_NAME`<br></br>
+
+---
+
+> #### hoodie.orc.max.file.size
+> Target file size for ORC base files.<br></br>
+> **Default Value**: 125829120 (Optional)<br></br>
+> `Config Param: ORC_FILE_MAX_SIZE`<br></br>
---
> #### hoodie.logfile.to.parquet.compression.ratio
> Expected additional compression as records move from log files to parquet. Used for merge_on_read table to send inserts into log files & control the size of compacted parquet file.<br></br>
> **Default Value**: 0.35 (Optional)<br></br>
-> `Config Param: LOGFILE_TO_PARQUET_COMPRESSION_RATIO`<br></br>
+> `Config Param: LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION`<br></br>
---
-> #### hoodie.parquet.page.size
-> Parquet page size. Page is the unit of read within a parquet file. Within a block, pages are compressed separately.<br></br>
-> **Default Value**: 1048576 (Optional)<br></br>
-> `Config Param: PARQUET_PAGE_SIZE_BYTES`<br></br>
+> #### hoodie.parquet.compression.ratio
+> Expected compression of parquet data used by Hudi, when it tries to size new parquet files. Increase this value, if bulk_insert is producing smaller than expected sized files<br></br>
+> **Default Value**: 0.1 (Optional)<br></br>
+> `Config Param: PARQUET_COMPRESSION_RATIO_FRACTION`<br></br>
---
-> #### hoodie.hfile.block.size
-> Lower values increase the size of metadata tracked within HFile, but can offer potentially faster lookup times.<br></br>
-> **Default Value**: 1048576 (Optional)<br></br>
-> `Config Param: HFILE_BLOCK_SIZE_BYTES`<br></br>
+> #### hoodie.parquet.compression.codec
+> Compression Codec for parquet files<br></br>
+> **Default Value**: gzip (Optional)<br></br>
+> `Config Param: PARQUET_COMPRESSION_CODEC_NAME`<br></br>
---
@@ -2510,10 +2631,18 @@ Configurations that control aspects around writing, sizing, reading base and log
Configurations that control the clustering table service in hudi, which optimizes the storage layout for better query performance by sorting and sizing data files.
`Config Class`: org.apache.hudi.config.HoodieClusteringConfig<br></br>
-> #### hoodie.clustering.updates.strategy
-> Determines how to handle updates, deletes to file groups that are under clustering. Default strategy just rejects the update<br></br>
-> **Default Value**: org.apache.hudi.client.clustering.update.strategy.SparkRejectUpdateStrategy (Optional)<br></br>
-> `Config Param: CLUSTERING_UPDATES_STRATEGY_PROP`<br></br>
+> #### hoodie.clustering.preserve.commit.metadata
+> When rewriting data, preserves existing hoodie_commit_time<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: PRESERVE_COMMIT_METADATA`<br></br>
+> `Since Version: 0.9.0`<br></br>
+
+---
+
+> #### hoodie.clustering.plan.strategy.max.num.groups
+> Maximum number of groups to create as part of ClusteringPlan. Increasing groups will increase parallelism<br></br>
+> **Default Value**: 30 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_MAX_GROUPS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2521,95 +2650,103 @@ Configurations that control the clustering table service in hudi, which optimize
> #### hoodie.clustering.async.max.commits
> Config to control frequency of async clustering<br></br>
> **Default Value**: 4 (Optional)<br></br>
-> `Config Param: ASYNC_CLUSTERING_MAX_COMMIT_PROP`<br></br>
+> `Config Param: ASYNC_CLUSTERING_MAX_COMMITS`<br></br>
> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.target.file.max.bytes
-> Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups<br></br>
-> **Default Value**: 1073741824 (Optional)<br></br>
-> `Config Param: CLUSTERING_TARGET_FILE_MAX_BYTES`<br></br>
+> #### hoodie.clustering.inline.max.commits
+> Config to control frequency of clustering planning<br></br>
+> **Default Value**: 4 (Optional)<br></br>
+> `Config Param: INLINE_CLUSTERING_MAX_COMMITS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.daybased.lookback.partitions
-> Number of partitions to list to create ClusteringPlan<br></br>
-> **Default Value**: 2 (Optional)<br></br>
-> `Config Param: CLUSTERING_TARGET_PARTITIONS`<br></br>
+> #### hoodie.clustering.plan.strategy.target.file.max.bytes
+> Each group can produce 'N' (CLUSTERING_MAX_GROUP_SIZE/CLUSTERING_TARGET_FILE_SIZE) output file groups<br></br>
+> **Default Value**: 1073741824 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_TARGET_FILE_MAX_BYTES`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.max.bytes.per.group
-> Each clustering operation can create multiple output file groups. Total amount of data processed by clustering operation is defined by below two properties (CLUSTERING_MAX_BYTES_PER_GROUP * CLUSTERING_MAX_NUM_GROUPS). Max amount of data to be included in one group<br></br>
-> **Default Value**: 2147483648 (Optional)<br></br>
-> `Config Param: CLUSTERING_MAX_BYTES_PER_GROUP`<br></br>
-> `Since Version: 0.7.0`<br></br>
+> #### hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions
+> Number of partitions to skip from latest when choosing partitions to create ClusteringPlan<br></br>
+> **Default Value**: 0 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
> #### hoodie.clustering.execution.strategy.class
> Config to provide a strategy class (subclass of RunClusteringStrategy) to define how the clustering plan is executed. By default, we sort the file groups in th plan by the specified columns, while meeting the configured target file sizes.<br></br>
> **Default Value**: org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy (Optional)<br></br>
-> `Config Param: CLUSTERING_EXECUTION_STRATEGY_CLASS`<br></br>
+> `Config Param: EXECUTION_STRATEGY_CLASS_NAME`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.max.num.groups
-> Maximum number of groups to create as part of ClusteringPlan. Increasing groups will increase parallelism<br></br>
-> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: CLUSTERING_MAX_NUM_GROUPS`<br></br>
+> #### hoodie.clustering.plan.strategy.max.bytes.per.group
+> Each clustering operation can create multiple output file groups. Total amount of data processed by clustering operation is defined by below two properties (CLUSTERING_MAX_BYTES_PER_GROUP * CLUSTERING_MAX_NUM_GROUPS). Max amount of data to be included in one group<br></br>
+> **Default Value**: 2147483648 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.inline
-> Turn on inline clustering - clustering will be run after each write operation is complete<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: INLINE_CLUSTERING_PROP`<br></br>
+> #### hoodie.clustering.plan.strategy.small.file.limit
+> Files smaller than the size specified here are candidates for clustering<br></br>
+> **Default Value**: 629145600 (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_SMALL_FILE_LIMIT`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.sort.columns
-> Columns to sort the data by when clustering<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: CLUSTERING_SORT_COLUMNS_PROPERTY`<br></br>
+> #### hoodie.clustering.async.enabled
+> Enable running of clustering service, asynchronously as inserts happen on the table.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ASYNC_CLUSTERING_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.inline.max.commits
-> Config to control frequency of clustering planning<br></br>
-> **Default Value**: 4 (Optional)<br></br>
-> `Config Param: INLINE_CLUSTERING_MAX_COMMIT_PROP`<br></br>
+> #### hoodie.clustering.plan.strategy.class
+> Config to provide a strategy class (subclass of ClusteringPlanStrategy) to create clustering plan i.e select what file groups are being clustered. Default strategy, looks at the last N (determined by hoodie.clustering.plan.strategy.daybased.lookback.partitions) day based partitions picks the small file slices within those partitions.<br></br>
+> **Default Value**: org.apache.hudi.client.clustering.plan.strategy.SparkRecentDaysClusteringPlanStrategy (Optional)<br></br>
+> `Config Param: PLAN_STRATEGY_CLASS_NAME`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.small.file.limit
-> Files smaller than the size specified here are candidates for clustering<br></br>
-> **Default Value**: 629145600 (Optional)<br></br>
-> `Config Param: CLUSTERING_PLAN_SMALL_FILE_LIMIT`<br></br>
+> #### hoodie.clustering.updates.strategy
+> Determines how to handle updates, deletes to file groups that are under clustering. Default strategy just rejects the update<br></br>
+> **Default Value**: org.apache.hudi.client.clustering.update.strategy.SparkRejectUpdateStrategy (Optional)<br></br>
+> `Config Param: UPDATES_STRATEGY`<br></br>
+> `Since Version: 0.7.0`<br></br>
+
+---
+
+> #### hoodie.clustering.inline
+> Turn on inline clustering - clustering will be run after each write operation is complete<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: INLINE_CLUSTERING`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.async.enabled
-> Enable running of clustering service, asynchronously as inserts happen on the table.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: ASYNC_CLUSTERING_ENABLE_OPT_KEY`<br></br>
+> #### hoodie.clustering.plan.strategy.sort.columns
+> Columns to sort the data by when clustering<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: PLAN_STRATEGY_SORT_COLUMNS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.clustering.plan.strategy.class
-> Config to provide a strategy class (subclass of ClusteringPlanStrategy) to create clustering plan i.e select what file groups are being clustered. Default strategy, looks at the last N (determined by hoodie.clustering.plan.strategy.daybased.lookback.partitions) day based partitions picks the small file slices within those partitions.<br></br>
-> **Default Value**: org.apache.hudi.client.clustering.plan.strategy.SparkRecentDaysClusteringPlanStrategy (Optional)<br></br>
-> `Config Param: CLUSTERING_PLAN_STRATEGY_CLASS`<br></br>
+> #### hoodie.clustering.plan.strategy.daybased.lookback.partitions
+> Number of partitions to list to create ClusteringPlan<br></br>
+> **Default Value**: 2 (Optional)<br></br>
+> `Config Param: DAYBASED_LOOKBACK_PARTITIONS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2619,26 +2756,34 @@ Configurations that control the clustering table service in hudi, which optimize
Configurations used by the Hudi Metadata Table. This table maintains the metadata about a given Hudi table (e.g file listings) to avoid overhead of accessing cloud storage, during queries.
`Config Class`: org.apache.hudi.common.config.HoodieMetadataConfig<br></br>
-> #### hoodie.metadata.enable
-> Enable the internal metadata table which serves table metadata like level file listings<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: METADATA_ENABLE_PROP`<br></br>
+> #### hoodie.metadata.compact.max.delta.commits
+> Controls how often the metadata table is compacted.<br></br>
+> **Default Value**: 24 (Optional)<br></br>
+> `Config Param: COMPACT_NUM_DELTA_COMMITS`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.insert.parallelism
-> Parallelism to use when inserting to the metadata table<br></br>
-> **Default Value**: 1 (Optional)<br></br>
-> `Config Param: METADATA_INSERT_PARALLELISM_PROP`<br></br>
+> #### hoodie.assume.date.partitioning
+> Should HoodieWriteClient assume the data is partitioned by dates, i.e three levels from base path. This is a stop-gap to support tables created by versions < 0.3.1. Will be removed eventually<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ASSUME_DATE_PARTITIONING`<br></br>
+> `Since Version: 0.3.0`<br></br>
+
+---
+
+> #### hoodie.metadata.validate
+> Validate contents of metadata table on each access; e.g against the actual listings from lake storage<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: VALIDATE_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.compact.max.delta.commits
-> Controls how often the metadata table is compacted.<br></br>
-> **Default Value**: 24 (Optional)<br></br>
-> `Config Param: METADATA_COMPACT_NUM_DELTA_COMMITS_PROP`<br></br>
+> #### hoodie.metadata.metrics.enable
+> Enable publishing of metrics around metadata table.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: METRICS_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2646,71 +2791,71 @@ Configurations used by the Hudi Metadata Table. This table maintains the metadat
> #### hoodie.metadata.cleaner.commits.retained
> Controls retention/history for metadata table.<br></br>
> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: CLEANER_COMMITS_RETAINED_PROP`<br></br>
+> `Config Param: CLEANER_COMMITS_RETAINED`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.keep.min.commits
-> Controls the archival of the metadata table’s timeline.<br></br>
-> **Default Value**: 20 (Optional)<br></br>
-> `Config Param: MIN_COMMITS_TO_KEEP_PROP`<br></br>
+> #### hoodie.file.listing.parallelism
+> Parallelism to use, when listing the table on lake storage.<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: FILE_LISTING_PARALLELISM_VALUE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.metrics.enable
-> Enable publishing of metrics around metadata table.<br></br>
+> #### hoodie.metadata.enable
+> Enable the internal metadata table which serves table metadata like level file listings<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: METADATA_METRICS_ENABLE_PROP`<br></br>
+> `Config Param: ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.assume.date.partitioning
-> Should HoodieWriteClient assume the data is partitioned by dates, i.e three levels from base path. This is a stop-gap to support tables created by versions < 0.3.1. Will be removed eventually<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: HOODIE_ASSUME_DATE_PARTITIONING_PROP`<br></br>
-> `Since Version: 0.3.0`<br></br>
+> #### hoodie.metadata.sync.enable
+> Enable syncing of metadata table from actions on the dataset<br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: SYNC_ENABLE`<br></br>
+> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.metadata.keep.max.commits
-> Controls the archival of the metadata table’s timeline.<br></br>
-> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: MAX_COMMITS_TO_KEEP_PROP`<br></br>
+> #### hoodie.metadata.clean.async
+> Enable asynchronous cleaning for metadata table<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: ASYNC_CLEAN_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.dir.filter.regex
-> Directories matching this regex, will be filtered out when initializing metadata table from lake storage for the first time.<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: DIRECTORY_FILTER_REGEX`<br></br>
+> #### hoodie.metadata.keep.max.commits
+> Controls the archival of the metadata table’s timeline.<br></br>
+> **Default Value**: 30 (Optional)<br></br>
+> `Config Param: MAX_COMMITS_TO_KEEP`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.validate
-> Validate contents of metadata table on each access; e.g against the actual listings from lake storage<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: METADATA_VALIDATE_PROP`<br></br>
+> #### hoodie.metadata.insert.parallelism
+> Parallelism to use when inserting to the metadata table<br></br>
+> **Default Value**: 1 (Optional)<br></br>
+> `Config Param: INSERT_PARALLELISM_VALUE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.metadata.clean.async
-> Enable asynchronous cleaning for metadata table<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: METADATA_ASYNC_CLEAN_PROP`<br></br>
+> #### hoodie.metadata.dir.filter.regex
+> Directories matching this regex, will be filtered out when initializing metadata table from lake storage for the first time.<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: DIR_FILTER_REGEX`<br></br>
> `Since Version: 0.7.0`<br></br>
---
-> #### hoodie.file.listing.parallelism
-> Parallelism to use, when listing the table on lake storage.<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: FILE_LISTING_PARALLELISM_PROP`<br></br>
+> #### hoodie.metadata.keep.min.commits
+> Controls the archival of the metadata table’s timeline.<br></br>
+> **Default Value**: 20 (Optional)<br></br>
+> `Config Param: MIN_COMMITS_TO_KEEP`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2723,23 +2868,15 @@ Configurations that control how you want to bootstrap your existing tables for t
> #### hoodie.bootstrap.partitionpath.translator.class
> Translates the partition paths from the bootstrapped data into how is laid out as a Hudi table.<br></br>
> **Default Value**: org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPartitionPathTranslator (Optional)<br></br>
-> `Config Param: BOOTSTRAP_PARTITION_PATH_TRANSLATOR_CLASS`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
-> #### hoodie.bootstrap.keygen.class
-> Key generator implementation to be used for generating keys from the bootstrapped dataset<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BOOTSTRAP_KEYGEN_CLASS`<br></br>
+> `Config Param: PARTITION_PATH_TRANSLATOR_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.mode.selector
-> Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped<br></br>
-> **Default Value**: org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector (Optional)<br></br>
-> `Config Param: BOOTSTRAP_MODE_SELECTOR`<br></br>
+> #### hoodie.bootstrap.full.input.provider
+> Class to use for reading the bootstrap dataset partitions/files, for Bootstrap mode FULL_RECORD<br></br>
+> **Default Value**: org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider (Optional)<br></br>
+> `Config Param: FULL_BOOTSTRAP_INPUT_PROVIDER_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2747,55 +2884,63 @@ Configurations that control how you want to bootstrap your existing tables for t
> #### hoodie.bootstrap.keygen.type
> Type of build-in key generator, currently support SIMPLE, COMPLEX, TIMESTAMP, CUSTOM, NON_PARTITION, GLOBAL_DELETE<br></br>
> **Default Value**: SIMPLE (Optional)<br></br>
-> `Config Param: BOOTSTRAP_KEYGEN_TYPE`<br></br>
+> `Config Param: KEYGEN_TYPE`<br></br>
> `Since Version: 0.9.0`<br></br>
---
-> #### hoodie.bootstrap.mode.selector.regex.mode
-> Bootstrap mode to apply for partition paths, that match regex above. METADATA_ONLY will generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will perform a full copy/rewrite of the data as a Hudi table.<br></br>
-> **Default Value**: METADATA_ONLY (Optional)<br></br>
-> `Config Param: BOOTSTRAP_MODE_SELECTOR_REGEX_MODE`<br></br>
+> #### hoodie.bootstrap.keygen.class
+> Key generator implementation to be used for generating keys from the bootstrapped dataset<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: KEYGEN_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.index.class
-> Implementation to use, for mapping a skeleton base file to a boostrap base file.<br></br>
-> **Default Value**: org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex (Optional)<br></br>
-> `Config Param: BOOTSTRAP_INDEX_CLASS_PROP`<br></br>
+> #### hoodie.bootstrap.parallelism
+> Parallelism value to be used to bootstrap data into hudi<br></br>
+> **Default Value**: 1500 (Optional)<br></br>
+> `Config Param: PARALLELISM_VALUE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.full.input.provider
-> Class to use for reading the bootstrap dataset partitions/files, for Bootstrap mode FULL_RECORD<br></br>
-> **Default Value**: org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider (Optional)<br></br>
-> `Config Param: FULL_BOOTSTRAP_INPUT_PROVIDER`<br></br>
+> #### hoodie.bootstrap.base.path
+> Base path of the dataset that needs to be bootstrapped as a Hudi table<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: BASE_PATH`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.parallelism
-> Parallelism value to be used to bootstrap data into hudi<br></br>
-> **Default Value**: 1500 (Optional)<br></br>
-> `Config Param: BOOTSTRAP_PARALLELISM`<br></br>
+> #### hoodie.bootstrap.mode.selector.regex
+> Matches each bootstrap dataset partition against this regex and applies the mode below to it.<br></br>
+> **Default Value**: .* (Optional)<br></br>
+> `Config Param: PARTITION_SELECTOR_REGEX_PATTERN`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.mode.selector.regex
-> Matches each bootstrap dataset partition against this regex and applies the mode below to it.<br></br>
-> **Default Value**: .* (Optional)<br></br>
-> `Config Param: BOOTSTRAP_MODE_SELECTOR_REGEX`<br></br>
+> #### hoodie.bootstrap.index.class
+> Implementation to use, for mapping a skeleton base file to a boostrap base file.<br></br>
+> **Default Value**: org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex (Optional)<br></br>
+> `Config Param: INDEX_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.bootstrap.base.path
-> Base path of the dataset that needs to be bootstrapped as a Hudi table<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: BOOTSTRAP_BASE_PATH_PROP`<br></br>
+> #### hoodie.bootstrap.mode.selector.regex.mode
+> Bootstrap mode to apply for partition paths, that match regex above. METADATA_ONLY will generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will perform a full copy/rewrite of the data as a Hudi table.<br></br>
+> **Default Value**: METADATA_ONLY (Optional)<br></br>
+> `Config Param: PARTITION_SELECTOR_REGEX_MODE`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.bootstrap.mode.selector
+> Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped<br></br>
+> **Default Value**: org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector (Optional)<br></br>
+> `Config Param: MODE_SELECTOR_CLASS_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2808,18 +2953,26 @@ These set of configs are used to enable monitoring and reporting of keyHudi stat
Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishes metrics on every commit, clean, rollback etc.
`Config Class`: org.apache.hudi.config.HoodieMetricsDatadogConfig<br></br>
-> #### hoodie.metrics.datadog.metric.tags
-> Datadog metric tags (comma-delimited) to be sent along with metrics data.<br></br>
+> #### hoodie.metrics.datadog.api.key.skip.validation
+> Before sending metrics via Datadog API, whether to skip validating Datadog API key or not. Default to false.<br></br>
+> **Default Value**: false (Optional)<br></br>
+> `Config Param: API_KEY_SKIP_VALIDATION`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.metrics.datadog.api.site
+> Datadog API site: EU or US<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_METRIC_TAGS`<br></br>
+> `Config Param: API_SITE_VALUE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.datadog.api.key.supplier
-> Datadog API key supplier to supply the API key at runtime. This will take effect if hoodie.metrics.datadog.api.key is not set.<br></br>
+> #### hoodie.metrics.datadog.metric.host
+> Datadog metric host to be sent along with metrics data.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_API_KEY_SUPPLIER`<br></br>
+> `Config Param: METRIC_HOST_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2827,7 +2980,7 @@ Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishe
> #### hoodie.metrics.datadog.metric.prefix
> Datadog metric prefix to be prepended to each metric name with a dot as delimiter. For example, if it is set to foo, foo. will be prepended.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_METRIC_PREFIX`<br></br>
+> `Config Param: METRIC_PREFIX_VALUE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2835,39 +2988,31 @@ Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishe
> #### hoodie.metrics.datadog.api.timeout.seconds
> Datadog API timeout in seconds. Default to 3.<br></br>
> **Default Value**: 3 (Optional)<br></br>
-> `Config Param: DATADOG_API_TIMEOUT_SECONDS`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
-> #### hoodie.metrics.datadog.report.period.seconds
-> Datadog reporting period in seconds. Default to 30.<br></br>
-> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: DATADOG_REPORT_PERIOD_SECONDS`<br></br>
+> `Config Param: API_TIMEOUT_IN_SECONDS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.datadog.metric.host
-> Datadog metric host to be sent along with metrics data.<br></br>
+> #### hoodie.metrics.datadog.api.key.supplier
+> Datadog API key supplier to supply the API key at runtime. This will take effect if hoodie.metrics.datadog.api.key is not set.<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_METRIC_HOST`<br></br>
+> `Config Param: API_KEY_SUPPLIER`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.datadog.api.key.skip.validation
-> Before sending metrics via Datadog API, whether to skip validating Datadog API key or not. Default to false.<br></br>
-> **Default Value**: false (Optional)<br></br>
-> `Config Param: DATADOG_API_KEY_SKIP_VALIDATION`<br></br>
+> #### hoodie.metrics.datadog.metric.tags
+> Datadog metric tags (comma-delimited) to be sent along with metrics data.<br></br>
+> **Default Value**: N/A (Required)<br></br>
+> `Config Param: METRIC_TAG_VALUES`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.datadog.api.site
-> Datadog API site: EU or US<br></br>
-> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_API_SITE`<br></br>
+> #### hoodie.metrics.datadog.report.period.seconds
+> Datadog reporting period in seconds. Default to 30.<br></br>
+> **Default Value**: 30 (Optional)<br></br>
+> `Config Param: REPORT_PERIOD_IN_SECONDS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2875,7 +3020,7 @@ Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishe
> #### hoodie.metrics.datadog.api.key
> Datadog API key<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: DATADOG_API_KEY`<br></br>
+> `Config Param: API_KEY`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2885,10 +3030,18 @@ Enables reporting on Hudi metrics using the Datadog reporter type. Hudi publishe
Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on every commit, clean, rollback etc.
`Config Class`: org.apache.hudi.config.HoodieMetricsPrometheusConfig<br></br>
-> #### hoodie.metrics.pushgateway.host
-> Hostname of the prometheus push gateway<br></br>
-> **Default Value**: localhost (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_HOST`<br></br>
+> #### hoodie.metrics.pushgateway.random.job.name.suffix
+> <br></br>
+> **Default Value**: true (Optional)<br></br>
+> `Config Param: PUSHGATEWAY_RANDOM_JOBNAME_SUFFIX`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.metrics.pushgateway.port
+> Port for the push gateway.<br></br>
+> **Default Value**: 9091 (Optional)<br></br>
+> `Config Param: PUSHGATEWAY_PORT_NUM`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2896,7 +3049,7 @@ Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on e
> #### hoodie.metrics.pushgateway.delete.on.shutdown
> <br></br>
> **Default Value**: true (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_DELETE_ON_SHUTDOWN`<br></br>
+> `Config Param: PUSHGATEWAY_DELETE_ON_SHUTDOWN_ENABLE`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2904,15 +3057,15 @@ Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on e
> #### hoodie.metrics.prometheus.port
> Port for prometheus server.<br></br>
> **Default Value**: 9090 (Optional)<br></br>
-> `Config Param: PROMETHEUS_PORT`<br></br>
+> `Config Param: PROMETHEUS_PORT_NUM`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.pushgateway.random.job.name.suffix
-> <br></br>
-> **Default Value**: true (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_RANDOM_JOB_NAME_SUFFIX`<br></br>
+> #### hoodie.metrics.pushgateway.job.name
+> Name of the push gateway job.<br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: PUSHGATEWAY_JOBNAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2920,23 +3073,15 @@ Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on e
> #### hoodie.metrics.pushgateway.report.period.seconds
> Reporting interval in seconds.<br></br>
> **Default Value**: 30 (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_REPORT_PERIOD_SECONDS`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
-> #### hoodie.metrics.pushgateway.job.name
-> Name of the push gateway job.<br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_JOB_NAME`<br></br>
+> `Config Param: PUSHGATEWAY_REPORT_PERIOD_IN_SECONDS`<br></br>
> `Since Version: 0.6.0`<br></br>
---
-> #### hoodie.metrics.pushgateway.port
-> Port for the push gateway.<br></br>
-> **Default Value**: 9091 (Optional)<br></br>
-> `Config Param: PUSHGATEWAY_PORT`<br></br>
+> #### hoodie.metrics.pushgateway.host
+> Hostname of the prometheus push gateway<br></br>
+> **Default Value**: localhost (Optional)<br></br>
+> `Config Param: PUSHGATEWAY_HOST_NAME`<br></br>
> `Since Version: 0.6.0`<br></br>
---
@@ -2946,18 +3091,42 @@ Enables reporting on Hudi metrics using Prometheus. Hudi publishes metrics on e
Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean, rollback etc. The following sections list the supported reporters.
`Config Class`: org.apache.hudi.config.HoodieMetricsConfig<br></br>
+> #### hoodie.metrics.reporter.type
+> Type of metrics reporter.<br></br>
+> **Default Value**: GRAPHITE (Optional)<br></br>
+> `Config Param: METRICS_REPORTER_TYPE_VALUE`<br></br>
+> `Since Version: 0.5.0`<br></br>
+
+---
+
> #### hoodie.metrics.jmx.host
> Jmx host to connect to<br></br>
> **Default Value**: localhost (Optional)<br></br>
-> `Config Param: JMX_HOST`<br></br>
+> `Config Param: JMX_HOST_NAME`<br></br>
> `Since Version: 0.5.1`<br></br>
---
+> #### hoodie.metrics.reporter.class
+> <br></br>
+> **Default Value**: (Optional)<br></br>
+> `Config Param: METRICS_REPORTER_CLASS_NAME`<br></br>
+> `Since Version: 0.6.0`<br></br>
+
+---
+
+> #### hoodie.metrics.graphite.port
+> Graphite port to connect to<br></br>
+> **Default Value**: 4756 (Optional)<br></br>
+> `Config Param: GRAPHITE_SERVER_PORT_NUM`<br></br>
+> `Since Version: 0.5.0`<br></br>
+
+---
+
> #### hoodie.metrics.executor.enable
> <br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: ENABLE_EXECUTOR_METRICS`<br></br>
+> `Config Param: EXECUTOR_METRICS_ENABLE`<br></br>
> `Since Version: 0.7.0`<br></br>
---
@@ -2965,7 +3134,7 @@ Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean
> #### hoodie.metrics.jmx.port
> Jmx port to connect to<br></br>
> **Default Value**: 9889 (Optional)<br></br>
-> `Config Param: JMX_PORT`<br></br>
+> `Config Param: JMX_PORT_NUM`<br></br>
> `Since Version: 0.5.1`<br></br>
---
@@ -2973,7 +3142,7 @@ Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean
> #### hoodie.metrics.graphite.host
> Graphite host to connect to<br></br>
> **Default Value**: localhost (Optional)<br></br>
-> `Config Param: GRAPHITE_SERVER_HOST`<br></br>
+> `Config Param: GRAPHITE_SERVER_HOST_NAME`<br></br>
> `Since Version: 0.5.0`<br></br>
---
@@ -2981,7 +3150,7 @@ Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean
> #### hoodie.metrics.on
> Turn on/off metrics reporting. off by default.<br></br>
> **Default Value**: false (Optional)<br></br>
-> `Config Param: METRICS_ON`<br></br>
+> `Config Param: TURN_METRICS_ON`<br></br>
> `Since Version: 0.5.0`<br></br>
---
@@ -2989,35 +3158,11 @@ Enables reporting on Hudi metrics. Hudi publishes metrics on every commit, clean
> #### hoodie.metrics.graphite.metric.prefix
> Standard prefix applied to all metrics. This helps to add datacenter, environment information for e.g<br></br>
> **Default Value**: N/A (Required)<br></br>
-> `Config Param: GRAPHITE_METRIC_PREFIX`<br></br>
+> `Config Param: GRAPHITE_METRIC_PREFIX_VALUE`<br></br>
> `Since Version: 0.5.1`<br></br>
---
-> #### hoodie.metrics.graphite.port
-> Graphite port to connect to<br></br>
-> **Default Value**: 4756 (Optional)<br></br>
-> `Config Param: GRAPHITE_SERVER_PORT`<br></br>
-> `Since Version: 0.5.0`<br></br>
-
----
-
-> #### hoodie.metrics.reporter.type
-> Type of metrics reporter.<br></br>
-> **Default Value**: GRAPHITE (Optional)<br></br>
-> `Config Param: METRICS_REPORTER_TYPE`<br></br>
-> `Since Version: 0.5.0`<br></br>
-
----
-
-> #### hoodie.metrics.reporter.class
-> <br></br>
-> **Default Value**: (Optional)<br></br>
-> `Config Param: METRICS_REPORTER_CLASS`<br></br>
-> `Since Version: 0.6.0`<br></br>
-
----
-
## Record Payload Config {#RECORD_PAYLOAD}
This is the lowest level of customization offered by Hudi. Record payloads define how to produce new values to upsert based on incoming new record and stored old record. Hudi provides default implementations such as OverwriteWithLatestAvroPayload which simply update table with the latest/last-written record. This can be overridden to a custom class extending HoodieRecordPayload class, on both datasource and WriteClient levels.
@@ -3029,14 +3174,14 @@ Payload related configs, that can be leveraged to control merges based on specif
> #### hoodie.payload.event.time.field
> Table column/field name to derive timestamp associated with the records. This canbe useful for e.g, determining the freshness of the table.<br></br>
> **Default Value**: ts (Optional)<br></br>
-> `Config Param: PAYLOAD_EVENT_TIME_FIELD_PROP`<br></br>
+> `Config Param: EVENT_TIME_FIELD`<br></br>
---
> #### hoodie.payload.ordering.field
> Table column/field name to order records that have the same key, before merging and writing to storage.<br></br>
> **Default Value**: ts (Optional)<br></br>
-> `Config Param: PAYLOAD_ORDERING_FIELD_PROP`<br></br>
+> `Config Param: ORDERING_FIELD`<br></br>
---
diff --git a/website/versioned_docs/version-0.9.0/cos_hoodie.md b/website/versioned_docs/version-0.9.0/cos_hoodie.md
new file mode 100644
index 0000000..dfde6e8
--- /dev/null
+++ b/website/versioned_docs/version-0.9.0/cos_hoodie.md
@@ -0,0 +1,71 @@
+---
+title: Tencent Cloud
+keywords: [ hudi, hive, tencent, cos, spark, presto]
+summary: In this page, we go over how to configure Hudi with COS filesystem.
+last_modified_at: 2020-04-21T11:38:24-10:00
+---
+In this page, we explain how to get your Hudi spark job to store into Tencent Cloud COS.
+
+## Tencent Cloud COS configs
+
+There are two configurations required for Hudi-COS compatibility:
+
+- Adding Tencent Cloud COS Credentials for Hudi
+- Adding required Jars to classpath
+
+### Tencent Cloud COS Credentials
+
+Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the `fs.defaultFS` with your COS bucket name, replace `fs.cosn.userinfo.secretId` with your COS secret Id, replace `fs.cosn.userinfo.secretKey` with your COS key. Hudi should be able to read/write from the bucket.
+
+```xml
+ <property>
+ <name>fs.defaultFS</name>
+ <value>cosn://bucketname</value>
+ <description>COS bucket name</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.userinfo.secretId</name>
+ <value>cos-secretId</value>
+ <description>Tencent Cloud Secret Id</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.userinfo.secretKey</name>
+ <value>cos-secretkey</value>
+ <description>Tencent Cloud Secret Key</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.bucket.region</name>
+ <value>ap-region</value>
+ <description>The region where the bucket is located.</description>
+ </property>
+
+ <property>
+ <name>fs.cosn.bucket.endpoint_suffix</name>
+ <value>cos.endpoint.suffix</value>
+ <description>
+ COS endpoint to connect to.
+ For public cloud users, it is recommended not to set this option, and only the correct area field is required.
+ </description>
+ </property>
+
+ <property>
+ <name>fs.cosn.impl</name>
+ <value>org.apache.hadoop.fs.CosFileSystem</value>
+ <description>The implementation class of the CosN Filesystem.</description>
+ </property>
+
+ <property>
+ <name>fs.AbstractFileSystem.cosn.impl</name>
+ <value>org.apache.hadoop.fs.CosN</value>
+ <description>The implementation class of the CosN AbstractFileSystem.</description>
+ </property>
+
+```
+
+### Tencent Cloud COS Libs
+COS hadoop libraries to add to our classpath
+
+- org.apache.hadoop:hadoop-cos:2.8.5
diff --git a/website/versioned_docs/version-0.9.0/deployment.md b/website/versioned_docs/version-0.9.0/deployment.md
new file mode 100644
index 0000000..20bf723
--- /dev/null
+++ b/website/versioned_docs/version-0.9.0/deployment.md
@@ -0,0 +1,577 @@
+---
+title: Deployment
+keywords: [ hudi, administration, operation, devops, deployment]
+summary: This section offers an overview of tools available to operate an ecosystem of Hudi
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+
+This section provides all the help you need to deploy and operate Hudi tables at scale.
+Specifically, we will cover the following aspects.
+
+ - [Deployment Model](#deploying) : How various Hudi components are deployed and managed.
+ - [Upgrading Versions](#upgrading) : Picking up new releases of Hudi, guidelines and general best-practices.
+ - [Migrating to Hudi](#migrating) : How to migrate your existing tables to Apache Hudi.
+ - [Interacting via CLI](#cli) : Using the CLI to perform maintenance or deeper introspection.
+ - [Troubleshooting](#troubleshooting) : Uncovering, triaging and resolving issues in production.
+
+## Deploying
+
+All in all, Hudi deploys with no long running servers or additional infrastructure cost to your data lake. In fact, Hudi pioneered this model of building a transactional distributed storage layer
+using existing infrastructure and its heartening to see other systems adopting similar approaches as well. Hudi writing is done via Spark jobs (DeltaStreamer or custom Spark datasource jobs), deployed per standard Apache Spark [recommendations](https://spark.apache.org/docs/latest/cluster-overview).
+Querying Hudi tables happens via libraries installed into Apache Hive, Apache Spark or PrestoDB and hence no additional infrastructure is necessary.
+
+A typical Hudi data ingestion can be achieved in 2 modes. In a single run mode, Hudi ingestion reads next batch of data, ingest them to Hudi table and exits. In continuous mode, Hudi ingestion runs as a long-running service executing ingestion in a loop.
+
+With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting delta files. Again, compaction can be performed in an asynchronous-mode by letting compaction run concurrently with ingestion or in a serial fashion with one after another.
+
+### DeltaStreamer
+
+[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+
+Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
+
+```java
+[hoodie]$ spark-submit --packages org.apache.hudi:hudi-utilities-bundle_2.11:0.5.3,org.apache.spark:spark-avro_2.11:2.4.4 \
+ --master yarn \
+ --deploy-mode cluster \
+ --num-executors 10 \
+ --executor-memory 3g \
+ --driver-memory 6g \
+ --conf spark.driver.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_driver.hprof" \
+ --conf spark.executor.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_executor.hprof" \
+ --queue hadoop-platform-queue \
+ --conf spark.scheduler.mode=FAIR \
+ --conf spark.yarn.executor.memoryOverhead=1072 \
+ --conf spark.yarn.driver.memoryOverhead=2048 \
+ --conf spark.task.cpus=1 \
+ --conf spark.executor.cores=1 \
+ --conf spark.task.maxFailures=10 \
+ --conf spark.memory.fraction=0.4 \
+ --conf spark.rdd.compress=true \
+ --conf spark.kryoserializer.buffer.max=200m \
+ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+ --conf spark.memory.storageFraction=0.1 \
+ --conf spark.shuffle.service.enabled=true \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --conf spark.ui.port=5555 \
+ --conf spark.driver.maxResultSize=3g \
+ --conf spark.executor.heartbeatInterval=120s \
+ --conf spark.network.timeout=600s \
+ --conf spark.eventLog.overwrite=true \
+ --conf spark.eventLog.enabled=true \
+ --conf spark.eventLog.dir=hdfs:///user/spark/applicationHistory \
+ --conf spark.yarn.max.executor.failures=10 \
+ --conf spark.sql.catalogImplementation=hive \
+ --conf spark.sql.shuffle.partitions=100 \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+```
+
+ - **Continuous Mode** : Here, deltastreamer runs an infinite loop with each round performing one ingestion round as described in **Run Once Mode**. The frequency of data ingestion can be controlled by the configuration "--min-sync-interval-seconds". For Merge-On-Read tables, Compaction is run in asynchronous fashion concurrently with ingestion unless disabled by passing the flag "--disable-compaction". Every ingestion run triggers a compaction request asynchronously and this frequency [...]
+
+Here is an example invocation for reading from kafka topic in a continuous mode and writing to Merge On Read table type in a yarn cluster.
+
+```java
+[hoodie]$ spark-submit --packages org.apache.hudi:hudi-utilities-bundle_2.11:0.5.3,org.apache.spark:spark-avro_2.11:2.4.4 \
+ --master yarn \
+ --deploy-mode cluster \
+ --num-executors 10 \
+ --executor-memory 3g \
+ --driver-memory 6g \
+ --conf spark.driver.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_driver.hprof" \
+ --conf spark.executor.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_executor.hprof" \
+ --queue hadoop-platform-queue \
+ --conf spark.scheduler.mode=FAIR \
+ --conf spark.yarn.executor.memoryOverhead=1072 \
+ --conf spark.yarn.driver.memoryOverhead=2048 \
+ --conf spark.task.cpus=1 \
+ --conf spark.executor.cores=1 \
+ --conf spark.task.maxFailures=10 \
+ --conf spark.memory.fraction=0.4 \
+ --conf spark.rdd.compress=true \
+ --conf spark.kryoserializer.buffer.max=200m \
+ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+ --conf spark.memory.storageFraction=0.1 \
+ --conf spark.shuffle.service.enabled=true \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --conf spark.ui.port=5555 \
+ --conf spark.driver.maxResultSize=3g \
+ --conf spark.executor.heartbeatInterval=120s \
+ --conf spark.network.timeout=600s \
+ --conf spark.eventLog.overwrite=true \
+ --conf spark.eventLog.enabled=true \
+ --conf spark.eventLog.dir=hdfs:///user/spark/applicationHistory \
+ --conf spark.yarn.max.executor.failures=10 \
+ --conf spark.sql.catalogImplementation=hive \
+ --conf spark.sql.shuffle.partitions=100 \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
+ --continuous
+```
+
+### Spark Datasource Writer Jobs
+
+As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+
+Here is an example invocation using spark datasource
+
+```java
+inputDF.write()
+ .format("org.apache.hudi")
+ .options(clientOpts) // any of the Hudi client opts can be passed in as well
+ .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")
+ .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")
+ .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")
+ .option(HoodieWriteConfig.TABLE_NAME, tableName)
+ .mode(SaveMode.Append)
+ .save(basePath);
+```
+
+## Upgrading
+
+New Hudi releases are listed on the [releases page](/releases), with detailed notes which list all the changes, with highlights in each release.
+At the end of the day, Hudi is a storage system and with that comes a lot of responsibilities, which we take seriously.
+
+As general guidelines,
+
+ - We strive to keep all changes backwards compatible (i.e new code can read old data/timeline files) and when we cannot, we will provide upgrade/downgrade tools via the CLI
+ - We cannot always guarantee forward compatibility (i.e old code being able to read data/timeline files written by a greater version). This is generally the norm, since no new features can be built otherwise.
+ However any large such changes, will be turned off by default, for smooth transition to newer release. After a few releases and once enough users deem the feature stable in production, we will flip the defaults in a subsequent release.
+ - Always upgrade the query bundles (mr-bundle, presto-bundle, spark-bundle) first and then upgrade the writers (deltastreamer, spark jobs using datasource). This often provides the best experience and it's easy to fix
+ any issues by rolling forward/back the writer code (which typically you might have more control over)
+ - With large, feature rich releases we recommend migrating slowly, by first testing in staging environments and running your own tests. Upgrading Hudi is no different than upgrading any database system.
+
+Note that release notes can override this information with specific instructions, applicable on case-by-case basis.
+
+## Migrating
+
+Currently migrating to Hudi can be done using two approaches
+
+- **Convert newer partitions to Hudi** : This model is suitable for large event tables (e.g: click streams, ad impressions), which also typically receive writes for the last few days alone. You can convert the last
+ N partitions to Hudi and proceed writing as if it were a Hudi table to begin with. The Hudi query side code is able to correctly handle both hudi and non-hudi data partitions.
+- **Full conversion to Hudi** : This model is suitable if you are currently bulk/full loading the table few times a day (e.g database ingestion). The full conversion of Hudi is simply a one-time step (akin to 1 run of your existing job),
+ which moves all of the data into the Hudi format and provides the ability to incrementally update for future writes.
+
+For more details, refer to the detailed [migration guide](/docs/migration_guide). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
+
+## CLI
+
+Once hudi has been built, the shell can be fired by via `cd hudi-cli && ./hudi-cli.sh`. A hudi table resides on DFS, in a location referred to as the `basePath` and
+we would need this location in order to connect to a Hudi table. Hudi library effectively manages this table internally, using `.hoodie` subfolder to track all metadata.
+
+To initialize a hudi table, use the following command.
+
+```java
+===================================================================
+* ___ ___ *
+* /\__\ ___ /\ \ ___ *
+* / / / /\__\ / \ \ /\ \ *
+* / /__/ / / / / /\ \ \ \ \ \ *
+* / \ \ ___ / / / / / \ \__\ / \__\ *
+* / /\ \ /\__\ / /__/ ___ / /__/ \ |__| / /\/__/ *
+* \/ \ \/ / / \ \ \ /\__\ \ \ \ / / / /\/ / / *
+* \ / / \ \ / / / \ \ / / / \ /__/ *
+* / / / \ \/ / / \ \/ / / \ \__\ *
+* / / / \ / / \ / / \/__/ *
+* \/__/ \/__/ \/__/ Apache Hudi CLI *
+* *
+===================================================================
+
+hudi->create --path /user/hive/warehouse/table1 --tableName hoodie_table_1 --tableType COPY_ON_WRITE
+.....
+```
+
+To see the description of hudi table, use the command:
+
+```java
+hudi:hoodie_table_1->desc
+18/09/06 15:57:19 INFO timeline.HoodieActiveTimeline: Loaded instants []
+ _________________________________________________________
+ | Property | Value |
+ |========================================================|
+ | basePath | ... |
+ | metaPath | ... |
+ | fileSystem | hdfs |
+ | hoodie.table.name | hoodie_table_1 |
+ | hoodie.table.type | COPY_ON_WRITE |
+ | hoodie.archivelog.folder| |
+```
+
+Following is a sample command to connect to a Hudi table contains uber trips.
+
+```java
+hudi:trips->connect --path /app/uber/trips
+
+16/10/05 23:20:37 INFO model.HoodieTableMetadata: All commits :HoodieCommits{commitList=[20161002045850, 20161002052915, 20161002055918, 20161002065317, 20161002075932, 20161002082904, 20161002085949, 20161002092936, 20161002105903, 20161002112938, 20161002123005, 20161002133002, 20161002155940, 20161002165924, 20161002172907, 20161002175905, 20161002190016, 20161002192954, 20161002195925, 20161002205935, 20161002215928, 20161002222938, 20161002225915, 20161002232906, 20161003003028, 201 [...]
+Metadata for table trips loaded
+```
+
+Once connected to the table, a lot of other commands become available. The shell has contextual autocomplete help (press TAB) and below is a list of all commands, few of which are reviewed in this section
+are reviewed
+
+```java
+hudi:trips->help
+* ! - Allows execution of operating system (OS) commands
+* // - Inline comment markers (start of line only)
+* ; - Inline comment markers (start of line only)
+* addpartitionmeta - Add partition metadata to a table, if not present
+* clear - Clears the console
+* cls - Clears the console
+* commit rollback - Rollback a commit
+* commits compare - Compare commits with another Hoodie table
+* commit showfiles - Show file level details of a commit
+* commit showpartitions - Show partition level details of a commit
+* commits refresh - Refresh the commits
+* commits show - Show the commits
+* commits sync - Compare commits with another Hoodie table
+* connect - Connect to a hoodie table
+* date - Displays the local date and time
+* exit - Exits the shell
+* help - List all commands usage
+* quit - Exits the shell
+* records deduplicate - De-duplicate a partition path contains duplicates & produce repaired files to replace with
+* script - Parses the specified resource file and executes its commands
+* stats filesizes - File Sizes. Display summary stats on sizes of files
+* stats wa - Write Amplification. Ratio of how many records were upserted to how many records were actually written
+* sync validate - Validate the sync by counting the number of records
+* system properties - Shows the shell's properties
+* utils loadClass - Load a class
+* version - Displays shell version
+
+hudi:trips->
+```
+
+
+### Inspecting Commits
+
+The task of upserting or inserting a batch of incoming records is known as a **commit** in Hudi. A commit provides basic atomicity guarantees such that only committed data is available for querying.
+Each commit has a monotonically increasing string/number called the **commit number**. Typically, this is the time at which we started the commit.
+
+To view some basic information about the last 10 commits,
+
+
+```java
+hudi:trips->commits show --sortBy "Total Bytes Written" --desc true --limit 10
+ ________________________________________________________________________________________________________________________________________________________________________
+ | CommitTime | Total Bytes Written| Total Files Added| Total Files Updated| Total Partitions Written| Total Records Written| Total Update Records Written| Total Errors|
+ |=======================================================================================================================================================================|
+ ....
+ ....
+ ....
+```
+
+At the start of each write, Hudi also writes a .inflight commit to the .hoodie folder. You can use the timestamp there to estimate how long the commit has been inflight
+
+
+```java
+$ hdfs dfs -ls /app/uber/trips/.hoodie/*.inflight
+-rw-r--r-- 3 vinoth supergroup 321984 2016-10-05 23:18 /app/uber/trips/.hoodie/20161005225920.inflight
+```
+
+
+### Drilling Down to a specific Commit
+
+To understand how the writes spread across specific partiions,
+
+
+```java
+hudi:trips->commit showpartitions --commit 20161005165855 --sortBy "Total Bytes Written" --desc true --limit 10
+ __________________________________________________________________________________________________________________________________________
+ | Partition Path| Total Files Added| Total Files Updated| Total Records Inserted| Total Records Updated| Total Bytes Written| Total Errors|
+ |=========================================================================================================================================|
+ ....
+ ....
+```
+
+If you need file level granularity , we can do the following
+
+
+```java
+hudi:trips->commit showfiles --commit 20161005165855 --sortBy "Partition Path"
+ ________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File ID | Previous Commit| Total Records Updated| Total Records Written| Total Bytes Written| Total Errors|
+ |=======================================================================================================================================================|
+ ....
+ ....
+```
+
+
+### FileSystem View
+
+Hudi views each partition as a collection of file-groups with each file-group containing a list of file-slices in commit order (See concepts).
+The below commands allow users to view the file-slices for a data-set.
+
+```java
+hudi:stock_ticks_mor->show fsview all
+ ....
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta File Size| Delta Files |
+ |==============================================================================================================================================================================================================================================================================================================================================================================================================|
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]|
+
+
+
+hudi:stock_ticks_mor->show fsview latest --partitionPath "2018/08/31"
+ ......
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ [...]
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta Size| Delta Size - compaction scheduled| Delta Size - compaction unscheduled| Delta To Base Ratio - compaction scheduled| Delta To Base Ratio - compaction unscheduled| Delta Files - compaction scheduled | Delta Files - compaction unscheduled|
+ |========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== [...]
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | 20.8 KB | 0.0 B | 0.0 B | 0.0 B | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]| [] |
+
+```
+
+
+### Statistics
+
+Since Hudi directly manages file sizes for DFS table, it might be good to get an overall picture
+
+
+```java
+hudi:trips->stats filesizes --partitionPath 2016/09/01 --sortBy "95th" --desc true --limit 10
+ ________________________________________________________________________________________________
+ | CommitTime | Min | 10th | 50th | avg | 95th | Max | NumFiles| StdDev |
+ |===============================================================================================|
+ | <COMMIT_ID> | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 2 | 2.3 KB |
+ ....
+ ....
+```
+
+In case of Hudi write taking much longer, it might be good to see the write amplification for any sudden increases
+
+
+```java
+hudi:trips->stats wa
+ __________________________________________________________________________
+ | CommitTime | Total Upserted| Total Written| Write Amplifiation Factor|
+ |=========================================================================|
+ ....
+ ....
+```
+
+
+### Archived Commits
+
+In order to limit the amount of growth of .commit files on DFS, Hudi archives older .commit files (with due respect to the cleaner policy) into a commits.archived file.
+This is a sequence file that contains a mapping from commitNumber => json with raw information about the commit (same that is nicely rolled up above).
+
+
+### Compactions
+
+To get an idea of the lag between compaction and writer applications, use the below command to list down all
+pending compactions.
+
+```java
+hudi:trips->compactions show all
+ ___________________________________________________________________
+ | Compaction Instant Time| State | Total FileIds to be Compacted|
+ |==================================================================|
+ | <INSTANT_1> | REQUESTED| 35 |
+ | <INSTANT_2> | INFLIGHT | 27 |
+```
+
+To inspect a specific compaction plan, use
+
+```java
+hudi:trips->compaction show --instant <INSTANT_1>
+ _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File Id | Base Instant | Data File Path | Total Delta Files| getMetrics |
+ |================================================================================================================================================================================================================================================
+ | 2018/07/17 | <UUID> | <INSTANT_1> | viewfs://ns-default/.../../UUID_<INSTANT>.parquet | 1 | {TOTAL_LOG_FILES=1.0, TOTAL_IO_READ_MB=1230.0, TOTAL_LOG_FILES_SIZE=2.51255751E8, TOTAL_IO_WRITE_MB=991.0, TOTAL_IO_MB=2221.0}|
+
+```
+
+To manually schedule or run a compaction, use the below command. This command uses spark launcher to perform compaction
+operations.
+
+**NOTE:** Make sure no other application is scheduling compaction for this table concurrently
+{: .notice--info}
+
+```java
+hudi:trips->help compaction schedule
+Keyword: compaction schedule
+Description: Schedule Compaction
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: false
+ Default if specified: '__NULL__'
+ Default if unspecified: '1G'
+
+* compaction schedule - Schedule Compaction
+```
+
+```java
+hudi:trips->help compaction run
+Keyword: compaction run
+Description: Run Compaction for given instant time
+ Keyword: tableName
+ Help: Table name
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: parallelism
+ Help: Parallelism for hoodie compaction
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: schemaFilePath
+ Help: Path for Avro schema file
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: retry
+ Help: Number of retries
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: compactionInstant
+ Help: Base path for the target hoodie table
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+* compaction run - Run Compaction for given instant time
+```
+
+### Validate Compaction
+
+Validating a compaction plan : Check if all the files necessary for compactions are present and are valid
+
+```java
+hudi:stock_ticks_mor->compaction validate --instant 20181005222611
+...
+
+ COMPACTION PLAN VALID
+
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error|
+ |==========================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | true | |
+
+
+
+hudi:stock_ticks_mor->compaction validate --instant 20181005222601
+
+ COMPACTION PLAN INVALID
+
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error |
+ |=====================================================================================================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | false| All log files specified in compaction operation is not present. Missing .... |
+```
+
+**NOTE:** The following commands must be executed without any other writer/ingestion application running.
+{: .notice--warning}
+
+Sometimes, it becomes necessary to remove a fileId from a compaction-plan inorder to speed-up or unblock compaction
+operation. Any new log-files that happened on this file after the compaction got scheduled will be safely renamed
+so that are preserved. Hudi provides the following CLI to support it
+
+
+### Unscheduling Compaction
+
+```java
+hudi:trips->compaction unscheduleFileId --fileId <FileUUID>
+....
+No File renames needed to unschedule file from pending compaction. Operation successful.
+```
+
+In other cases, an entire compaction plan needs to be reverted. This is supported by the following CLI
+
+```java
+hudi:trips->compaction unschedule --compactionInstant <compactionInstant>
+.....
+No File renames needed to unschedule pending compaction. Operation successful.
+```
+
+### Repair Compaction
+
+The above compaction unscheduling operations could sometimes fail partially (e:g -> DFS temporarily unavailable). With
+partial failures, the compaction operation could become inconsistent with the state of file-slices. When you run
+`compaction validate`, you can notice invalid compaction operations if there is one. In these cases, the repair
+command comes to the rescue, it will rearrange the file-slices so that there is no loss and the file-slices are
+consistent with the compaction plan
+
+```java
+hudi:stock_ticks_mor->compaction repair --instant 20181005222611
+......
+Compaction successfully repaired
+.....
+```
+
+## Troubleshooting
+
+Section below generally aids in debugging Hudi failures. Off the bat, the following metadata is added to every record to help triage issues easily using standard Hadoop SQL engines (Hive/PrestoDB/Spark)
+
+ - **_hoodie_record_key** - Treated as a primary key within each DFS partition, basis of all updates/inserts
+ - **_hoodie_commit_time** - Last commit that touched this record
+ - **_hoodie_file_name** - Actual file name containing the record (super useful to triage duplicates)
+ - **_hoodie_partition_path** - Path from basePath that identifies the partition containing this record
+
+ For performance related issues, please refer to the [tuning guide](https://cwiki.apache.org/confluence/display/HUDI/Tuning+Guide)
+
+
+### Missing records
+
+Please check if there were any write errors using the admin commands above, during the window at which the record could have been written.
+If you do find errors, then the record was not actually written by Hudi, but handed back to the application to decide what to do with it.
+
+### Duplicates
+
+First of all, please confirm if you do indeed have duplicates **AFTER** ensuring the query is accessing the Hudi table [properly](/docs/querying_data) .
+
+ - If confirmed, please use the metadata fields above, to identify the physical files & partition files containing the records .
+ - If duplicates span files across partitionpath, then this means your application is generating different partitionPaths for same recordKey, Please fix your app
+ - if duplicates span multiple files within the same partitionpath, please engage with mailing list. This should not happen. You can use the `records deduplicate` command to fix your data.
+
+### Spark failures {#spark-ui}
+
+Typical upsert() DAG looks like below. Note that Hudi client also caches intermediate RDDs to intelligently profile workload and size files and spark parallelism.
+Also Spark UI shows sortByKey twice due to the probe job also being shown, nonetheless its just a single sort.
+
+<figure>
+ <img className="docimage" src={require("/assets/images/hudi_upsert_dag.png").default} alt="hudi_upsert_dag.png" />
+</figure>
+
+At a high level, there are two steps
+
+**Index Lookup to identify files to be changed**
+
+ - Job 1 : Triggers the input data read, converts to HoodieRecord object and then stops at obtaining a spread of input records to target partition paths
+ - Job 2 : Load the set of file names which we need check against
+ - Job 3 & 4 : Actual lookup after smart sizing of spark join parallelism, by joining RDDs in 1 & 2 above
+ - Job 5 : Have a tagged RDD of recordKeys with locations
+
+**Performing the actual writing of data**
+
+ - Job 6 : Lazy join of incoming records against recordKey, location to provide a final set of HoodieRecord which now contain the information about which file/partitionpath they are found at (or null if insert). Then also profile the workload again to determine sizing of files
+ - Job 7 : Actual writing of data (update + insert + insert turned to updates to maintain file size)
+
+Depending on the exception source (Hudi/Spark), the above knowledge of the DAG can be used to pinpoint the actual issue. The most often encountered failures result from YARN/DFS temporary failures.
+In the future, a more sophisticated debug/management UI would be added to the project, that can help automate some of this debugging.
diff --git a/website/versioned_docs/version-0.9.0/docker_demo.md b/website/versioned_docs/version-0.9.0/docker_demo.md
new file mode 100644
index 0000000..0f1a194
--- /dev/null
+++ b/website/versioned_docs/version-0.9.0/docker_demo.md
@@ -0,0 +1,1235 @@
+---
+title: Docker Demo
+keywords: [ hudi, docker, demo]
+toc: true
+last_modified_at: 2019-12-30T15:59:57-04:00
+---
+
+## A Demo using docker containers
+
+Lets use a real world example to see how hudi works end to end. For this purpose, a self contained
+data infrastructure is brought up in a local docker cluster within your computer.
+
+The steps have been tested on a Mac laptop
+
+### Prerequisites
+
+ * Docker Setup : For Mac, Please follow the steps as defined in [https://docs.docker.com/v17.12/docker-for-mac/install/]. For running Spark-SQL queries, please ensure atleast 6 GB and 4 CPUs are allocated to Docker (See Docker -> Preferences -> Advanced). Otherwise, spark-SQL queries could be killed because of memory issues.
+ * kafkacat : A command-line utility to publish/consume from kafka topics. Use `brew install kafkacat` to install kafkacat.
+ * /etc/hosts : The demo references many services running in container by the hostname. Add the following settings to /etc/hosts
+
+ ```java
+ 127.0.0.1 adhoc-1
+ 127.0.0.1 adhoc-2
+ 127.0.0.1 namenode
+ 127.0.0.1 datanode1
+ 127.0.0.1 hiveserver
+ 127.0.0.1 hivemetastore
+ 127.0.0.1 kafkabroker
+ 127.0.0.1 sparkmaster
+ 127.0.0.1 zookeeper
+ ```
+ * Java : Java SE Development Kit 8.
+ * Maven : A build automation tool for Java projects.
+ * jq : A lightweight and flexible command-line JSON processor. Use `brew install jq` to install jq.
+
+Also, this has not been tested on some environments like Docker on Windows.
+
+
+## Setting up Docker Cluster
+
+
+### Build Hudi
+
+The first step is to build hudi. **Note** This step builds hudi on default supported scala version - 2.11.
+```java
+cd <HUDI_WORKSPACE>
+mvn package -DskipTests
+```
+
+### Bringing up Demo Cluster
+
+The next step is to run the docker compose script and setup configs for bringing up the cluster.
+This should pull the docker images from docker hub and setup docker cluster.
+
+```java
+cd docker
+./setup_demo.sh
+....
+....
+....
+Stopping spark-worker-1 ... done
+Stopping hiveserver ... done
+Stopping hivemetastore ... done
+Stopping historyserver ... done
+.......
+......
+Creating network "compose_default" with the default driver
+Creating volume "compose_namenode" with default driver
+Creating volume "compose_historyserver" with default driver
+Creating volume "compose_hive-metastore-postgresql" with default driver
+Creating hive-metastore-postgresql ... done
+Creating namenode ... done
+Creating zookeeper ... done
+Creating kafkabroker ... done
+Creating hivemetastore ... done
+Creating historyserver ... done
+Creating hiveserver ... done
+Creating datanode1 ... done
+Creating presto-coordinator-1 ... done
+Creating sparkmaster ... done
+Creating presto-worker-1 ... done
+Creating adhoc-1 ... done
+Creating adhoc-2 ... done
+Creating spark-worker-1 ... done
+Copying spark default config and setting up configs
+Copying spark default config and setting up configs
+$ docker ps
+```
+
+At this point, the docker cluster will be up and running. The demo cluster brings up the following services
+
+ * HDFS Services (NameNode, DataNode)
+ * Spark Master and Worker
+ * Hive Services (Metastore, HiveServer2 along with PostgresDB)
+ * Kafka Broker and a Zookeeper Node (Kafka will be used as upstream source for the demo)
+ * Adhoc containers to run Hudi/Hive CLI commands
+
+## Demo
+
+Stock Tracker data will be used to showcase different Hudi query types and the effects of Compaction.
+
+Take a look at the directory `docker/demo/data`. There are 2 batches of stock data - each at 1 minute granularity.
+The first batch contains stocker tracker data for some stock symbols during the first hour of trading window
+(9:30 a.m to 10:30 a.m). The second batch contains tracker data for next 30 mins (10:30 - 11 a.m). Hudi will
+be used to ingest these batches to a table which will contain the latest stock tracker data at hour level granularity.
+The batches are windowed intentionally so that the second batch contains updates to some of the rows in the first batch.
+
+### Step 1 : Publish the first batch to Kafka
+
+Upload the first batch to Kafka topic 'stock ticks' `cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks -P`
+
+To check if the new topic shows up, use
+```java
+kafkacat -b kafkabroker -L -J | jq .
+{
+ "originating_broker": {
+ "id": 1001,
+ "name": "kafkabroker:9092/1001"
+ },
+ "query": {
+ "topic": "*"
+ },
+ "brokers": [
+ {
+ "id": 1001,
+ "name": "kafkabroker:9092"
+ }
+ ],
+ "topics": [
+ {
+ "topic": "stock_ticks",
+ "partitions": [
+ {
+ "partition": 0,
+ "leader": 1001,
+ "replicas": [
+ {
+ "id": 1001
+ }
+ ],
+ "isrs": [
+ {
+ "id": 1001
+ }
+ ]
+ }
+ ]
+ }
+ ]
+}
+```
+
+### Step 2: Incrementally ingest data from Kafka topic
+
+Hudi comes with a tool named DeltaStreamer. This tool can connect to variety of data sources (including Kafka) to
+pull changes and apply to Hudi table using upsert/insert primitives. Here, we will use the tool to download
+json data from kafka topic and ingest to both COW and MOR tables we initialized in the previous step. This tool
+automatically initializes the tables in the file-system if they do not exist yet.
+
+```java
+docker exec -it adhoc-2 /bin/bash
+
+# Run the following spark-submit command to execute the delta-streamer and ingest to stock_ticks_cow table in HDFS
+spark-submit \
+ --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE \
+ --table-type COPY_ON_WRITE \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_cow \
+ --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+
+# Run the following spark-submit command to execute the delta-streamer and ingest to stock_ticks_mor table in HDFS
+spark-submit \
+ --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
+ --disable-compaction
+
+# As part of the setup (Look at setup_demo.sh), the configs needed for DeltaStreamer is uploaded to HDFS. The configs
+# contain mostly Kafa connectivity settings, the avro-schema to be used for ingesting along with key and partitioning fields.
+
+exit
+```
+
+You can use HDFS web-browser to look at the tables
+`http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow`.
+
+You can explore the new partition folder created in the table along with a "commit" / "deltacommit"
+file under .hoodie which signals a successful commit.
+
+There will be a similar setup when you browse the MOR table
+`http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_mor`
+
+
+### Step 3: Sync with Hive
+
+At this step, the tables are available in HDFS. We need to sync with Hive to create new Hive tables and add partitions
+inorder to run Hive queries against those tables.
+
+```java
+docker exec -it adhoc-2 /bin/bash
+
+# This command takes in HiveServer URL and COW Hudi table location in HDFS and sync the HDFS state to Hive
+/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
+ --jdbc-url jdbc:hive2://hiveserver:10000 \
+ --user hive \
+ --pass hive \
+ --partitioned-by dt \
+ --base-path /user/hive/warehouse/stock_ticks_cow \
+ --database default \
+ --table stock_ticks_cow
+.....
+2020-01-25 19:51:28,953 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_cow
+.....
+
+# Now run hive-sync for the second data-set in HDFS using Merge-On-Read (MOR table type)
+/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
+ --jdbc-url jdbc:hive2://hiveserver:10000 \
+ --user hive \
+ --pass hive \
+ --partitioned-by dt \
+ --base-path /user/hive/warehouse/stock_ticks_mor \
+ --database default \
+ --table stock_ticks_mor
+...
+2020-01-25 19:51:51,066 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_mor_ro
+...
+2020-01-25 19:51:51,569 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_mor_rt
+....
+
+exit
+```
+After executing the above command, you will notice
+
+1. A hive table named `stock_ticks_cow` created which supports Snapshot and Incremental queries on Copy On Write table.
+2. Two new tables `stock_ticks_mor_rt` and `stock_ticks_mor_ro` created for the Merge On Read table. The former
+supports Snapshot and Incremental queries (providing near-real time data) while the later supports ReadOptimized queries.
+
+
+### Step 4 (a): Run Hive Queries
+
+Run a hive query to find the latest timestamp ingested for stock symbol 'GOOG'. You will notice that both snapshot
+(for both COW and MOR _rt table) and read-optimized queries (for MOR _ro table) give the same value "10:29 a.m" as Hudi create a
+parquet file for the first batch of data.
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 \
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
+ --hiveconf hive.stats.autogather=false
+
+# List Tables
+0: jdbc:hive2://hiveserver:10000> show tables;
++---------------------+--+
+| tab_name |
++---------------------+--+
+| stock_ticks_cow |
+| stock_ticks_mor_ro |
+| stock_ticks_mor_rt |
++---------------------+--+
+3 rows selected (1.199 seconds)
+0: jdbc:hive2://hiveserver:10000>
+
+
+# Look at partitions that were added
+0: jdbc:hive2://hiveserver:10000> show partitions stock_ticks_mor_rt;
++----------------+--+
+| partition |
++----------------+--+
+| dt=2018-08-31 |
++----------------+--+
+1 row selected (0.24 seconds)
+
+
+# COPY-ON-WRITE Queries:
+=========================
+
+
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+
+Now, run a projection query:
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924221953 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924221953 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against M-O-R table. Lets look at both
+ReadOptimized and Snapshot(realtime data) queries supported by M-O-R table
+
+# Run ReadOptimized Query. Notice that the latest timestamp is 10:29
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (6.326 seconds)
+
+
+# Run Snapshot Query. Notice that the latest timestamp is again 10:29
+
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (1.606 seconds)
+
+
+# Run Read Optimized and Snapshot project queries
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+exit
+```
+
+### Step 4 (b): Run Spark-SQL Queries
+Hudi support Spark as query processor just like Hive. Here are the same hive queries
+running in spark-sql
+
+```java
+docker exec -it adhoc-1 /bin/bash
+$SPARK_INSTALL/bin/spark-shell \
+ --jars $HUDI_SPARK_BUNDLE \
+ --master local[2] \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --deploy-mode client \
+ --driver-memory 1G \
+ --executor-memory 3G \
+ --num-executors 1 \
+ --packages org.apache.spark:spark-avro_2.11:2.4.4
+...
+
+Welcome to
+ ____ __
+ / __/__ ___ _____/ /__
+ _\ \/ _ \/ _ `/ __/ '_/
+ /___/ .__/\_,_/_/ /_/\_\ version 2.4.4
+ /_/
+
+Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
+Type in expressions to have them evaluated.
+Type :help for more information.
+
+scala> spark.sql("show tables").show(100, false)
++--------+------------------+-----------+
+|database|tableName |isTemporary|
++--------+------------------+-----------+
+|default |stock_ticks_cow |false |
+|default |stock_ticks_mor_ro|false |
+|default |stock_ticks_mor_rt|false |
++--------+------------------+-----------+
+
+# Copy-On-Write Table
+
+## Run max timestamp query against COW table
+
+scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
+[Stage 0:> (0 + 1) / 1]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
+SLF4J: Defaulting to no-operation (NOP) logger implementation
+SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:29:00|
++------+-------------------+
+
+## Projection Query
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG'").show(100, false)
++-------------------+------+-------------------+------+---------+--------+
+|_hoodie_commit_time|symbol|ts |volume|open |close |
++-------------------+------+-------------------+------+---------+--------+
+|20180924221953 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
+|20180924221953 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
++-------------------+------+-------------------+------+---------+--------+
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against M-O-R table. Lets look at both
+ReadOptimized and Snapshot queries supported by M-O-R table
+
+# Run ReadOptimized Query. Notice that the latest timestamp is 10:29
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:29:00|
++------+-------------------+
+
+
+# Run Snapshot Query. Notice that the latest timestamp is again 10:29
+
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:29:00|
++------+-------------------+
+
+# Run Read Optimized and Snapshot project queries
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG'").show(100, false)
++-------------------+------+-------------------+------+---------+--------+
+|_hoodie_commit_time|symbol|ts |volume|open |close |
++-------------------+------+-------------------+------+---------+--------+
+|20180924222155 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
+|20180924222155 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
++-------------------+------+-------------------+------+---------+--------+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
++-------------------+------+-------------------+------+---------+--------+
+|_hoodie_commit_time|symbol|ts |volume|open |close |
++-------------------+------+-------------------+------+---------+--------+
+|20180924222155 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
+|20180924222155 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
++-------------------+------+-------------------+------+---------+--------+
+```
+
+### Step 4 (c): Run Presto Queries
+
+Here are the Presto queries for similar Hive and Spark queries. Currently, Presto does not support snapshot or incremental queries on Hudi tables.
+
+```java
+docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
+presto> show catalogs;
+ Catalog
+-----------
+ hive
+ jmx
+ localfile
+ system
+(4 rows)
+
+Query 20190817_134851_00000_j8rcz, FINISHED, 1 node
+Splits: 19 total, 19 done (100.00%)
+0:04 [0 rows, 0B] [0 rows/s, 0B/s]
+
+presto> use hive.default;
+USE
+presto:default> show tables;
+ Table
+--------------------
+ stock_ticks_cow
+ stock_ticks_mor_ro
+ stock_ticks_mor_rt
+(3 rows)
+
+Query 20190822_181000_00001_segyw, FINISHED, 2 nodes
+Splits: 19 total, 19 done (100.00%)
+0:05 [3 rows, 99B] [0 rows/s, 18B/s]
+
+
+# COPY-ON-WRITE Queries:
+=========================
+
+
+presto:default> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20190822_181011_00002_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:12 [197 rows, 613B] [16 rows/s, 50B/s]
+
+presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180221 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822180221 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20190822_181141_00003_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [109 rows/s, 341B/s]
+
+
+# Merge-On-Read Queries:
+==========================
+
+Lets run similar queries against M-O-R table.
+
+# Run ReadOptimized Query. Notice that the latest timestamp is 10:29
+ presto:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20190822_181158_00004_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:02 [197 rows, 613B] [110 rows/s, 343B/s]
+
+
+presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180250 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822180250 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20190822_181256_00006_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [92 rows/s, 286B/s]
+
+presto:default> exit
+```
+
+### Step 5: Upload second batch to Kafka and run DeltaStreamer to ingest
+
+Upload the second batch of data and ingest this batch using delta-streamer. As this batch does not bring in any new
+partitions, there is no need to run hive-sync
+
+```java
+cat docker/demo/data/batch_2.json | kafkacat -b kafkabroker -t stock_ticks -P
+
+# Within Docker container, run the ingestion command
+docker exec -it adhoc-2 /bin/bash
+
+# Run the following spark-submit command to execute the delta-streamer and ingest to stock_ticks_cow table in HDFS
+spark-submit \
+ --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE \
+ --table-type COPY_ON_WRITE \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_cow \
+ --target-table stock_ticks_cow \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+
+# Run the following spark-submit command to execute the delta-streamer and ingest to stock_ticks_mor table in HDFS
+spark-submit \
+ --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
+ --disable-compaction
+
+exit
+```
+
+With Copy-On-Write table, the second ingestion by DeltaStreamer resulted in a new version of Parquet file getting created.
+See `http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow/2018/08/31`
+
+With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file.
+Take a look at the HDFS filesystem to get an idea: `http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_mor/2018/08/31`
+
+### Step 6 (a): Run Hive Queries
+
+With Copy-On-Write table, the Snapshot query immediately sees the changes as part of second batch once the batch
+got committed as each ingestion creates newer versions of parquet files.
+
+With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file.
+This is the time, when ReadOptimized and Snapshot queries will provide different results. ReadOptimized query will still
+return "10:29 am" as it will only read from the Parquet file. Snapshot query will do on-the-fly merge and return
+latest committed data which is "10:59 a.m".
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 \
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
+ --hiveconf hive.stats.autogather=false
+
+# Copy On Write Table:
+
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+1 row selected (1.932 seconds)
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924221953 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224524 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+
+# Merge On Read Table:
+
+# Read Optimized Query
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+--+
+1 row selected (1.6 seconds)
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+# Snapshot Query
+0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
+WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
++---------+----------------------+--+
+| symbol | _c1 |
++---------+----------------------+--+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+--+
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224537 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+exit
+```
+
+### Step 6 (b): Run Spark SQL Queries
+
+Running the same queries in Spark-SQL:
+
+```java
+docker exec -it adhoc-1 /bin/bash
+$SPARK_INSTALL/bin/spark-shell \
+ --jars $HUDI_SPARK_BUNDLE \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --deploy-mode client \
+ --driver-memory 1G \
+ --master local[2] \
+ --executor-memory 3G \
+ --num-executors 1 \
+ --packages org.apache.spark:spark-avro_2.11:2.4.4
+
+# Copy On Write Table:
+
+scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
++------+-------------------+
+|symbol|max(ts) |
++------+-------------------+
+|GOOG |2018-08-31 10:59:00|
++------+-------------------+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG'").show(100, false)
+
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924221953 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224524 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+
+# Merge On Read Table:
+
+# Read Optimized Query
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+
+| symbol | _c1 |
++---------+----------------------+
+| GOOG | 2018-08-31 10:29:00 |
++---------+----------------------+
+1 row selected (1.6 seconds)
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924222155 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
++----------------------+---------+----------------------+---------+------------+-----------+
+
+# Snapshot Query
+scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
++---------+----------------------+
+| symbol | _c1 |
++---------+----------------------+
+| GOOG | 2018-08-31 10:59:00 |
++---------+----------------------+
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
++----------------------+---------+----------------------+---------+------------+-----------+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+
+| 20180924222155 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924224537 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+
+
+exit
+```
+
+### Step 6 (c): Run Presto Queries
+
+Running the same queries on Presto for ReadOptimized queries.
+
+```java
+docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
+presto> use hive.default;
+USE
+
+# Copy On Write Table:
+
+presto:default>select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:59:00
+(1 row)
+
+Query 20190822_181530_00007_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:02 [197 rows, 613B] [125 rows/s, 389B/s]
+
+presto:default>select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180221 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822181433 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215
+(2 rows)
+
+Query 20190822_181545_00008_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:02 [197 rows, 613B] [106 rows/s, 332B/s]
+
+As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
+
+
+# Merge On Read Table:
+
+# Read Optimized Query
+presto:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
+ symbol | _col1
+--------+---------------------
+ GOOG | 2018-08-31 10:29:00
+(1 row)
+
+Query 20190822_181602_00009_segyw, FINISHED, 1 node
+Splits: 49 total, 49 done (100.00%)
+0:01 [197 rows, 613B] [139 rows/s, 435B/s]
+
+presto:default>select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+ _hoodie_commit_time | symbol | ts | volume | open | close
+---------------------+--------+---------------------+--------+-----------+----------
+ 20190822180250 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02
+ 20190822180250 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085
+(2 rows)
+
+Query 20190822_181615_00010_segyw, FINISHED, 1 node
+Splits: 17 total, 17 done (100.00%)
+0:01 [197 rows, 613B] [154 rows/s, 480B/s]
+
+presto:default> exit
+```
+
+### Step 7 (a): Incremental Query for COPY-ON-WRITE Table
+
+With 2 batches of data ingested, lets showcase the support for incremental queries in Hudi Copy-On-Write tables
+
+Lets take the same projection query example
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 \
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
+ --hiveconf hive.stats.autogather=false
+
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924064621 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
+| 20180924065039 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+```
+
+As you notice from the above queries, there are 2 commits - 20180924064621 and 20180924065039 in timeline order.
+When you follow the steps, you will be getting different timestamps for commits. Substitute them
+in place of the above timestamps.
+
+To show the effects of incremental-query, let us assume that a reader has already seen the changes as part of
+ingesting first batch. Now, for the reader to see effect of the second batch, he/she has to keep the start timestamp to
+the commit time of the first batch (20180924064621) and run incremental query
+
+Hudi incremental mode provides efficient scanning for incremental queries by filtering out files that do not have any
+candidate rows using hudi-managed metadata.
+
+```java
+docker exec -it adhoc-2 /bin/bash
+beeline -u jdbc:hive2://hiveserver:10000 \
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
+ --hiveconf hive.stats.autogather=false
+
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.mode=INCREMENTAL;
+No rows affected (0.009 seconds)
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.max.commits=3;
+No rows affected (0.009 seconds)
+0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.start.timestamp=20180924064621;
+```
+
+With the above setting, file-ids that do not have any updates from the commit 20180924065039 is filtered out without scanning.
+Here is the incremental query :
+
+```java
+0: jdbc:hive2://hiveserver:10000>
+0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG' and `_hoodie_commit_time` > '20180924064621';
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+| 20180924065039 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+--+
+1 row selected (0.83 seconds)
+0: jdbc:hive2://hiveserver:10000>
+```
+
+### Step 7 (b): Incremental Query with Spark SQL:
+
+```java
+docker exec -it adhoc-1 /bin/bash
+$SPARK_INSTALL/bin/spark-shell \
+ --jars $HUDI_SPARK_BUNDLE \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --deploy-mode client \
+ --driver-memory 1G \
+ --master local[2] \
+ --executor-memory 3G \
+ --num-executors 1 \
+ --packages org.apache.spark:spark-avro_2.11:2.4.4
+
+Welcome to
+ ____ __
+ / __/__ ___ _____/ /__
+ _\ \/ _ \/ _ `/ __/ '_/
+ /___/ .__/\_,_/_/ /_/\_\ version 2.4.4
+ /_/
+
+Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
+Type in expressions to have them evaluated.
+Type :help for more information.
+
+scala> import org.apache.hudi.DataSourceReadOptions
+import org.apache.hudi.DataSourceReadOptions
+
+# In the below query, 20180925045257 is the first commit's timestamp
+scala> val hoodieIncViewDF = spark.read.format("org.apache.hudi").option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL).option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY, "20180924064621").load("/user/hive/warehouse/stock_ticks_cow")
+SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
+SLF4J: Defaulting to no-operation (NOP) logger implementation
+SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
+hoodieIncViewDF: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 15 more fields]
+
+scala> hoodieIncViewDF.registerTempTable("stock_ticks_cow_incr_tmp1")
+warning: there was one deprecation warning; re-run with -deprecation for details
+
+scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow_incr_tmp1 where symbol = 'GOOG'").show(100, false);
++----------------------+---------+----------------------+---------+------------+-----------+
+| _hoodie_commit_time | symbol | ts | volume | open | close |
++----------------------+---------+----------------------+---------+------------+-----------+
+| 20180924065039 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
++----------------------+---------+----------------------+---------+------------+-----------+
+```
+
+### Step 8: Schedule and Run Compaction for Merge-On-Read table
+
+Lets schedule and run a compaction to create a new version of columnar file so that read-optimized readers will see fresher data.
+Again, You can use Hudi CLI to manually schedule and run compaction
+
+```java
+docker exec -it adhoc-1 /bin/bash
+root@adhoc-1:/opt# /var/hoodie/ws/hudi-cli/hudi-cli.sh
+...
+Table command getting loaded
+HoodieSplashScreen loaded
+===================================================================
+* ___ ___ *
+* /\__\ ___ /\ \ ___ *
+* / / / /\__\ / \ \ /\ \ *
+* / /__/ / / / / /\ \ \ \ \ \ *
+* / \ \ ___ / / / / / \ \__\ / \__\ *
+* / /\ \ /\__\ / /__/ ___ / /__/ \ |__| / /\/__/ *
+* \/ \ \/ / / \ \ \ /\__\ \ \ \ / / / /\/ / / *
+* \ / / \ \ / / / \ \ / / / \ /__/ *
+* / / / \ \/ / / \ \/ / / \ \__\ *
+* / / / \ / / \ / / \/__/ *
+* \/__/ \/__/ \/__/ Apache Hudi CLI *
+* *
+===================================================================
+
+Welcome to Apache Hudi CLI. Please type help if you are looking for help.
+hudi->connect --path /user/hive/warehouse/stock_ticks_mor
+18/09/24 06:59:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+18/09/24 06:59:35 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
+18/09/24 06:59:35 INFO util.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://namenode:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1261652683_11, ugi=root (auth:SIMPLE)]]]
+18/09/24 06:59:35 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
+18/09/24 06:59:36 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1) from /user/hive/warehouse/stock_ticks_mor
+Metadata for table stock_ticks_mor loaded
+hoodie:stock_ticks_mor->compactions show all
+20/02/10 03:41:32 INFO timeline.HoodieActiveTimeline: Loaded instants [[20200210015059__clean__COMPLETED], [20200210015059__deltacommit__COMPLETED], [20200210022758__clean__COMPLETED], [20200210022758__deltacommit__COMPLETED], [==>20200210023843__compaction__REQUESTED]]
+___________________________________________________________________
+| Compaction Instant Time| State | Total FileIds to be Compacted|
+|==================================================================|
+
+# Schedule a compaction. This will use Spark Launcher to schedule compaction
+hoodie:stock_ticks_mor->compaction schedule
+....
+Compaction successfully completed for 20180924070031
+
+# Now refresh and check again. You will see that there is a new compaction requested
+
+hoodie:stock_ticks->connect --path /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:01:16 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:01:16 INFO util.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://namenode:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1261652683_11, ugi=root (auth:SIMPLE)]]]
+18/09/24 07:01:16 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
+18/09/24 07:01:16 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1) from /user/hive/warehouse/stock_ticks_mor
+Metadata for table stock_ticks_mor loaded
+
+hoodie:stock_ticks_mor->compactions show all
+18/09/24 06:34:12 INFO timeline.HoodieActiveTimeline: Loaded instants [[20180924041125__clean__COMPLETED], [20180924041125__deltacommit__COMPLETED], [20180924042735__clean__COMPLETED], [20180924042735__deltacommit__COMPLETED], [==>20180924063245__compaction__REQUESTED]]
+___________________________________________________________________
+| Compaction Instant Time| State | Total FileIds to be Compacted|
+|==================================================================|
+| 20180924070031 | REQUESTED| 1 |
+
+# Execute the compaction. The compaction instant value passed below must be the one displayed in the above "compactions show all" query
+hoodie:stock_ticks_mor->compaction run --compactionInstant 20180924070031 --parallelism 2 --sparkMemory 1G --schemaFilePath /var/demo/config/schema.avsc --retry 1
+....
+Compaction successfully completed for 20180924070031
+
+## Now check if compaction is completed
+
+hoodie:stock_ticks_mor->connect --path /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:03:00 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
+18/09/24 07:03:00 INFO util.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://namenode:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1261652683_11, ugi=root (auth:SIMPLE)]]]
+18/09/24 07:03:00 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
+18/09/24 07:03:00 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1) from /user/hive/warehouse/stock_ticks_mor
+Metadata for table stock_ticks_mor loaded
+
+hoodie:stock_ticks->compactions show all
+18/09/24 07:03:15 INFO timeline.HoodieActiveTimeline: Loaded instants [[20180924064636__clean__COMPLETED], [20180924064636__deltacommit__COMPLETED], [20180924065057__clean__COMPLETED], [20180924065057__deltacommit__COMPLETED], [20180924070031__commit__COMPLETED]]
+___________________________________________________________________
... 3883 lines suppressed ...