You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by yi...@apache.org on 2024/12/07 02:06:14 UTC
(hudi) branch asf-site updated: [HUDI-8672][DOCS] Update configuration docs based on release-1.0.0 branch (#12439)
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 741f603b7bd [HUDI-8672][DOCS] Update configuration docs based on release-1.0.0 branch (#12439)
741f603b7bd is described below
commit 741f603b7bd79814a7b6ae548216e48929eda94e
Author: Y Ethan Guo <et...@gmail.com>
AuthorDate: Fri Dec 6 18:06:03 2024 -0800
[HUDI-8672][DOCS] Update configuration docs based on release-1.0.0 branch (#12439)
---
website/docs/basic_configurations.md | 186 +++---
website/docs/configurations.md | 1046 ++++++++++++++++++----------------
2 files changed, 658 insertions(+), 574 deletions(-)
diff --git a/website/docs/basic_configurations.md b/website/docs/basic_configurations.md
index 08d5ac717d9..84f589d84db 100644
--- a/website/docs/basic_configurations.md
+++ b/website/docs/basic_configurations.md
@@ -1,7 +1,7 @@
---
title: Basic Configurations
summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](/docs/configurations) page.
-last_modified_at: 2024-07-01T15:09:57.63
+last_modified_at: 2024-12-06T17:38:05.854
---
@@ -33,37 +33,44 @@ Configurations of the Hudi Table like type of ingestion, storage formats, hive t
[**Basic Configs**](#Hudi-Table-Basic-Configs-basic-configs)
-| Config Name | Default | Description [...]
-| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BOOTSTRAP_BASE_PATH` [...]
-| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name that will be used for incremental query.If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME` [...]
-| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.<br />`Config Param: TABLE_CHECKSUM`<br />`Since Version: 0.11.0` [...]
-| [hoodie.table.create.schema](#hoodietablecreateschema) | (N/A) | Schema used when creating the table, for the first time.<br />`Config Param: CREATE_SCHEMA` [...]
-| [hoodie.table.keygenerator.class](#hoodietablekeygeneratorclass) | (N/A) | Key Generator class property for the hoodie table<br />`Config Param: KEY_GENERATOR_CLASS_NAME` [...]
-| [hoodie.table.metadata.partitions](#hoodietablemetadatapartitions) | (N/A) | Comma-separated list of metadata partitions that have been completely built and in-sync with data table. These partitions are ready for use by the readers<br />`Config Param: TABLE_METADATA_PARTITIONS`<br />`Since Version: 0.11.0` [...]
-| [hoodie.table.metadata.partitions.inflight](#hoodietablemetadatapartitionsinflight) | (N/A) | Comma-separated list of metadata partitions whose building is in progress. These partitions are not yet ready for use by the readers.<br />`Config Param: TABLE_METADATA_PARTITIONS_INFLIGHT`<br />`Since Version: 0.11.0` [...]
-| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: NAME` [...]
-| [hoodie.table.partition.fields](#hoodietablepartitionfields) | (N/A) | Fields used to partition the table. Concatenated values of these fields are used as the partition path, by invoking toString()<br />`Config Param: PARTITION_FIELDS` [...]
-| [hoodie.table.precombine.field](#hoodietableprecombinefield) | (N/A) | Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br />`Config Param: PRECOMBINE_FIELD` [...]
-| [hoodie.table.recordkey.fields](#hoodietablerecordkeyfields) | (N/A) | Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br />`Config Param: RECORDKEY_FIELDS` [...]
-| [hoodie.table.secondary.indexes.metadata](#hoodietablesecondaryindexesmetadata) | (N/A) | The metadata of secondary indexes<br />`Config Param: SECONDARY_INDEXES_METADATA`<br />`Since Version: 0.13.0` [...]
-| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | (N/A) | Version of timeline used, by the table.<br />`Config Param: TIMELINE_LAYOUT_VERSION` [...]
-| [hoodie.archivelog.folder](#hoodiearchivelogfolder) | archived | path under the meta folder, to store archived timeline instants at.<br />`Config Param: ARCHIVELOG_FOLDER` [...]
-| [hoodie.bootstrap.index.class](#hoodiebootstrapindexclass) | org.apache.hudi.common.bootstrap.index.hfile.HFileBootstrapIndex | Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_CLASS_NAME` [...]
-| [hoodie.bootstrap.index.enable](#hoodiebootstrapindexenable) | true | Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined, default true.<br />`Config Param: BOOTSTRAP_INDEX_ENABLE` [...]
-| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | org.apache.hudi.common.model.DefaultHoodieRecordPayload | Payload class to use for performing compactions, i.e merge delta logs with current base file and then produce a new base file.<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
-| [hoodie.compaction.payload.type](#hoodiecompactionpayloadtype) | HOODIE_AVRO_DEFAULT | org.apache.hudi.common.model.RecordPayloadType: Payload to use for merging records AWS_DMS_AVRO: Provides support for seamlessly applying changes captured via Amazon Database Migration Service onto S3. HOODIE_AVRO: A payload to wrap a existing Hoodie Avro Record. Useful to create a HoodieRecord over existing Gener [...]
-| [hoodie.compaction.record.merger.strategy](#hoodiecompactionrecordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in hoodie.datasource.write.record.merger.impls which has the same merger strategy id<br />`Config Param: RECORD_MERGER_STRATEGY`<br />`Since Version: 0.13.0` [...]
-| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` [...]
-| [hoodie.partition.metafile.use.base.format](#hoodiepartitionmetafileusebaseformat) | false | If true, partition metafiles are saved in the same format as base-files for this dataset (e.g. Parquet / ORC). If false (default) partition metafiles are saved as properties files.<br />`Config Param: PARTITION_METAFILE_USE_BASE_FORMAT` [...]
-| [hoodie.populate.meta.fields](#hoodiepopulatemetafields) | true | When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br />`Config Param: POPULATE_META_FIELDS` [...]
-| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | Base file format to store all the base file data.<br />`Config Param: BASE_FILE_FORMAT` [...]
-| [hoodie.table.cdc.enabled](#hoodietablecdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode.<br />`Config Param: CDC_ENABLED`<br />`Since Version: 0.13.0` [...]
-| [hoodie.table.cdc.supplemental.logging.mode](#hoodietablecdcsupplementalloggingmode) | DATA_BEFORE_AFTER | org.apache.hudi.common.table.cdc.HoodieCDCSupplementalLoggingMode: Change log capture supplemental logging mode. The supplemental log is used for accelerating the generation of change log details. OP_KEY_ONLY: Only keeping record keys in the supplemental logs, so the reader needs to figure out the update before image [...]
-| [hoodie.table.log.file.format](#hoodietablelogfileformat) | HOODIE_LOG | Log format used for the delta logs.<br />`Config Param: LOG_FILE_FORMAT` [...]
-| [hoodie.table.timeline.timezone](#hoodietabletimelinetimezone) | LOCAL | User can set hoodie commit timeline timezone, such as utc, local and so on. local is default<br />`Config Param: TIMELINE_TIMEZONE` [...]
-| [hoodie.table.type](#hoodietabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TYPE` [...]
-| [hoodie.table.version](#hoodietableversion) | ZERO | Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br />`Config Param: VERSION` [...]
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BOOTSTRAP_BASE_PATH` [...]
+| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | (N/A) | Payload class to use for performing merges, compactions, i.e merge delta logs with current base file and then produce a new base file.<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name. If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME` [...]
+| [hoodie.record.merge.strategy.id](#hoodierecordmergestrategyid) | (N/A) | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in `hoodie.write.record.merge.custom.implementation.classes` which has the same merger strategy id<br />`Config Param: RECORD_MERGE_STRATEGY_ID`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.<br />`Config Param: TABLE_CHECKSUM`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.create.schema](#hoodietablecreateschema) | (N/A) | Schema used when creating the table<br />`Config Param: CREATE_SCHEMA` [...]
+| [hoodie.table.index.defs.path](#hoodietableindexdefspath) | (N/A) | Relative path to table base path where the index definitions are stored<br />`Config Param: RELATIVE_INDEX_DEFINITION_PATH`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.keygenerator.class](#hoodietablekeygeneratorclass) | (N/A) | Key Generator class property for the hoodie table<br />`Config Param: KEY_GENERATOR_CLASS_NAME` [...]
+| [hoodie.table.keygenerator.type](#hoodietablekeygeneratortype) | (N/A) | Key Generator type to determine key generator class<br />`Config Param: KEY_GENERATOR_TYPE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.metadata.partitions](#hoodietablemetadatapartitions) | (N/A) | Comma-separated list of metadata partitions that have been completely built and in-sync with data table. These partitions are ready for use by the readers<br />`Config Param: TABLE_METADATA_PARTITIONS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.metadata.partitions.inflight](#hoodietablemetadatapartitionsinflight) | (N/A) | Comma-separated list of metadata partitions whose building is in progress. These partitions are not yet ready for use by the readers.<br />`Config Param: TABLE_METADATA_PARTITIONS_INFLIGHT`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: NAME` [...]
+| [hoodie.table.partition.fields](#hoodietablepartitionfields) | (N/A) | Comma separated field names used to partition the table. These field names also include the partition type which is used by custom key generators<br />`Config Param: PARTITION_FIELDS` [...]
+| [hoodie.table.precombine.field](#hoodietableprecombinefield) | (N/A) | Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.table.recordkey.fields](#hoodietablerecordkeyfields) | (N/A) | Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br />`Config Param: RECORDKEY_FIELDS` [...]
+| [hoodie.table.secondary.indexes.metadata](#hoodietablesecondaryindexesmetadata) | (N/A) | The metadata of secondary indexes<br />`Config Param: SECONDARY_INDEXES_METADATA`<br />`Since Version: 0.13.0` [...]
+| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | (N/A) | Version of timeline used, by the table.<br />`Config Param: TIMELINE_LAYOUT_VERSION` [...]
+| [hoodie.archivelog.folder](#hoodiearchivelogfolder) | archived | path under the meta folder, to store archived timeline instants at.<br />`Config Param: ARCHIVELOG_FOLDER` [...]
+| [hoodie.bootstrap.index.class](#hoodiebootstrapindexclass) | org.apache.hudi.common.bootstrap.index.hfile.HFileBootstrapIndex | Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_CLASS_NAME` [...]
+| [hoodie.bootstrap.index.enable](#hoodiebootstrapindexenable) | true | Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined, default true.<br />`Config Param: BOOTSTRAP_INDEX_ENABLE` [...]
+| [hoodie.bootstrap.index.type](#hoodiebootstrapindextype) | HFILE | Bootstrap index type determines which implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_TYPE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` [...]
+| [hoodie.partition.metafile.use.base.format](#hoodiepartitionmetafileusebaseformat) | false | If true, partition metafiles are saved in the same format as base-files for this dataset (e.g. Parquet / ORC). If false (default) partition metafiles are saved as properties files.<br />`Config Param: PARTITION_METAFILE_USE_BASE_FORMAT` [...]
+| [hoodie.populate.meta.fields](#hoodiepopulatemetafields) | true | When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br />`Config Param: POPULATE_META_FIELDS` [...]
+| [hoodie.record.merge.mode](#hoodierecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge reco [...]
+| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | Base file format to store all the base file data.<br />`Config Param: BASE_FILE_FORMAT` [...]
+| [hoodie.table.cdc.enabled](#hoodietablecdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode.<br />`Config Param: CDC_ENABLED`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.cdc.supplemental.logging.mode](#hoodietablecdcsupplementalloggingmode) | DATA_BEFORE_AFTER | org.apache.hudi.common.table.cdc.HoodieCDCSupplementalLoggingMode: Change log capture supplemental logging mode. The supplemental log is used for accelerating the generation of change log details. OP_KEY_ONLY: Only keeping record keys in the supplemental logs, so the reader needs to figure out the update before image [...]
+| [hoodie.table.initial.version](#hoodietableinitialversion) | EIGHT | Initial Version of table when the table was created. Used for upgrade/downgrade to identify what upgrade/downgrade paths happened on the table. This is only configured when the table is initially setup.<br />`Config Param: INITIAL_VERSION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.log.file.format](#hoodietablelogfileformat) | HOODIE_LOG | Log format used for the delta logs.<br />`Config Param: LOG_FILE_FORMAT` [...]
+| [hoodie.table.multiple.base.file.formats.enable](#hoodietablemultiplebasefileformatsenable) | false | When set to true, the table can support reading and writing multiple base file formats.<br />`Config Param: MULTIPLE_BASE_FILE_FORMATS_ENABLE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.timeline.timezone](#hoodietabletimelinetimezone) | LOCAL | User can set hoodie commit timeline timezone, such as utc, local and so on. local is default<br />`Config Param: TIMELINE_TIMEZONE` [...]
+| [hoodie.table.type](#hoodietabletype) | COPY_ON_WRITE | The table type for the underlying data.<br />`Config Param: TYPE` [...]
+| [hoodie.table.version](#hoodietableversion) | EIGHT | Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br />`Config Param: VERSION` [...]
+| [hoodie.timeline.history.path](#hoodietimelinehistorypath) | history | path under the meta folder, to store timeline history at.<br />`Config Param: TIMELINE_HISTORY_PATH` [...]
+| [hoodie.timeline.path](#hoodietimelinepath) | timeline | path under the meta folder, to store timeline instants at.<br />`Config Param: TIMELINE_PATH` [...]
---
## Spark Datasource Configs {#SPARK_DATASOURCE}
@@ -80,12 +87,14 @@ Options useful for reading tables via `read.format.option(...)`
[**Basic Configs**](#Read-Options-basic-configs)
-| Config Name | Default | Description [...]
-| --------------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.datasource.read.begin.instanttime](#hoodiedatasourcereadbegininstanttime) | (N/A) | Required when `hoodie.datasource.query.type` is set to `incremental`. Represents the instant time to start incrementally pulling data from. The instanttime here need not necessarily correspond to an instant on the timeline. New data written with an instant_time > BEGIN_INSTANTTIME are fetched out. For e.g: ‘20170901080000’ will get all new data written after Sep 1, 2017 08:00AM. Note that [...]
-| [hoodie.datasource.read.end.instanttime](#hoodiedatasourcereadendinstanttime) | (N/A) | Used when `hoodie.datasource.query.type` is set to `incremental`. Represents the instant time to limit incrementally fetched data to. When not specified latest commit time from timeline is assumed by default. When specified, new data written with an instant_time <= END_INSTANTTIME are fetched out. Point in time type queries make more sense with begin and end instant times specified. Note [...]
-| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files)<br />`Config Param: QUERY_TYPE` [...]
-| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: READ_PRE_COMBINE_FIELD` [...]
+| Config Name | Default | Description [...]
+| -------------------------------------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.read.begin.instanttime](#hoodiedatasourcereadbegininstanttime) | (N/A) | Required when `hoodie.datasource.query.type` is set to `incremental`. Represents the completion time to start incrementally pulling data from. The completion time here need not necessarily correspond to an instant on the timeline. New data written with completion_time >= START_COMMIT are fetched out. For e.g: ‘20170901080000’ will get all new data written on or after Sep 1, 201 [...]
+| [hoodie.datasource.read.end.instanttime](#hoodiedatasourcereadendinstanttime) | (N/A) | Used when `hoodie.datasource.query.type` is set to `incremental`. Represents the completion time to limit incrementally fetched data to. When not specified latest commit completion time from timeline is assumed by default. When specified, new data written with completion_time <= END_COMMIT are fetched out. Point in time type queries make more sense with begin and end completion [...]
+| [hoodie.datasource.read.incr.table.version](#hoodiedatasourcereadincrtableversion) | (N/A) | The table version assumed for incremental read<br />`Config Param: INCREMENTAL_READ_TABLE_VERSION` [...]
+| [hoodie.datasource.read.streaming.table.version](#hoodiedatasourcereadstreamingtableversion) | (N/A) | The table version assumed for streaming read<br />`Config Param: STREAMING_READ_TABLE_VERSION` [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files)<br />`Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: READ_PRE_COMBINE_FIELD` [...]
---
@@ -113,21 +122,23 @@ Options useful for writing tables via `write.format.option(...)`
[**Basic Configs**](#Write-Options-basic-configs)
-| Config Name | Default | Description |
-| ------------------------------------------------------------------------------------------------ | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
-| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD` |
-| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD` |
-| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
-| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
-| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
-| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
-| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
-| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
-| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING` |
-| [hoodie.datasource.write.operation](#hoodiedatasourcewriteoperation) | upsert | Whether to do upsert, insert or bulk_insert for the write operation. Use bulk_insert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br />`Config Param: OPERATION` |
-| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD` |
-| [hoodie.datasource.write.table.type](#hoodiedatasourcewritetabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TABLE_TYPE` |
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` [...]
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD` [...]
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD` [...]
+| [hoodie.datasource.write.secondarykey.column](#hoodiedatasourcewritesecondarykeycolumn) | (N/A) | Columns that constitute the secondary key component. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: SECONDARYKEY_COLUMN_NAME` [...]
+| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` [...]
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` [...]
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` [...]
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING` [...]
+| [hoodie.datasource.write.operation](#hoodiedatasourcewriteoperation) | upsert | Whether to do upsert, insert or bulk_insert for the write operation. Use bulk_insert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br />`Config Param: OPERATION` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.datasource.write.table.type](#hoodiedatasourcewritetabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TABLE_TYPE` [...]
+| [hoodie.write.record.merge.mode](#hoodiewriterecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge records, i.e., the record with the large [...]
---
## Flink Sql Configs {#FLINK_SQL}
@@ -148,6 +159,7 @@ Flink jobs using the SQL can be configured through the options in WITH clause. T
| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name to register to Hive metastore<br /> `Config Param: DATABASE_NAME` [...]
| [hoodie.table.name](#hoodietablename) | (N/A) | Table name to register to Hive metastore<br /> `Config Param: TABLE_NAME` [...]
| [path](#path) | (N/A) | Base path for the target hoodie table. The path would be created if it does not exist, otherwise a Hoodie table expects to be initialized successfully<br /> `Config Param: PATH` [...]
+| [read.commits.limit](#readcommitslimit) | (N/A) | The maximum number of commits allowed to read in each instant check, if it is streaming read, the avg read instants number per-second would be 'read.commits.limit'/'read.streaming.check-interval', by default no limit<br /> `Config Param: READ_COMMITS_LIMIT` [...]
| [read.end-commit](#readend-commit) | (N/A) | End commit instant for reading, the commit time format should be 'yyyyMMddHHmmss'<br /> `Config Param: READ_END_COMMIT` [...]
| [read.start-commit](#readstart-commit) | (N/A) | Start commit instant for reading, the commit time format should be 'yyyyMMddHHmmss', by default reading from the latest instant for streaming read<br /> `Config Param: READ_START_COMMIT` [...]
| [archive.max_commits](#archivemax_commits) | 50 | Max number of commits to keep before archiving older commits into a sequential log, default 50<br /> `Config Param: ARCHIVE_MAX_COMMITS` [...]
@@ -171,10 +183,12 @@ Flink jobs using the SQL can be configured through the options in WITH clause. T
| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | | Partition path field. Value to be used at the `partitionPath` component of `HoodieKey`. Actual value obtained by invoking .toString(), default ''<br /> `Config Param: PARTITION_PATH_FIELD` [...]
| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | uuid | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br /> `Config Param: RECORD_KEY_FIELD` [...]
| [index.type](#indextype) | FLINK_STATE | Index type of Flink write job, default is using state backed index.<br /> `Config Param: INDEX_TYPE` [...]
+| [lookup.join.cache.ttl](#lookupjoincachettl) | PT1H | The cache TTL (e.g. 10min) for the build table in lookup join.<br /> `Config Param: LOOKUP_JOIN_CACHE_TTL` [...]
| [metadata.compaction.delta_commits](#metadatacompactiondelta_commits) | 10 | Max delta commits for metadata table to trigger compaction, default 10<br /> `Config Param: METADATA_COMPACTION_DELTA_COMMITS` [...]
-| [metadata.enabled](#metadataenabled) | false | Enable the internal metadata table which serves table metadata like level file listings, default disabled<br /> `Config Param: METADATA_ENABLED` [...]
+| [metadata.enabled](#metadataenabled) | true | Enable the internal metadata table which serves table metadata like level file listings, default enabled<br /> `Config Param: METADATA_ENABLED` [...]
| [precombine.field](#precombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br /> `Config Param: PRECOMBINE_FIELD` [...]
| [read.streaming.enabled](#readstreamingenabled) | false | Whether to read as streaming source, default false<br /> `Config Param: READ_AS_STREAMING` [...]
+| [read.streaming.skip_insertoverwrite](#readstreamingskip_insertoverwrite) | false | Whether to skip insert overwrite instants to avoid reading base files of insert overwrite operations for streaming read. In streaming scenarios, insert overwrite is usually used to repair data, here you can control the visibility of downstream streaming read.<br /> `Config Param: READ_STREAMING_SKIP_INSERT_OVERWRITE` [...]
| [table.type](#tabletype) | COPY_ON_WRITE | Type of table to write. COPY_ON_WRITE (or) MERGE_ON_READ<br /> `Config Param: TABLE_TYPE` [...]
| [write.operation](#writeoperation) | upsert | The write operation, that this write should do<br /> `Config Param: OPERATION` [...]
| [write.parquet.max.file.size](#writeparquetmaxfilesize) | 120 | Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /> `Config Param: WRITE_PARQUET_MAX_FILE_SIZE` [...]
@@ -208,11 +222,14 @@ Configurations used by the Hudi Metadata Table. This table maintains the metadat
[**Basic Configs**](#Metadata-Configs-basic-configs)
-| Config Name | Default | Description |
-| ---------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| [hoodie.metadata.enable](#hoodiemetadataenable) | true | Enable the internal metadata table which serves table metadata like level file listings<br />`Config Param: ENABLE`<br />`Since Version: 0.7.0` |
-| [hoodie.metadata.index.bloom.filter.enable](#hoodiemetadataindexbloomfilterenable) | false | Enable indexing bloom filters of user data files under metadata table. When enabled, metadata table will have a partition to store the bloom filter index and will be used during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_BLOOM_FILTER`<br />`Since Version: 0.11.0` |
-| [hoodie.metadata.index.column.stats.enable](#hoodiemetadataindexcolumnstatsenable) | false | Enable indexing column ranges of user data files under metadata table key lookups. When enabled, metadata table will have a partition to store the column ranges and will be used for pruning files during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_COLUMN_STATS`<br />`Since Version: 0.11.0` |
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.metadata.enable](#hoodiemetadataenable) | true | Enable the internal metadata table which serves table metadata like level file listings<br />`Config Param: ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.metadata.index.bloom.filter.enable](#hoodiemetadataindexbloomfilterenable) | false | Enable indexing bloom filters of user data files under metadata table. When enabled, metadata table will have a partition to store the bloom filter index and will be used during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_BLOOM_FILTER`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.column.stats.enable](#hoodiemetadataindexcolumnstatsenable) | false | Enable indexing column ranges of user data files under metadata table key lookups. When enabled, metadata table will have a partition to store the column ranges and will be used for pruning files during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_COLUMN_STATS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.expression.enable](#hoodiemetadataindexexpressionenable) | false | Enable expression index within the metadata table. When this configuration property is enabled (`true`), the Hudi writer automatically keeps all expression indexes consistent with the data table. When disabled (`false`), all expression indexes are deleted. Note that individual expression index can only be created through a `CREATE INDEX` and deleted through a `DROP INDEX` statement [...]
+| [hoodie.metadata.index.partition.stats.enable](#hoodiemetadataindexpartitionstatsenable) | false | Enable aggregating stats for each column at the storage partition level.<br />`Config Param: ENABLE_METADATA_INDEX_PARTITION_STATS`<br />`Since Version: 1.0.0` [...]
+| [hoodie.metadata.index.secondary.enable](#hoodiemetadataindexsecondaryenable) | true | Enable secondary index within the metadata table. When this configuration property is enabled (`true`), the Hudi writer automatically keeps all secondary indexes consistent with the data table. When disabled (`false`), all secondary indexes are deleted. Note that individual secondary index can only be created through a `CREATE INDEX` and deleted through a `DROP INDEX` statement in [...]
---
@@ -272,10 +289,10 @@ Cleaning (reclamation of older/unused file groups/slices).
[**Basic Configs**](#Clean-Configs-basic-configs)
-| Config Name | Default | Description |
-| ---------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.clean.async](#hoodiecleanasync) | false | Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_CLEAN` |
-| [hoodie.cleaner.commits.retained](#hoodiecleanercommitsretained) | 10 | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br />`Config Param: CLEANER_COMMITS_RETAINED` |
+| Config Name | Default | Description |
+| ------------------------------------------------------------ | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.clean.async.enabled](#hoodiecleanasyncenabled) | false | Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_CLEAN` |
+| [hoodie.clean.commits.retained](#hoodiecleancommitsretained) | 10 | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br />`Config Param: CLEANER_COMMITS_RETAINED` |
---
@@ -345,12 +362,17 @@ Configurations that control write behavior on Hudi tables. These can be directly
[**Basic Configs**](#Write-Configurations-basic-configs)
-| Config Name | Default | Description [...]
-| --------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` [...]
-| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br />`Config Param: TBL_NAME` [...]
-| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD_NAME` [...]
-| [hoodie.write.concurrency.mode](#hoodiewriteconcurrencymode) | SINGLE_WRITER | org.apache.hudi.common.model.WriteConcurrencyMode: Concurrency modes for write operations. SINGLE_WRITER(default): Only one active writer to the table. Maximizes throughput. OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table with lazy conflict resolution using locks. This means that only one writer succeeds if multiple writers write to the same file group. [...]
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br />`Config Param: TBL_NAME` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD_NAME` [...]
+| [hoodie.instant_state.timeline_server_based.enabled](#hoodieinstant_statetimeline_server_basedenabled) | false | If enabled, writers get instant state from timeline server rather than requesting DFS directly<br />`Config Param: INSTANT_STATE_TIMELINE_SERVER_BASED`<br />`Since Version: 1.0.0` [...]
+| [hoodie.instant_state.timeline_server_based.force_refresh.request.number](#hoodieinstant_statetimeline_server_basedforce_refreshrequestnumber) | 100 | Number of requests to trigger instant state cache refreshing<br />`Config Param: INSTANT_STATE_TIMELINE_SERVER_BASED_FORCE_REFRESH_REQUEST_NUMBER`<br />`Since Version: 1.0.0` [...]
+| [hoodie.write.auto.upgrade](#hoodiewriteautoupgrade) | true | If enabled, writers automatically migrate the table to the specified write table version if the current table version is lower.<br />`Config Param: AUTO_UPGRADE_VERSION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.write.concurrency.mode](#hoodiewriteconcurrencymode) | SINGLE_WRITER | org.apache.hudi.common.model.WriteConcurrencyMode: Concurrency modes for write operations. SINGLE_WRITER(default): Only one active writer to the table. Maximizes throughput. OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table with lazy conflict resolution using locks. This means that only on [...]
+| [hoodie.write.record.merge.mode](#hoodiewriterecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge rec [...]
+| [hoodie.write.table.version](#hoodiewritetableversion) | 8 | The table version this writer is storing the table in. This should match the current table version.<br />`Config Param: WRITE_TABLE_VERSION`<br />`Since Version: 1.0.0` [...]
---
@@ -390,6 +412,7 @@ Hudi maintains keys (record key + partition path) for uniquely identifying a par
| ------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD_NAME` |
+| [hoodie.datasource.write.secondarykey.column](#hoodiedatasourcewritesecondarykeycolumn) | (N/A) | Columns that constitute the secondary key component. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: SECONDARYKEY_COLUMN_NAME` |
| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` |
---
@@ -407,9 +430,28 @@ Configurations that control indexing behavior, which tags incoming records as ei
[**Basic Configs**](#Common-Index-Configs-basic-configs)
-| Config Name | Default | Description [...]
-| ------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.index.type](#hoodieindextype) | (N/A) | org.apache.hudi.index.HoodieIndex$IndexType: Determines how input records are indexed, i.e., looked up based on the key for the location in the existing table. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java engines. HBASE: uses an external managed Apache HBase table to store record key to location mapping. HBase index is a global index, enforcing key uniqueness across all partitions in the table. INMEMORY: Use [...]
+| Config Name | Default | Description |
+| ------------------------------------------------------------------ | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.expression.index.function](#hoodieexpressionindexfunction) | (N/A) | Function to be used for building the expression index.<br />`Config Param: INDEX_FUNCTION`<br />`Since Version: 1.0.0` |
+| [hoodie.expression.index.name](#hoodieexpressionindexname) | (N/A) | Name of the expression index. This is also used for the partition name in the metadata table.<br />`Config Param: INDEX_NAME`<br />`Since Version: 1.0.0` |
+| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Index definition checksum is used to guard against partial writes in HDFS. It is added as the last entry in index.properties and then used to validate while reading table config.<br />`Config Param: INDEX_DEFINITION_CHECKSUM`<br />`Since Version: 1.0.0` |
+| [hoodie.expression.index.type](#hoodieexpressionindextype) | COLUMN_STATS | Type of the expression index. Default is `column_stats` if there are no functions and expressions in the command. Valid options could be BITMAP, COLUMN_STATS, LUCENE, etc. If index_type is not provided, and there are functions or expressions in the command then a expression index using column stats will be created.<br />`Config Param: INDEX_TYPE`<br />`Since Version: 1.0.0` |
+---
+
+
+#### Common Index Configs {#Common-Index-Configs}
+
+
+
+
+
+[**Basic Configs**](#Common-Index-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.index.type](#hoodieindextype) | (N/A) | org.apache.hudi.index.HoodieIndex$IndexType: Determines how input records are indexed, i.e., looked up based on the key for the location in the existing table. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java engines. HBASE: uses an external managed Apache HBase table to store record key to location mapping. HBase index is a global index, enforcing key uniqueness across all partitions i [...]
+| [hoodie.bucket.index.query.pruning](#hoodiebucketindexquerypruning) | true | Control if table with bucket index use bucket query or not<br />`Config Param: BUCKET_QUERY_INDEX` [...]
---
## Metastore and Catalog Sync Configs {#META_SYNC}
diff --git a/website/docs/configurations.md b/website/docs/configurations.md
index 87f545d951d..0758147e0df 100644
--- a/website/docs/configurations.md
+++ b/website/docs/configurations.md
@@ -5,7 +5,7 @@ permalink: /docs/configurations.html
summary: This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
toc_min_heading_level: 2
toc_max_heading_level: 4
-last_modified_at: 2024-08-16T12:54:21.684
+last_modified_at: 2024-12-06T17:38:05.818
---
@@ -20,7 +20,7 @@ hoodie.datasource.hive_sync.support_timestamp false
```
It helps to have a central configuration file for your common cross job configurations/tunings, so all the jobs on your cluster can utilize it. It also works with Spark SQL DML/DDL, and helps avoid having to pass configs inside the SQL statements.
-Hudi always loads the configuration file under default directory `file:/etc/hudi/conf`, if exists, to set the default configs. You can specify a different configuration directory location by setting the `HUDI_CONF_DIR` environment variable.
+Hudi always loads the configuration file under default directory `file:/etc/hudi/conf`, if exists, to set the default configs. Besides, you can specify another configuration directory location by setting the `HUDI_CONF_DIR` environment variable. The configs stored in `HUDI_CONF_DIR/hudi-defaults.conf` are loaded, overriding any configs already set by the config file in the default directory.
- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
- [**Flink Sql Configs**](#FLINK_SQL): These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
- [**Write Client Configs**](#WRITE_CLIENT): Internally, the Hudi datasource uses a RDD based HoodieWriteClient API to actually perform writes to storage. These configs provide deep control over lower level aspects like file sizing, compression, parallelism, compaction, write schema, cleaning etc. Although Hudi provides sane defaults, from time-time these configs may need to be tweaked to optimize for specific workloads.
@@ -38,10 +38,9 @@ In the tables below **(N/A)** means there is no default value set
## Externalized Config File
Instead of directly passing configuration settings to every Hudi job, you can also centrally set them in a configuration
-file `hudi-defaults.conf`. Hudi always loads the configuration file under default directory `file:/etc/hudi/conf`, if exists,
-to set the default configs. Besides, you can specify another configuration directory location by setting the `HUDI_CONF_DIR`
-environment variable. The configs stored in `HUDI_CONF_DIR/hudi-defaults.conf` are loaded, overriding any configs already set
-by the config file in the default directory.
+file `hudi-defaults.conf`. By default, Hudi would load the configuration file under `/etc/hudi/conf` directory. You can
+specify a different configuration directory location by setting the `HUDI_CONF_DIR` environment variable. This can be
+useful for uniformly enforcing repeated configs (like Hive sync or write/index tuning), across your entire data lake.
## Hudi Table Config {#TABLE_CONFIG}
Basic Hudi Table configuration parameters.
@@ -55,37 +54,44 @@ Configurations of the Hudi Table like type of ingestion, storage formats, hive t
[**Basic Configs**](#Hudi-Table-Basic-Configs-basic-configs)
-| Config Name | Default | Description [...]
-| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BOOTSTRAP_BASE_PATH` [...]
-| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name that will be used for incremental query.If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME` [...]
-| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.<br />`Config Param: TABLE_CHECKSUM`<br />`Since Version: 0.11.0` [...]
-| [hoodie.table.create.schema](#hoodietablecreateschema) | (N/A) | Schema used when creating the table, for the first time.<br />`Config Param: CREATE_SCHEMA` [...]
-| [hoodie.table.keygenerator.class](#hoodietablekeygeneratorclass) | (N/A) | Key Generator class property for the hoodie table<br />`Config Param: KEY_GENERATOR_CLASS_NAME` [...]
-| [hoodie.table.metadata.partitions](#hoodietablemetadatapartitions) | (N/A) | Comma-separated list of metadata partitions that have been completely built and in-sync with data table. These partitions are ready for use by the readers<br />`Config Param: TABLE_METADATA_PARTITIONS`<br />`Since Version: 0.11.0` [...]
-| [hoodie.table.metadata.partitions.inflight](#hoodietablemetadatapartitionsinflight) | (N/A) | Comma-separated list of metadata partitions whose building is in progress. These partitions are not yet ready for use by the readers.<br />`Config Param: TABLE_METADATA_PARTITIONS_INFLIGHT`<br />`Since Version: 0.11.0` [...]
-| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: NAME` [...]
-| [hoodie.table.partition.fields](#hoodietablepartitionfields) | (N/A) | Fields used to partition the table. Concatenated values of these fields are used as the partition path, by invoking toString()<br />`Config Param: PARTITION_FIELDS` [...]
-| [hoodie.table.precombine.field](#hoodietableprecombinefield) | (N/A) | Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br />`Config Param: PRECOMBINE_FIELD` [...]
-| [hoodie.table.recordkey.fields](#hoodietablerecordkeyfields) | (N/A) | Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br />`Config Param: RECORDKEY_FIELDS` [...]
-| [hoodie.table.secondary.indexes.metadata](#hoodietablesecondaryindexesmetadata) | (N/A) | The metadata of secondary indexes<br />`Config Param: SECONDARY_INDEXES_METADATA`<br />`Since Version: 0.13.0` [...]
-| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | (N/A) | Version of timeline used, by the table.<br />`Config Param: TIMELINE_LAYOUT_VERSION` [...]
-| [hoodie.archivelog.folder](#hoodiearchivelogfolder) | archived | path under the meta folder, to store archived timeline instants at.<br />`Config Param: ARCHIVELOG_FOLDER` [...]
-| [hoodie.bootstrap.index.class](#hoodiebootstrapindexclass) | org.apache.hudi.common.bootstrap.index.hfile.HFileBootstrapIndex | Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_CLASS_NAME` [...]
-| [hoodie.bootstrap.index.enable](#hoodiebootstrapindexenable) | true | Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined, default true.<br />`Config Param: BOOTSTRAP_INDEX_ENABLE` [...]
-| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | org.apache.hudi.common.model.DefaultHoodieRecordPayload | Payload class to use for performing compactions, i.e merge delta logs with current base file and then produce a new base file.<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
-| [hoodie.compaction.payload.type](#hoodiecompactionpayloadtype) | HOODIE_AVRO_DEFAULT | org.apache.hudi.common.model.RecordPayloadType: Payload to use for merging records AWS_DMS_AVRO: Provides support for seamlessly applying changes captured via Amazon Database Migration Service onto S3. HOODIE_AVRO: A payload to wrap a existing Hoodie Avro Record. Useful to create a HoodieRecord over existing Gener [...]
-| [hoodie.compaction.record.merger.strategy](#hoodiecompactionrecordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in hoodie.datasource.write.record.merger.impls which has the same merger strategy id<br />`Config Param: RECORD_MERGER_STRATEGY`<br />`Since Version: 0.13.0` [...]
-| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` [...]
-| [hoodie.partition.metafile.use.base.format](#hoodiepartitionmetafileusebaseformat) | false | If true, partition metafiles are saved in the same format as base-files for this dataset (e.g. Parquet / ORC). If false (default) partition metafiles are saved as properties files.<br />`Config Param: PARTITION_METAFILE_USE_BASE_FORMAT` [...]
-| [hoodie.populate.meta.fields](#hoodiepopulatemetafields) | true | When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br />`Config Param: POPULATE_META_FIELDS` [...]
-| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | Base file format to store all the base file data.<br />`Config Param: BASE_FILE_FORMAT` [...]
-| [hoodie.table.cdc.enabled](#hoodietablecdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode.<br />`Config Param: CDC_ENABLED`<br />`Since Version: 0.13.0` [...]
-| [hoodie.table.cdc.supplemental.logging.mode](#hoodietablecdcsupplementalloggingmode) | DATA_BEFORE_AFTER | org.apache.hudi.common.table.cdc.HoodieCDCSupplementalLoggingMode: Change log capture supplemental logging mode. The supplemental log is used for accelerating the generation of change log details. OP_KEY_ONLY: Only keeping record keys in the supplemental logs, so the reader needs to figure out the update before image [...]
-| [hoodie.table.log.file.format](#hoodietablelogfileformat) | HOODIE_LOG | Log format used for the delta logs.<br />`Config Param: LOG_FILE_FORMAT` [...]
-| [hoodie.table.timeline.timezone](#hoodietabletimelinetimezone) | LOCAL | User can set hoodie commit timeline timezone, such as utc, local and so on. local is default<br />`Config Param: TIMELINE_TIMEZONE` [...]
-| [hoodie.table.type](#hoodietabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TYPE` [...]
-| [hoodie.table.version](#hoodietableversion) | ZERO | Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br />`Config Param: VERSION` [...]
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.bootstrap.base.path](#hoodiebootstrapbasepath) | (N/A) | Base path of the dataset that needs to be bootstrapped as a Hudi table<br />`Config Param: BOOTSTRAP_BASE_PATH` [...]
+| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | (N/A) | Payload class to use for performing merges, compactions, i.e merge delta logs with current base file and then produce a new base file.<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name. If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME` [...]
+| [hoodie.record.merge.strategy.id](#hoodierecordmergestrategyid) | (N/A) | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in `hoodie.write.record.merge.custom.implementation.classes` which has the same merger strategy id<br />`Config Param: RECORD_MERGE_STRATEGY_ID`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.<br />`Config Param: TABLE_CHECKSUM`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.create.schema](#hoodietablecreateschema) | (N/A) | Schema used when creating the table<br />`Config Param: CREATE_SCHEMA` [...]
+| [hoodie.table.index.defs.path](#hoodietableindexdefspath) | (N/A) | Relative path to table base path where the index definitions are stored<br />`Config Param: RELATIVE_INDEX_DEFINITION_PATH`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.keygenerator.class](#hoodietablekeygeneratorclass) | (N/A) | Key Generator class property for the hoodie table<br />`Config Param: KEY_GENERATOR_CLASS_NAME` [...]
+| [hoodie.table.keygenerator.type](#hoodietablekeygeneratortype) | (N/A) | Key Generator type to determine key generator class<br />`Config Param: KEY_GENERATOR_TYPE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.metadata.partitions](#hoodietablemetadatapartitions) | (N/A) | Comma-separated list of metadata partitions that have been completely built and in-sync with data table. These partitions are ready for use by the readers<br />`Config Param: TABLE_METADATA_PARTITIONS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.metadata.partitions.inflight](#hoodietablemetadatapartitionsinflight) | (N/A) | Comma-separated list of metadata partitions whose building is in progress. These partitions are not yet ready for use by the readers.<br />`Config Param: TABLE_METADATA_PARTITIONS_INFLIGHT`<br />`Since Version: 0.11.0` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: NAME` [...]
+| [hoodie.table.partition.fields](#hoodietablepartitionfields) | (N/A) | Comma separated field names used to partition the table. These field names also include the partition type which is used by custom key generators<br />`Config Param: PARTITION_FIELDS` [...]
+| [hoodie.table.precombine.field](#hoodietableprecombinefield) | (N/A) | Field used in preCombining before actual write. By default, when two records have the same key value, the largest value for the precombine field determined by Object.compareTo(..), is picked.<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.table.recordkey.fields](#hoodietablerecordkeyfields) | (N/A) | Columns used to uniquely identify the table. Concatenated values of these fields are used as the record key component of HoodieKey.<br />`Config Param: RECORDKEY_FIELDS` [...]
+| [hoodie.table.secondary.indexes.metadata](#hoodietablesecondaryindexesmetadata) | (N/A) | The metadata of secondary indexes<br />`Config Param: SECONDARY_INDEXES_METADATA`<br />`Since Version: 0.13.0` [...]
+| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | (N/A) | Version of timeline used, by the table.<br />`Config Param: TIMELINE_LAYOUT_VERSION` [...]
+| [hoodie.archivelog.folder](#hoodiearchivelogfolder) | archived | path under the meta folder, to store archived timeline instants at.<br />`Config Param: ARCHIVELOG_FOLDER` [...]
+| [hoodie.bootstrap.index.class](#hoodiebootstrapindexclass) | org.apache.hudi.common.bootstrap.index.hfile.HFileBootstrapIndex | Implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_CLASS_NAME` [...]
+| [hoodie.bootstrap.index.enable](#hoodiebootstrapindexenable) | true | Whether or not, this is a bootstrapped table, with bootstrap base data and an mapping index defined, default true.<br />`Config Param: BOOTSTRAP_INDEX_ENABLE` [...]
+| [hoodie.bootstrap.index.type](#hoodiebootstrapindextype) | HFILE | Bootstrap index type determines which implementation to use, for mapping base files to bootstrap base file, that contain actual data.<br />`Config Param: BOOTSTRAP_INDEX_TYPE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` [...]
+| [hoodie.partition.metafile.use.base.format](#hoodiepartitionmetafileusebaseformat) | false | If true, partition metafiles are saved in the same format as base-files for this dataset (e.g. Parquet / ORC). If false (default) partition metafiles are saved as properties files.<br />`Config Param: PARTITION_METAFILE_USE_BASE_FORMAT` [...]
+| [hoodie.populate.meta.fields](#hoodiepopulatemetafields) | true | When enabled, populates all meta fields. When disabled, no meta fields are populated and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing<br />`Config Param: POPULATE_META_FIELDS` [...]
+| [hoodie.record.merge.mode](#hoodierecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge reco [...]
+| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | Base file format to store all the base file data.<br />`Config Param: BASE_FILE_FORMAT` [...]
+| [hoodie.table.cdc.enabled](#hoodietablecdcenabled) | false | When enable, persist the change data if necessary, and can be queried as a CDC query mode.<br />`Config Param: CDC_ENABLED`<br />`Since Version: 0.13.0` [...]
+| [hoodie.table.cdc.supplemental.logging.mode](#hoodietablecdcsupplementalloggingmode) | DATA_BEFORE_AFTER | org.apache.hudi.common.table.cdc.HoodieCDCSupplementalLoggingMode: Change log capture supplemental logging mode. The supplemental log is used for accelerating the generation of change log details. OP_KEY_ONLY: Only keeping record keys in the supplemental logs, so the reader needs to figure out the update before image [...]
+| [hoodie.table.initial.version](#hoodietableinitialversion) | EIGHT | Initial Version of table when the table was created. Used for upgrade/downgrade to identify what upgrade/downgrade paths happened on the table. This is only configured when the table is initially setup.<br />`Config Param: INITIAL_VERSION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.log.file.format](#hoodietablelogfileformat) | HOODIE_LOG | Log format used for the delta logs.<br />`Config Param: LOG_FILE_FORMAT` [...]
+| [hoodie.table.multiple.base.file.formats.enable](#hoodietablemultiplebasefileformatsenable) | false | When set to true, the table can support reading and writing multiple base file formats.<br />`Config Param: MULTIPLE_BASE_FILE_FORMATS_ENABLE`<br />`Since Version: 1.0.0` [...]
+| [hoodie.table.timeline.timezone](#hoodietabletimelinetimezone) | LOCAL | User can set hoodie commit timeline timezone, such as utc, local and so on. local is default<br />`Config Param: TIMELINE_TIMEZONE` [...]
+| [hoodie.table.type](#hoodietabletype) | COPY_ON_WRITE | The table type for the underlying data.<br />`Config Param: TYPE` [...]
+| [hoodie.table.version](#hoodietableversion) | EIGHT | Version of table, used for running upgrade/downgrade steps between releases with potentially breaking/backwards compatible changes.<br />`Config Param: VERSION` [...]
+| [hoodie.timeline.history.path](#hoodietimelinehistorypath) | history | path under the meta folder, to store timeline history at.<br />`Config Param: TIMELINE_HISTORY_PATH` [...]
+| [hoodie.timeline.path](#hoodietimelinepath) | timeline | path under the meta folder, to store timeline instants at.<br />`Config Param: TIMELINE_PATH` [...]
[**Advanced Configs**](#Hudi-Table-Basic-Configs-advanced-configs)
@@ -109,12 +115,14 @@ Options useful for reading tables via `read.format.option(...)`
[**Basic Configs**](#Read-Options-basic-configs)
-| Config Name | Default | Description [...]
-| --------------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.datasource.read.begin.instanttime](#hoodiedatasourcereadbegininstanttime) | (N/A) | Required when `hoodie.datasource.query.type` is set to `incremental`. Represents the instant time to start incrementally pulling data from. The instanttime here need not necessarily correspond to an instant on the timeline. New data written with an instant_time > BEGIN_INSTANTTIME are fetched out. For e.g: ‘20170901080000’ will get all new data written after Sep 1, 2017 08:00AM. Note that [...]
-| [hoodie.datasource.read.end.instanttime](#hoodiedatasourcereadendinstanttime) | (N/A) | Used when `hoodie.datasource.query.type` is set to `incremental`. Represents the instant time to limit incrementally fetched data to. When not specified latest commit time from timeline is assumed by default. When specified, new data written with an instant_time <= END_INSTANTTIME are fetched out. Point in time type queries make more sense with begin and end instant times specified. Note [...]
-| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files)<br />`Config Param: QUERY_TYPE` [...]
-| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: READ_PRE_COMBINE_FIELD` [...]
+| Config Name | Default | Description [...]
+| -------------------------------------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.read.begin.instanttime](#hoodiedatasourcereadbegininstanttime) | (N/A) | Required when `hoodie.datasource.query.type` is set to `incremental`. Represents the completion time to start incrementally pulling data from. The completion time here need not necessarily correspond to an instant on the timeline. New data written with completion_time >= START_COMMIT are fetched out. For e.g: ‘20170901080000’ will get all new data written on or after Sep 1, 201 [...]
+| [hoodie.datasource.read.end.instanttime](#hoodiedatasourcereadendinstanttime) | (N/A) | Used when `hoodie.datasource.query.type` is set to `incremental`. Represents the completion time to limit incrementally fetched data to. When not specified latest commit completion time from timeline is assumed by default. When specified, new data written with completion_time <= END_COMMIT are fetched out. Point in time type queries make more sense with begin and end completion [...]
+| [hoodie.datasource.read.incr.table.version](#hoodiedatasourcereadincrtableversion) | (N/A) | The table version assumed for incremental read<br />`Config Param: INCREMENTAL_READ_TABLE_VERSION` [...]
+| [hoodie.datasource.read.streaming.table.version](#hoodiedatasourcereadstreamingtableversion) | (N/A) | The table version assumed for streaming read<br />`Config Param: STREAMING_READ_TABLE_VERSION` [...]
+| [hoodie.datasource.query.type](#hoodiedatasourcequerytype) | snapshot | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files)<br />`Config Param: QUERY_TYPE` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: READ_PRE_COMBINE_FIELD` [...]
[**Advanced Configs**](#Read-Options-advanced-configs)
@@ -123,8 +131,9 @@ Options useful for reading tables via `read.format.option(...)`
| ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| [as.of.instant](#asofinstant) | (N/A) | The query instant for time travel. Without specified this option, we query the latest snapshot.<br />`Config Param: TIME_TRAVEL_AS_OF_INSTANT` [...]
| [hoodie.datasource.read.paths](#hoodiedatasourcereadpaths) | (N/A) | Comma separated list of file paths to read within a Hudi table.<br />`Config Param: READ_PATHS` [...]
-| [hoodie.datasource.merge.type](#hoodiedatasourcemergetype) | payload_combine | For Snapshot query on merge on read table, control whether we invoke the record payload implementation to merge (payload_combine) or skip merging altogetherskip_merge<br />`Config Param: REALTIME_MERGE` [...]
+| [hoodie.datasource.merge.type](#hoodiedatasourcemergetype) | payload_combine | For Snapshot query on merge on read table. Use this key to define how the payloads are merged, in 1) skip_merge: read the base file records plus the log file records without merging; 2) payload_combine: read the base file records first, for each record in base file, checks whether the key is in the log file [...]
| [hoodie.datasource.query.incremental.format](#hoodiedatasourcequeryincrementalformat) | latest_state | This config is used alone with the 'incremental' query type.When set to 'latest_state', it returns the latest records' values.When set to 'cdc', it returns the cdc data.<br />`Config Param: INCREMENTAL_FORMAT`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.read.create.filesystem.relation](#hoodiedatasourcereadcreatefilesystemrelation) | false | When this is set, the relation created by DefaultSource is for a view representing the result set of the table valued function hudi_filesystem_view(...)<br />`Config Param: CREATE_FILESYSTEM_RELATION`<br />`Since Version: 1.0.0` [...]
| [hoodie.datasource.read.extract.partition.values.from.path](#hoodiedatasourcereadextractpartitionvaluesfrompath) | false | When set to true, values for partition columns (partition values) will be extracted from physical partition path (default Spark behavior). When set to false partition values will be read from the data file (in Hudi partition columns are persisted by default). This config is a fallback allowing to preserve exis [...]
| [hoodie.datasource.read.file.index.listing.mode](#hoodiedatasourcereadfileindexlistingmode) | lazy | Overrides Hudi's file-index implementation's file listing mode: when set to 'eager', file-index will list all partition paths and corresponding file slices w/in them eagerly, during initialization, prior to partition-pruning kicking in, meaning that all partitions will be listed including ones that might be s [...]
| [hoodie.datasource.read.file.index.listing.partition-path-prefix.analysis.enabled](#hoodiedatasourcereadfileindexlistingpartition-path-prefixanalysisenabled) | true | Controls whether partition-path prefix analysis is enabled w/in the file-index, allowing to avoid necessity to recursively list deep folder structures of partitioned tables w/ multiple partition columns, by carefully analyzing provided partition-column predicates and deducing corresponding partition-path prefi [...]
@@ -132,9 +141,11 @@ Options useful for reading tables via `read.format.option(...)`
| [hoodie.datasource.read.incr.filters](#hoodiedatasourcereadincrfilters) | | For use-cases like DeltaStreamer which reads from Hoodie Incremental table and applies opaque map functions, filters appearing late in the sequence of transformations cannot be automatically pushed down. This option allows setting filters directly on Hoodie Source.<br />`Config Param: PUSH_DOWN_INCR_FILTERS` [...]
| [hoodie.datasource.read.incr.path.glob](#hoodiedatasourcereadincrpathglob) | | For the use-cases like users only want to incremental pull from certain partitions instead of the full table. This option allows using glob pattern to directly filter on path.<br />`Config Param: INCR_PATH_GLOB` [...]
| [hoodie.datasource.read.schema.use.end.instanttime](#hoodiedatasourcereadschemauseendinstanttime) | false | Uses end instant schema when incrementally fetched data to. Default: users latest instant schema.<br />`Config Param: INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME` [...]
-| [hoodie.datasource.read.use.new.parquet.file.format](#hoodiedatasourcereadusenewparquetfileformat) | false | Read using the new Hudi parquet file format. The new Hudi parquet file format is introduced as an experimental feature in 0.14.0. Currently, the new Hudi parquet file format only applies to bootstrap and MOR queries. Schema evolution is also not supported by the new file format.<br />`Config Param: USE_NEW_HUD [...]
+| [hoodie.datasource.read.table.valued.function.filesystem.relation.subpath](#hoodiedatasourcereadtablevaluedfunctionfilesystemrelationsubpath) | | A regex under the table's base path to get file system view information<br />`Config Param: FILESYSTEM_RELATION_ARG_SUBPATH`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.read.table.valued.function.timeline.relation](#hoodiedatasourcereadtablevaluedfunctiontimelinerelation) | false | When this is set, the relation created by DefaultSource is for a view representing the result set of the table valued function hudi_query_timeline(...)<br />`Config Param: CREATE_TIMELINE_RELATION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.datasource.read.table.valued.function.timeline.relation.archived](#hoodiedatasourcereadtablevaluedfunctiontimelinerelationarchived) | false | When this is set, the result set of the table valued function hudi_query_timeline(...) will include archived timeline<br />`Config Param: TIMELINE_RELATION_ARG_ARCHIVED_TIMELINE`<br />`Since Version: 1.0.0` [...]
| [hoodie.datasource.streaming.startOffset](#hoodiedatasourcestreamingstartOffset) | earliest | Start offset to pull data from hoodie streaming source. allow earliest, latest, and specified start instant time<br />`Config Param: START_OFFSET`<br />`Since Version: 0.13.0` [...]
-| [hoodie.enable.data.skipping](#hoodieenabledataskipping) | false | Enables data-skipping allowing queries to leverage indexes to reduce the search space by skipping over files<br />`Config Param: ENABLE_DATA_SKIPPING`<br />`Since Version: 0.10.0` [...]
+| [hoodie.enable.data.skipping](#hoodieenabledataskipping) | true | Enables data-skipping allowing queries to leverage indexes to reduce the search space by skipping over files<br />`Config Param: ENABLE_DATA_SKIPPING`<br />`Since Version: 0.10.0` [...]
| [hoodie.file.index.enable](#hoodiefileindexenable) | true | Enables use of the spark file index implementation for Hudi, that speeds up listing of large tables.<br />`Config Param: ENABLE_HOODIE_FILE_INDEX` [...]
| [hoodie.read.timeline.holes.resolution.policy](#hoodiereadtimelineholesresolutionpolicy) | FAIL | When doing incremental queries, there could be hollow commits (requested or inflight commits that are not the latest) that are produced by concurrent writers and could lead to potential data loss. This config allows users to have different ways of handling this situation. The valid values are [FAIL, BLOCK, USE [...]
| [hoodie.schema.on.read.enable](#hoodieschemaonreadenable) | false | Enables support for Schema Evolution feature<br />`Config Param: SCHEMA_EVOLUTION_ENABLED` [...]
@@ -164,78 +175,79 @@ Options useful for writing tables via `write.format.option(...)`
[**Basic Configs**](#Write-Options-basic-configs)
-| Config Name | Default | Description |
-| ------------------------------------------------------------------------------------------------ | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` |
-| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD` |
-| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD` |
-| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
-| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` |
-| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` |
-| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` |
-| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` |
-| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` |
-| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING` |
-| [hoodie.datasource.write.operation](#hoodiedatasourcewriteoperation) | upsert | Whether to do upsert, insert or bulk_insert for the write operation. Use bulk_insert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br />`Config Param: OPERATION` |
-| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD` |
-| [hoodie.datasource.write.table.type](#hoodiedatasourcewritetabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TABLE_TYPE` |
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.mode](#hoodiedatasourcehive_syncmode) | (N/A) | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.<br />`Config Param: HIVE_SYNC_MODE` [...]
+| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD` [...]
+| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD` [...]
+| [hoodie.datasource.write.secondarykey.column](#hoodiedatasourcewritesecondarykeycolumn) | (N/A) | Columns that constitute the secondary key component. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: SECONDARYKEY_COLUMN_NAME` [...]
+| [hoodie.clustering.async.enabled](#hoodieclusteringasyncenabled) | false | Enable running of clustering service, asynchronously as inserts happen on the table.<br />`Config Param: ASYNC_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.clustering.inline](#hoodieclusteringinline) | false | Turn on inline clustering - clustering will be run after each write operation is complete<br />`Config Param: INLINE_CLUSTERING_ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.datasource.hive_sync.enable](#hoodiedatasourcehive_syncenable) | false | When set to true, register/sync the table to Apache Hive metastore.<br />`Config Param: HIVE_SYNC_ENABLED` [...]
+| [hoodie.datasource.hive_sync.jdbcurl](#hoodiedatasourcehive_syncjdbcurl) | jdbc:hive2://localhost:10000 | Hive metastore url<br />`Config Param: HIVE_URL` [...]
+| [hoodie.datasource.hive_sync.metastore.uris](#hoodiedatasourcehive_syncmetastoreuris) | thrift://localhost:9083 | Hive metastore url<br />`Config Param: METASTORE_URIS` [...]
+| [hoodie.datasource.meta.sync.enable](#hoodiedatasourcemetasyncenable) | false | Enable Syncing the Hudi Table with an external meta store or data catalog.<br />`Config Param: META_SYNC_ENABLED` [...]
+| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING` [...]
+| [hoodie.datasource.write.operation](#hoodiedatasourcewriteoperation) | upsert | Whether to do upsert, insert or bulk_insert for the write operation. Use bulk_insert to load new data into a table, and there on use upsert/insert. bulk insert uses a disk based write path to scale to load large inputs without need to cache it.<br />`Config Param: OPERATION` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD` [...]
+| [hoodie.datasource.write.table.type](#hoodiedatasourcewritetabletype) | COPY_ON_WRITE | The table type for the underlying data, for this write. This can’t change between writes.<br />`Config Param: TABLE_TYPE` [...]
+| [hoodie.write.record.merge.mode](#hoodiewriterecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge records, i.e., the record with the large [...]
[**Advanced Configs**](#Write-Options-advanced-configs)
-| Config Name | Default | Description [...]
-| ------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` [...]
-| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` [...]
-| [hoodie.datasource.overwrite.mode](#hoodiedatasourceoverwritemode) | (N/A) | Controls whether overwrite use dynamic or static mode, if not configured, respect spark.sql.sources.partitionOverwriteMode<br />`Config Param: OVERWRITE_MODE`<br />`Since Version: 0.14.0` [...]
-| [hoodie.datasource.write.partitions.to.delete](#hoodiedatasourcewritepartitionstodelete) | (N/A) | Comma separated list of partitions to delete. Allows use of wildcard *<br />`Config Param: PARTITIONS_TO_DELETE` [...]
-| [hoodie.datasource.write.table.name](#hoodiedatasourcewritetablename) | (N/A) | Table name for the datasource write. Also used to register the table into meta stores.<br />`Config Param: TABLE_NAME` [...]
-| [hoodie.datasource.compaction.async.enable](#hoodiedatasourcecompactionasyncenable) | true | Controls whether async compaction should be turned on for MOR table writing.<br />`Config Param: ASYNC_COMPACT_ENABLE` [...]
-| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: HIVE_ASSUME_DATE_PARTITION` [...]
-| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` [...]
-| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: HIVE_BASE_FILE_FORMAT` [...]
-| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` [...]
-| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` [...]
-| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` [...]
-| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: HIVE_DATABASE` [...]
-| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` [...]
-| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: HIVE_PARTITION_EXTRACTOR_CLASS` [...]
-| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: HIVE_PARTITION_FIELDS` [...]
-| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` [...]
-| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` [...]
-| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` [...]
-| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` [...]
-| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` [...]
-| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: HIVE_TABLE` [...]
-| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` [...]
-| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Config [...]
-| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` [...]
-| [hoodie.datasource.insert.dup.policy](#hoodiedatasourceinsertduppolicy) | none | **Note** This is only applicable to Spark SQL writing.<br />When operation type is set to "insert", users can optionally enforce a dedup policy. This policy will be employed when records being ingested already exists in storage. Default policy is none and no action will be tak [...]
-| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: HIVE_CONDITIONAL_SYNC` [...]
-| [hoodie.datasource.write.commitmeta.key.prefix](#hoodiedatasourcewritecommitmetakeyprefix) | _ | Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata. This is useful to store checkpointing information, in a consistent way with the hudi timeline<br />`Config Param: COMMIT_METADATA_KEYPREFIX` [...]
-| [hoodie.datasource.write.drop.partition.columns](#hoodiedatasourcewritedroppartitioncolumns) | false | When set to true, will not write the partition columns into hudi. By default, false.<br />`Config Param: DROP_PARTITION_COLUMNS` [...]
-| [hoodie.datasource.write.insert.drop.duplicates](#hoodiedatasourcewriteinsertdropduplicates) | false | If set to true, records from the incoming dataframe will not overwrite existing records with the same key during the write operation. <br /> **Note** Just for Insert operation in Spark SQL writing since 0.14.0, users can switch to the config `hoodie.datasource.insert.dup.policy [...]
-| [hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass) | org.apache.hudi.keygen.SimpleKeyGenerator | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`<br />`Config Param: KEYGENERATOR_CLASS_NAME` [...]
-| [hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled](#hoodiedatasourcewritekeygeneratorconsistentlogicaltimestampenabled) | false | When set to true, consistent value will be generated for a logical timestamp type column, like timestamp-millis and timestamp-micros, irrespective of whether row-writer is enabled. Disabled by default so as not to break the pipeline that deploy either fully row-writer path or non row [...]
-| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Should we url encode the partition path value, before creating the folder structure.<br />`Config Param: URL_ENCODE_PARTITIONING` [...]
-| [hoodie.datasource.write.payload.class](#hoodiedatasourcewritepayloadclass) | org.apache.hudi.common.model.DefaultHoodieRecordPayload | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
-| [hoodie.datasource.write.payload.type](#hoodiedatasourcewritepayloadtype) | HOODIE_AVRO_DEFAULT | org.apache.hudi.common.model.RecordPayloadType: Payload to use for merging records AWS_DMS_AVRO: Provides support for seamlessly applying changes captured via Amazon Database Migration Service onto S3. HOODIE_AVRO: A payload to wrap a existing Hoodie Avro Record. Useful to cr [...]
-| [hoodie.datasource.write.reconcile.schema](#hoodiedatasourcewritereconcileschema) | false | This config controls how writer's schema will be selected based on the incoming batch's schema as well as existing table's one. When schema reconciliation is DISABLED, incoming batch's schema will be picked as a writer-schema (therefore updating table's schema). When schema reconcili [...]
-| [hoodie.datasource.write.record.merger.impls](#hoodiedatasourcewriterecordmergerimpls) | org.apache.hudi.common.model.HoodieAvroRecordMerger | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These merger impls will filter by hoodie.datasource.write.record.merger.strategy Hudi will pick most efficient implementation to perform merging/combining of the records (during upd [...]
-| [hoodie.datasource.write.record.merger.strategy](#hoodiedatasourcewriterecordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in hoodie.datasource.write.record.merger.impls which has the same merger strategy id<br />`Config Param: RECORD_MERGER_STRATEGY`<br />`Since Version: 0.13.0` [...]
-| [hoodie.datasource.write.row.writer.enable](#hoodiedatasourcewriterowwriterenable) | true | When set to true, will perform write operations directly using the spark native `Row` representation, avoiding any additional conversion costs.<br />`Config Param: ENABLE_ROW_WRITER` [...]
-| [hoodie.datasource.write.streaming.checkpoint.identifier](#hoodiedatasourcewritestreamingcheckpointidentifier) | default_single_writer | A stream identifier used for HUDI to fetch the right checkpoint(`batch id` to be more specific) corresponding this writer. Please note that keep the identifier an unique value for different writer if under multi-writer scenario. If the value is not set, will only keep the checkpoint [...]
-| [hoodie.datasource.write.streaming.disable.compaction](#hoodiedatasourcewritestreamingdisablecompaction) | false | By default for MOR table, async compaction is enabled with spark streaming sink. By setting this config to true, we can disable it and the expectation is that, users will schedule and execute compaction in a different process/job altogether. Some users may wish to run it separately t [...]
-| [hoodie.datasource.write.streaming.ignore.failed.batch](#hoodiedatasourcewritestreamingignorefailedbatch) | false | Config to indicate whether to ignore any non exception error (e.g. writestatus error) within a streaming microbatch. Turning this on, could hide the write status errors while the spark checkpoint moves ahead.So, would recommend users to use this with caution.<br />`Config Param: STRE [...]
-| [hoodie.datasource.write.streaming.retry.count](#hoodiedatasourcewritestreamingretrycount) | 3 | Config to indicate how many times streaming job should retry for a failed micro batch.<br />`Config Param: STREAMING_RETRY_CNT` [...]
-| [hoodie.datasource.write.streaming.retry.interval.ms](#hoodiedatasourcewritestreamingretryintervalms) | 2000 | Config to indicate how long (by millisecond) before a retry should issued for failed microbatch<br />`Config Param: STREAMING_RETRY_INTERVAL_MS` [...]
-| [hoodie.meta.sync.client.tool.class](#hoodiemetasyncclienttoolclass) | org.apache.hudi.hive.HiveSyncTool | Sync tool class name used to sync to metastore. Defaults to Hive.<br />`Config Param: META_SYNC_CLIENT_TOOL_CLASS_NAME` [...]
-| [hoodie.spark.sql.insert.into.operation](#hoodiesparksqlinsertintooperation) | insert | Sql write operation to use with INSERT_INTO spark sql command. This comes with 3 possible values, bulk_insert, insert and upsert. bulk_insert is generally meant for initial loads and is known to be performant compared to insert. But bulk_insert may not do small file management. If yo [...]
-| [hoodie.spark.sql.optimized.writes.enable](#hoodiesparksqloptimizedwritesenable) | true | Controls whether spark sql prepped update, delete, and merge are enabled.<br />`Config Param: SPARK_SQL_OPTIMIZED_WRITES`<br />`Since Version: 0.14.0` [...]
-| [hoodie.sql.bulk.insert.enable](#hoodiesqlbulkinsertenable) | false | When set to true, the sql insert statement will use bulk insert. This config is deprecated as of 0.14.0. Please use hoodie.spark.sql.insert.into.operation instead.<br />`Config Param: SQL_ENABLE_BULK_INSERT` [...]
-| [hoodie.sql.insert.mode](#hoodiesqlinsertmode) | upsert | Insert mode when insert data to pk-table. The optional modes are: upsert, strict and non-strict.For upsert mode, insert statement do the upsert operation for the pk-table which will update the duplicate record.For strict mode, insert statement will keep the primary key uniqueness con [...]
-| [hoodie.streamer.source.kafka.value.deserializer.class](#hoodiestreamersourcekafkavaluedeserializerclass) | io.confluent.kafka.serializers.KafkaAvroDeserializer | This class is used by kafka client to deserialize the records<br />`Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.9.0` [...]
-| [hoodie.write.set.null.for.missing.columns](#hoodiewritesetnullformissingcolumns) | false | When a nullable column is missing from incoming batch during a write operation, the write operation will fail schema compatibility check. Set this option to true will make the missing column be filled with null values to successfully complete the write operation.<br />`Config Param [...]
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` [...]
+| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` [...]
+| [hoodie.datasource.overwrite.mode](#hoodiedatasourceoverwritemode) | (N/A) | Controls whether overwrite use dynamic or static mode, if not configured, respect spark.sql.sources.partitionOverwriteMode<br />`Config Param: OVERWRITE_MODE`<br />`Since Version: 0.14.0` [...]
+| [hoodie.datasource.write.partitions.to.delete](#hoodiedatasourcewritepartitionstodelete) | (N/A) | Comma separated list of partitions to delete. Allows use of wildcard *<br />`Config Param: PARTITIONS_TO_DELETE` [...]
+| [hoodie.datasource.write.payload.class](#hoodiedatasourcewritepayloadclass) | (N/A) | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br />`Config Param: PAYLOAD_CLASS_NAME` [...]
+| [hoodie.datasource.write.table.name](#hoodiedatasourcewritetablename) | (N/A) | Table name for the datasource write. Also used to register the table into meta stores.<br />`Config Param: TABLE_NAME` [...]
+| [hoodie.write.record.merge.custom.implementation.classes](#hoodiewriterecordmergecustomimplementationclasses) | (N/A) | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These record merge impls will filter by hoodie.write.record.merge.strategy.idHudi will pick most efficient implementation to perform merging/combining of the records (during update, re [...]
+| [hoodie.write.record.merge.strategy.id](#hoodiewriterecordmergestrategyid) | (N/A) | ID of record merge strategy. Hudi will pick HoodieRecordMerger implementations in `hoodie.write.record.merge.custom.implementation.classes` which has the same merge strategy id<br />`Config Param: RECORD_MERGE_STRATEGY_ID`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.compaction.async.enable](#hoodiedatasourcecompactionasyncenable) | true | Controls whether async compaction should be turned on for MOR table writing.<br />`Config Param: ASYNC_COMPACT_ENABLE` [...]
+| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` [...]
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: HIVE_BASE_FILE_FORMAT` [...]
+| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` [...]
+| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` [...]
+| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` [...]
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: HIVE_DATABASE` [...]
+| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` [...]
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: HIVE_PARTITION_EXTRACTOR_CLASS` [...]
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: HIVE_PARTITION_FIELDS` [...]
+| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` [...]
+| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` [...]
+| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` [...]
+| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` [...]
+| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` [...]
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: HIVE_TABLE` [...]
+| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` [...]
+| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Config Pa [...]
+| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` [...]
+| [hoodie.datasource.insert.dup.policy](#hoodiedatasourceinsertduppolicy) | none | **Note** This is only applicable to Spark SQL writing.<br />When operation type is set to "insert", users can optionally enforce a dedup policy. This policy will be employed when records being ingested already exists in storage. Default policy is none and no action will be taken. [...]
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: HIVE_CONDITIONAL_SYNC` [...]
+| [hoodie.datasource.write.commitmeta.key.prefix](#hoodiedatasourcewritecommitmetakeyprefix) | _ | Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata. This is useful to store checkpointing information, in a consistent way with the hudi timeline<br />`Config Param: COMMIT_METADATA_KEYPREFIX` [...]
+| [hoodie.datasource.write.drop.partition.columns](#hoodiedatasourcewritedroppartitioncolumns) | false | When set to true, will not write the partition columns into hudi. By default, false.<br />`Config Param: DROP_PARTITION_COLUMNS` [...]
+| [hoodie.datasource.write.insert.drop.duplicates](#hoodiedatasourcewriteinsertdropduplicates) | false | If set to true, records from the incoming dataframe will not overwrite existing records with the same key during the write operation. <br /> **Note** Just for Insert operation in Spark SQL writing since 0.14.0, users can switch to the config `hoodie.datasource.insert.dup.policy` i [...]
+| [hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass) | org.apache.hudi.keygen.SimpleKeyGenerator | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`<br />`Config Param: KEYGENERATOR_CLASS_NAME` [...]
+| [hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled](#hoodiedatasourcewritekeygeneratorconsistentlogicaltimestampenabled) | false | When set to true, consistent value will be generated for a logical timestamp type column, like timestamp-millis and timestamp-micros, irrespective of whether row-writer is enabled. Disabled by default so as not to break the pipeline that deploy either fully row-writer path or non row-wr [...]
+| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Should we url encode the partition path value, before creating the folder structure.<br />`Config Param: URL_ENCODE_PARTITIONING` [...]
+| [hoodie.datasource.write.reconcile.schema](#hoodiedatasourcewritereconcileschema) | false | This config controls how writer's schema will be selected based on the incoming batch's schema as well as existing table's one. When schema reconciliation is DISABLED, incoming batch's schema will be picked as a writer-schema (therefore updating table's schema). When schema reconciliati [...]
+| [hoodie.datasource.write.row.writer.enable](#hoodiedatasourcewriterowwriterenable) | true | When set to true, will perform write operations directly using the spark native `Row` representation, avoiding any additional conversion costs.<br />`Config Param: ENABLE_ROW_WRITER` [...]
+| [hoodie.datasource.write.streaming.checkpoint.identifier](#hoodiedatasourcewritestreamingcheckpointidentifier) | default_single_writer | A stream identifier used for HUDI to fetch the right checkpoint(`batch id` to be more specific) corresponding this writer. Please note that keep the identifier an unique value for different writer if under multi-writer scenario. If the value is not set, will only keep the checkpoint inf [...]
+| [hoodie.datasource.write.streaming.disable.compaction](#hoodiedatasourcewritestreamingdisablecompaction) | false | By default for MOR table, async compaction is enabled with spark streaming sink. By setting this config to true, we can disable it and the expectation is that, users will schedule and execute compaction in a different process/job altogether. Some users may wish to run it separately to m [...]
+| [hoodie.datasource.write.streaming.ignore.failed.batch](#hoodiedatasourcewritestreamingignorefailedbatch) | false | Config to indicate whether to ignore any non exception error (e.g. writestatus error) within a streaming microbatch. Turning this on, could hide the write status errors while the spark checkpoint moves ahead.So, would recommend users to use this with caution.<br />`Config Param: STREAMI [...]
+| [hoodie.datasource.write.streaming.retry.count](#hoodiedatasourcewritestreamingretrycount) | 3 | Config to indicate how many times streaming job should retry for a failed micro batch.<br />`Config Param: STREAMING_RETRY_CNT` [...]
+| [hoodie.datasource.write.streaming.retry.interval.ms](#hoodiedatasourcewritestreamingretryintervalms) | 2000 | Config to indicate how long (by millisecond) before a retry should issued for failed microbatch<br />`Config Param: STREAMING_RETRY_INTERVAL_MS` [...]
+| [hoodie.meta.sync.client.tool.class](#hoodiemetasyncclienttoolclass) | org.apache.hudi.hive.HiveSyncTool | Sync tool class name used to sync to metastore. Defaults to Hive.<br />`Config Param: META_SYNC_CLIENT_TOOL_CLASS_NAME` [...]
+| [hoodie.spark.sql.insert.into.operation](#hoodiesparksqlinsertintooperation) | insert | Sql write operation to use with INSERT_INTO spark sql command. This comes with 3 possible values, bulk_insert, insert and upsert. bulk_insert is generally meant for initial loads and is known to be performant compared to insert. But bulk_insert may not do small file management. If you p [...]
+| [hoodie.spark.sql.merge.into.partial.updates](#hoodiesparksqlmergeintopartialupdates) | true | Whether to write partial updates to the data blocks containing updates in MOR tables with Spark SQL MERGE INTO statement. The data blocks containing partial updates have a schema with a subset of fields compared to the full schema of the table.<br />`Config Param: ENABLE_MERGE_INTO_PART [...]
+| [hoodie.spark.sql.optimized.writes.enable](#hoodiesparksqloptimizedwritesenable) | true | Controls whether spark sql prepped update, delete, and merge are enabled.<br />`Config Param: SPARK_SQL_OPTIMIZED_WRITES`<br />`Since Version: 0.14.0` [...]
+| [hoodie.sql.bulk.insert.enable](#hoodiesqlbulkinsertenable) | false | When set to true, the sql insert statement will use bulk insert. This config is deprecated as of 0.14.0. Please use hoodie.spark.sql.insert.into.operation instead.<br />`Config Param: SQL_ENABLE_BULK_INSERT` [...]
+| [hoodie.sql.insert.mode](#hoodiesqlinsertmode) | upsert | Insert mode when insert data to pk-table. The optional modes are: upsert, strict and non-strict.For upsert mode, insert statement do the upsert operation for the pk-table which will update the duplicate record.For strict mode, insert statement will keep the primary key uniqueness constr [...]
+| [hoodie.streamer.source.kafka.value.deserializer.class](#hoodiestreamersourcekafkavaluedeserializerclass) | io.confluent.kafka.serializers.KafkaAvroDeserializer | This class is used by kafka client to deserialize the records<br />`Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.9.0` [...]
+| [hoodie.write.set.null.for.missing.columns](#hoodiewritesetnullformissingcolumns) | false | When a nullable column is missing from incoming batch during a write operation, the write operation will fail schema compatibility check. Set this option to true will make the missing column be filled with null values to successfully complete the write operation.<br />`Config Param: S [...]
---
@@ -250,9 +262,9 @@ The following set of configurations help validate new data before commits.
| Config Name | Default | Description |
| ------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [hoodie.precommit.validators](#hoodieprecommitvalidators) | | Comma separated list of class names that can be invoked to validate commit<br />`Config Param: VALIDATOR_CLASS_NAMES` |
-| [hoodie.precommit.validators.equality.sql.queries](#hoodieprecommitvalidatorsequalitysqlqueries) | | Spark SQL queries to run on table before committing new data to validate state before and after commit. Multiple queries separated by ';' delimiter are supported. Example: "select count(*) from <TABLE_NAME> Note <TABLE_NAME> is replaced by table state before and after commit.<br />`Config Param: EQUALITY_SQL_QUERIES` |
-| [hoodie.precommit.validators.inequality.sql.queries](#hoodieprecommitvalidatorsinequalitysqlqueries) | | Spark SQL queries to run on table before committing new data to validate state before and after commit.Multiple queries separated by ';' delimiter are supported.Example query: 'select count(*) from <TABLE_NAME> where col=null'Note <TABLE_NAME> variable is expected to be present in query.<br />`Config Param: INEQUALITY_SQL_QUERIES` |
-| [hoodie.precommit.validators.single.value.sql.queries](#hoodieprecommitvalidatorssinglevaluesqlqueries) | | Spark SQL queries to run on table before committing new data to validate state after commit.Multiple queries separated by ';' delimiter are supported.Expected result is included as part of query separated by '#'. Example query: 'query1#result1:query2#result2'Note <TABLE_NAME> variable is expected to be present in query.<br />`Config Param: SINGLE_VALUE_SQL_QUERIES` |
+| [hoodie.precommit.validators.equality.sql.queries](#hoodieprecommitvalidatorsequalitysqlqueries) | | Spark SQL queries to run on table before committing new data to validate state before and after commit. Multiple queries separated by ';' delimiter are supported. Example: "select count(*) from \<TABLE_NAME\> Note \<TABLE_NAME\> is replaced by table state before and after commit.<br />`Config Param: EQUALITY_SQL_QUERIES` |
+| [hoodie.precommit.validators.inequality.sql.queries](#hoodieprecommitvalidatorsinequalitysqlqueries) | | Spark SQL queries to run on table before committing new data to validate state before and after commit.Multiple queries separated by ';' delimiter are supported.Example query: 'select count(*) from \<TABLE_NAME\> where col=null'Note \<TABLE_NAME\> variable is expected to be present in query.<br />`Config Param: INEQUALITY_SQL_QUERIES` |
+| [hoodie.precommit.validators.single.value.sql.queries](#hoodieprecommitvalidatorssinglevaluesqlqueries) | | Spark SQL queries to run on table before committing new data to validate state after commit.Multiple queries separated by ';' delimiter are supported.Expected result is included as part of query separated by '#'. Example query: 'query1#result1:query2#result2'Note \<TABLE_NAME\> variable is expected to be present in query.<br />`Config Param: SINGLE_VALUE_SQL_QUERIES` |
---
## Flink Sql Configs {#FLINK_SQL}
@@ -272,6 +284,7 @@ Flink jobs using the SQL can be configured through the options in WITH clause. T
| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name to register to Hive metastore<br /> `Config Param: DATABASE_NAME` [...]
| [hoodie.table.name](#hoodietablename) | (N/A) | Table name to register to Hive metastore<br /> `Config Param: TABLE_NAME` [...]
| [path](#path) | (N/A) | Base path for the target hoodie table. The path would be created if it does not exist, otherwise a Hoodie table expects to be initialized successfully<br /> `Config Param: PATH` [...]
+| [read.commits.limit](#readcommitslimit) | (N/A) | The maximum number of commits allowed to read in each instant check, if it is streaming read, the avg read instants number per-second would be 'read.commits.limit'/'read.streaming.check-interval', by default no limit<br /> `Config Param: READ_COMMITS_LIMIT` [...]
| [read.end-commit](#readend-commit) | (N/A) | End commit instant for reading, the commit time format should be 'yyyyMMddHHmmss'<br /> `Config Param: READ_END_COMMIT` [...]
| [read.start-commit](#readstart-commit) | (N/A) | Start commit instant for reading, the commit time format should be 'yyyyMMddHHmmss', by default reading from the latest instant for streaming read<br /> `Config Param: READ_START_COMMIT` [...]
| [archive.max_commits](#archivemax_commits) | 50 | Max number of commits to keep before archiving older commits into a sequential log, default 50<br /> `Config Param: ARCHIVE_MAX_COMMITS` [...]
@@ -295,10 +308,12 @@ Flink jobs using the SQL can be configured through the options in WITH clause. T
| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | | Partition path field. Value to be used at the `partitionPath` component of `HoodieKey`. Actual value obtained by invoking .toString(), default ''<br /> `Config Param: PARTITION_PATH_FIELD` [...]
| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | uuid | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br /> `Config Param: RECORD_KEY_FIELD` [...]
| [index.type](#indextype) | FLINK_STATE | Index type of Flink write job, default is using state backed index.<br /> `Config Param: INDEX_TYPE` [...]
+| [lookup.join.cache.ttl](#lookupjoincachettl) | PT1H | The cache TTL (e.g. 10min) for the build table in lookup join.<br /> `Config Param: LOOKUP_JOIN_CACHE_TTL` [...]
| [metadata.compaction.delta_commits](#metadatacompactiondelta_commits) | 10 | Max delta commits for metadata table to trigger compaction, default 10<br /> `Config Param: METADATA_COMPACTION_DELTA_COMMITS` [...]
-| [metadata.enabled](#metadataenabled) | false | Enable the internal metadata table which serves table metadata like level file listings, default disabled<br /> `Config Param: METADATA_ENABLED` [...]
+| [metadata.enabled](#metadataenabled) | true | Enable the internal metadata table which serves table metadata like level file listings, default enabled<br /> `Config Param: METADATA_ENABLED` [...]
| [precombine.field](#precombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br /> `Config Param: PRECOMBINE_FIELD` [...]
| [read.streaming.enabled](#readstreamingenabled) | false | Whether to read as streaming source, default false<br /> `Config Param: READ_AS_STREAMING` [...]
+| [read.streaming.skip_insertoverwrite](#readstreamingskip_insertoverwrite) | false | Whether to skip insert overwrite instants to avoid reading base files of insert overwrite operations for streaming read. In streaming scenarios, insert overwrite is usually used to repair data, here you can control the visibility of downstream streaming read.<br /> `Config Param: READ_STREAMING_SKIP_INSERT_OVERWRITE` [...]
| [table.type](#tabletype) | COPY_ON_WRITE | Type of table to write. COPY_ON_WRITE (or) MERGE_ON_READ<br /> `Config Param: TABLE_TYPE` [...]
| [write.operation](#writeoperation) | upsert | The write operation, that this write should do<br /> `Config Param: OPERATION` [...]
| [write.parquet.max.file.size](#writeparquetmaxfilesize) | 120 | Target size for parquet files produced by Hudi write phases. For DFS, this needs to be aligned with the underlying filesystem block size for optimal performance.<br /> `Config Param: WRITE_PARQUET_MAX_FILE_SIZE` [...]
@@ -358,23 +373,25 @@ Flink jobs using the SQL can be configured through the options in WITH clause. T
| [hive_sync.username](#hive_syncusername) | hive | Username for hive sync, default 'hive'<br /> `Config Param: HIVE_SYNC_USERNAME` [...]
| [hoodie.bucket.index.hash.field](#hoodiebucketindexhashfield) | | Index key field. Value to be used as hashing to find the bucket ID. Should be a subset of or equal to the recordKey fields. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br /> `Config Par [...]
| [hoodie.bucket.index.num.buckets](#hoodiebucketindexnumbuckets) | 4 | Hudi bucket number per partition. Only affected if using Hudi bucket index.<br /> `Config Param: BUCKET_INDEX_NUM_BUCKETS` [...]
-| [hoodie.datasource.merge.type](#hoodiedatasourcemergetype) | payload_combine | For Snapshot query on merge on read table. Use this key to define how the payloads are merged, in 1) skip_merge: read the base file records plus the log file records; 2) payload_combine: read the base file records first, for each record in base file, checks whether the key is in t [...]
+| [hoodie.datasource.merge.type](#hoodiedatasourcemergetype) | payload_combine | For Snapshot query on merge on read table. Use this key to define how the payloads are merged, in 1) skip_merge: read the base file records plus the log file records without merging; 2) payload_combine: read the base file records first, for each record in base file, checks whether [...]
| [hoodie.datasource.write.keygenerator.type](#hoodiedatasourcewritekeygeneratortype) | SIMPLE | Key generator type, that implements will extract the key out of incoming record. **Note** This is being actively worked on. Please use `hoodie.datasource.write.keygenerator.class` instead.<br /> `Config Param: KEYGEN_TYPE` [...]
| [hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode) | false | Whether to encode the partition path url, default false<br /> `Config Param: URL_ENCODE_PARTITIONING` [...]
| [hoodie.index.bucket.engine](#hoodieindexbucketengine) | SIMPLE | Type of bucket index engine. Available options: [SIMPLE | CONSISTENT_HASHING]<br /> `Config Param: BUCKET_INDEX_ENGINE_TYPE` [...]
+| [hoodie.write.table.version](#hoodiewritetableversion) | 8 | Table version produced by this writer.<br /> `Config Param: WRITE_TABLE_VERSION` [...]
| [index.bootstrap.enabled](#indexbootstrapenabled) | false | Whether to bootstrap the index state from existing hoodie table, default false<br /> `Config Param: INDEX_BOOTSTRAP_ENABLED` [...]
| [index.global.enabled](#indexglobalenabled) | true | Whether to update index for the old partition path if same key record with different partition path came in, default true<br /> `Config Param: INDEX_GLOBAL_ENABLED` [...]
| [index.partition.regex](#indexpartitionregex) | .* | Whether to load partitions in state if partition path matching， default `*`<br /> `Config Param: INDEX_PARTITION_REGEX` [...]
| [index.state.ttl](#indexstatettl) | 0.0 | Index state ttl in days, default stores the index permanently<br /> `Config Param: INDEX_STATE_TTL` [...]
| [partition.default_name](#partitiondefault_name) | __HIVE_DEFAULT_PARTITION__ | The default partition name in case the dynamic partition column value is null/empty string<br /> `Config Param: PARTITION_DEFAULT_NAME` [...]
| [payload.class](#payloadclass) | org.apache.hudi.common.model.EventTimeAvroPayload | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for the option in-effective<br /> `Config Param: PAYLOAD_CLASS_NAME` [...]
-| [read.data.skipping.enabled](#readdataskippingenabled) | false | Enables data-skipping allowing queries to leverage indexes to reduce the search space byskipping over files<br /> `Config Param: READ_DATA_SKIPPING_ENABLED` [...]
+| [read.cdc.from.changelog](#readcdcfromchangelog) | true | Whether to consume the delta changes only from the cdc changelog files. When CDC is enabled, i). for COW table, the changelog is generated on each file update; ii). for MOR table, the changelog is generated on compaction. By default, always read from the changelog file, once it is [...]
+| [read.data.skipping.enabled](#readdataskippingenabled) | false | Enables data-skipping allowing queries to leverage indexes to reduce the search space by skipping over files<br /> `Config Param: READ_DATA_SKIPPING_ENABLED` [...]
| [read.streaming.check-interval](#readstreamingcheck-interval) | 60 | Check interval for streaming read of SECOND, default 1 minute<br /> `Config Param: READ_STREAMING_CHECK_INTERVAL` [...]
| [read.streaming.skip_clustering](#readstreamingskip_clustering) | true | Whether to skip clustering instants to avoid reading base files of clustering operations for streaming read to improve read performance.<br /> `Config Param: READ_STREAMING_SKIP_CLUSTERING` [...]
| [read.streaming.skip_compaction](#readstreamingskip_compaction) | true | Whether to skip compaction instants and avoid reading compacted base files for streaming read to improve read performance. This option can be used to avoid reading duplicates when changelog mode is enabled, it is a solution to keep data integrity <br /> `Config Param: READ_STREAMI [...]
| [read.utc-timezone](#readutc-timezone) | true | Use UTC timezone or local timezone to the conversion between epoch time and LocalDateTime. Hive 0.x/1.x/2.x use local timezone. But Hive 3.x use UTC timezone, by default true<br /> `Config Param: READ_UTC_TIMEZONE` [...]
| [record.merger.impls](#recordmergerimpls) | org.apache.hudi.common.model.HoodieAvroRecordMerger | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These merger impls will filter by record.merger.strategy. Hudi will pick most efficient implementation to perform merging/combining of the records (during update, reading MOR tab [...]
-| [record.merger.strategy](#recordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in record.merger.impls which has the same merger strategy id<br /> `Config Param: RECORD_MERGER_STRATEGY` [...]
+| [record.merger.strategy](#recordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in record.merger.impls which has the same merger strategy id<br /> `Config Param: RECORD_MERGER_STRATEGY_ID` [...]
| [write.batch.size](#writebatchsize) | 256.0 | Batch buffer size in MB to flush data into the underneath filesystem, default 256MB<br /> `Config Param: WRITE_BATCH_SIZE` [...]
| [write.bulk_insert.shuffle_input](#writebulk_insertshuffle_input) | true | Whether to shuffle the inputs by specific fields for bulk insert tasks, default true<br /> `Config Param: WRITE_BULK_INSERT_SHUFFLE_INPUT` [...]
| [write.bulk_insert.sort_input](#writebulk_insertsort_input) | true | Whether to sort the inputs by specific fields for bulk insert tasks, default true<br /> `Config Param: WRITE_BULK_INSERT_SORT_INPUT` [...]
@@ -460,11 +477,14 @@ Configurations used by the Hudi Metadata Table. This table maintains the metadat
[**Basic Configs**](#Metadata-Configs-basic-configs)
-| Config Name | Default | Description |
-| ---------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| [hoodie.metadata.enable](#hoodiemetadataenable) | true | Enable the internal metadata table which serves table metadata like level file listings<br />`Config Param: ENABLE`<br />`Since Version: 0.7.0` |
-| [hoodie.metadata.index.bloom.filter.enable](#hoodiemetadataindexbloomfilterenable) | false | Enable indexing bloom filters of user data files under metadata table. When enabled, metadata table will have a partition to store the bloom filter index and will be used during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_BLOOM_FILTER`<br />`Since Version: 0.11.0` |
-| [hoodie.metadata.index.column.stats.enable](#hoodiemetadataindexcolumnstatsenable) | false | Enable indexing column ranges of user data files under metadata table key lookups. When enabled, metadata table will have a partition to store the column ranges and will be used for pruning files during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_COLUMN_STATS`<br />`Since Version: 0.11.0` |
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.metadata.enable](#hoodiemetadataenable) | true | Enable the internal metadata table which serves table metadata like level file listings<br />`Config Param: ENABLE`<br />`Since Version: 0.7.0` [...]
+| [hoodie.metadata.index.bloom.filter.enable](#hoodiemetadataindexbloomfilterenable) | false | Enable indexing bloom filters of user data files under metadata table. When enabled, metadata table will have a partition to store the bloom filter index and will be used during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_BLOOM_FILTER`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.column.stats.enable](#hoodiemetadataindexcolumnstatsenable) | false | Enable indexing column ranges of user data files under metadata table key lookups. When enabled, metadata table will have a partition to store the column ranges and will be used for pruning files during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_COLUMN_STATS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.expression.enable](#hoodiemetadataindexexpressionenable) | false | Enable expression index within the metadata table. When this configuration property is enabled (`true`), the Hudi writer automatically keeps all expression indexes consistent with the data table. When disabled (`false`), all expression indexes are deleted. Note that individual expression index can only be created through a `CREATE INDEX` and deleted through a `DROP INDEX` statement [...]
+| [hoodie.metadata.index.partition.stats.enable](#hoodiemetadataindexpartitionstatsenable) | false | Enable aggregating stats for each column at the storage partition level.<br />`Config Param: ENABLE_METADATA_INDEX_PARTITION_STATS`<br />`Since Version: 1.0.0` [...]
+| [hoodie.metadata.index.secondary.enable](#hoodiemetadataindexsecondaryenable) | true | Enable secondary index within the metadata table. When this configuration property is enabled (`true`), the Hudi writer automatically keeps all secondary indexes consistent with the data table. When disabled (`false`), all secondary indexes are deleted. Note that individual secondary index can only be created through a `CREATE INDEX` and deleted through a `DROP INDEX` statement in [...]
[**Advanced Configs**](#Metadata-Configs-advanced-configs)
@@ -475,7 +495,6 @@ Configurations used by the Hudi Metadata Table. This table maintains the metadat
| [hoodie.metadata.index.column.stats.column.list](#hoodiemetadataindexcolumnstatscolumnlist) | (N/A) | Comma-separated list of columns for which column stats index will be built. If not set, all columns will be indexed<br />`Config Param: COLUMN_STATS_INDEX_FOR_COLUMNS`<br />`Since Version: 0.11.0` [...]
| [hoodie.metadata.index.column.stats.processing.mode.override](#hoodiemetadataindexcolumnstatsprocessingmodeoverride) | (N/A) | By default Column Stats Index is automatically determining whether it should be read and processed either'in-memory' (w/in executing process) or using Spark (on a cluster), based on some factors like the size of the Index and how many columns are read. This config allows to override this behavior.<br />`Config Param: COLUMN_STATS_INDEX_PROCESSIN [...]
| [_hoodie.metadata.ignore.spurious.deletes](#_hoodiemetadataignorespuriousdeletes) | true | There are cases when extra files are requested to be deleted from metadata table which are never added before. This config determines how to handle such spurious deletes<br />`Config Param: IGNORE_SPURIOUS_DELETES`<br />`Since Version: 0.10.0` [...]
-| [hoodie.assume.date.partitioning](#hoodieassumedatepartitioning) | false | Should HoodieWriteClient assume the data is partitioned by dates, i.e three levels from base path. This is a stop-gap to support tables created by versions < 0.3.1. Will be removed eventually<br />`Config Param: ASSUME_DATE_PARTITIONING`<br />`Since Version: 0.3.0` [...]
| [hoodie.file.listing.parallelism](#hoodiefilelistingparallelism) | 200 | Parallelism to use, when listing the table on lake storage.<br />`Config Param: FILE_LISTING_PARALLELISM_VALUE`<br />`Since Version: 0.7.0` [...]
| [hoodie.metadata.auto.initialize](#hoodiemetadataautoinitialize) | true | Initializes the metadata table by reading from the file system when the table is first created. Enabled by default. Warning: This should only be disabled when manually constructing the metadata table outside of typical Hudi writer flows.<br />`Config Param: AUTO_INITIALIZE`<br />`Since Version: 0.14.0` [...]
| [hoodie.metadata.compact.max.delta.commits](#hoodiemetadatacompactmaxdeltacommits) | 10 | Controls how often the metadata table is compacted.<br />`Config Param: COMPACT_NUM_DELTA_COMMITS`<br />`Since Version: 0.7.0` [...]
@@ -486,7 +505,13 @@ Configurations used by the Hudi Metadata Table. This table maintains the metadat
| [hoodie.metadata.index.check.timeout.seconds](#hoodiemetadataindexchecktimeoutseconds) | 900 | After the async indexer has finished indexing upto the base instant, it will ensure that all inflight writers reliably write index updates as well. If this timeout expires, then the indexer will abort itself safely.<br />`Config Param: METADATA_INDEX_CHECK_TIMEOUT_SECONDS`<br />`Since Version: 0.11.0` [...]
| [hoodie.metadata.index.column.stats.file.group.count](#hoodiemetadataindexcolumnstatsfilegroupcount) | 2 | Metadata column stats partition file group count. This controls the size of the base and log files and read parallelism in the column stats index partition. The recommendation is to size the file group count such that the base files are under 1GB.<br />`Config Param: METADATA_INDEX_COLUMN_STATS_FILE_GROUP_COUNT`<br />`Since Version: 0.11.0` [...]
| [hoodie.metadata.index.column.stats.inMemory.projection.threshold](#hoodiemetadataindexcolumnstatsinMemoryprojectionthreshold) | 100000 | When reading Column Stats Index, if the size of the expected resulting projection is below the in-memory threshold (counted by the # of rows), it will be attempted to be loaded "in-memory" (ie not using the execution engine like Spark, Flink, etc). If the value is above the threshold execution engine will be used to compose the projection.<br /> [...]
+| [hoodie.metadata.index.column.stats.max.columns.to.index](#hoodiemetadataindexcolumnstatsmaxcolumnstoindex) | 32 | Maximum number of columns to generate column stats for. If the config `hoodie.metadata.index.column.stats.column.list` is set, this config will be ignored. If the config `hoodie.metadata.index.column.stats.column.list` is not set, the column stats of the first `n` columns (`n` defined by this config) in the table schema are generated.<br />`Conf [...]
| [hoodie.metadata.index.column.stats.parallelism](#hoodiemetadataindexcolumnstatsparallelism) | 200 | Parallelism to use, when generating column stats index.<br />`Config Param: COLUMN_STATS_INDEX_PARALLELISM`<br />`Since Version: 0.11.0` [...]
+| [hoodie.metadata.index.expression.file.group.count](#hoodiemetadataindexexpressionfilegroupcount) | 2 | Metadata expression index partition file group count.<br />`Config Param: EXPRESSION_INDEX_FILE_GROUP_COUNT`<br />`Since Version: 1.0.0` [...]
+| [hoodie.metadata.index.expression.parallelism](#hoodiemetadataindexexpressionparallelism) | 200 | Parallelism to use, when generating expression index.<br />`Config Param: EXPRESSION_INDEX_PARALLELISM`<br />`Since Version: 1.0.0` [...]
+| [hoodie.metadata.index.partition.stats.file.group.count](#hoodiemetadataindexpartitionstatsfilegroupcount) | 1 | Metadata partition stats file group count. This controls the size of the base and log files and read parallelism in the partition stats index.<br />`Config Param: METADATA_INDEX_PARTITION_STATS_FILE_GROUP_COUNT`<br />`Since Version: 1.0.0` [...]
+| [hoodie.metadata.index.partition.stats.parallelism](#hoodiemetadataindexpartitionstatsparallelism) | 200 | Parallelism to use, when generating partition stats index.<br />`Config Param: PARTITION_STATS_INDEX_PARALLELISM`<br />`Since Version: 1.0.0` [...]
+| [hoodie.metadata.index.secondary.parallelism](#hoodiemetadataindexsecondaryparallelism) | 200 | Parallelism to use, when generating secondary index.<br />`Config Param: SECONDARY_INDEX_PARALLELISM`<br />`Since Version: 1.0.0` [...]
| [hoodie.metadata.log.compaction.blocks.threshold](#hoodiemetadatalogcompactionblocksthreshold) | 5 | Controls the criteria to log compacted files groups in metadata table.<br />`Config Param: LOG_COMPACT_BLOCKS_THRESHOLD`<br />`Since Version: 0.14.0` [...]
| [hoodie.metadata.log.compaction.enable](#hoodiemetadatalogcompactionenable) | false | This configs enables logcompaction for the metadata table.<br />`Config Param: ENABLE_LOG_COMPACTION_ON_METADATA_TABLE`<br />`Since Version: 0.14.0` [...]
| [hoodie.metadata.max.deltacommits.when_pending](#hoodiemetadatamaxdeltacommitswhen_pending) | 1000 | When there is a pending instant in data table, this config limits the allowed number of deltacommits in metadata table to prevent the metadata table's timeline from growing unboundedly as compaction won't be triggered due to the pending data table instant.<br />`Config Param: METADATA_MAX_NUM_DELTACOMMITS_WHEN_PENDING`<br />`Since Version: 0.14. [...]
@@ -513,14 +538,14 @@ Configurations used by the Hudi Metaserver.
[**Advanced Configs**](#Metaserver-Configs-advanced-configs)
-| Config Name | Default | Description |
-| --------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name that will be used for incremental query.If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME`<br />`Since Version: 0.13.0` |
-| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: TABLE_NAME`<br />`Since Version: 0.13.0` |
-| [hoodie.metaserver.connect.retries](#hoodiemetaserverconnectretries) | 3 | Number of retries while opening a connection to metaserver<br />`Config Param: METASERVER_CONNECTION_RETRIES`<br />`Since Version: 0.13.0` |
-| [hoodie.metaserver.connect.retry.delay](#hoodiemetaserverconnectretrydelay) | 1 | Number of seconds for the client to wait between consecutive connection attempts<br />`Config Param: METASERVER_CONNECTION_RETRY_DELAY`<br />`Since Version: 0.13.0` |
-| [hoodie.metaserver.enabled](#hoodiemetaserverenabled) | false | Enable Hudi metaserver for storing Hudi tables' metadata.<br />`Config Param: METASERVER_ENABLE`<br />`Since Version: 0.13.0` |
-| [hoodie.metaserver.uris](#hoodiemetaserveruris) | thrift://localhost:9090 | Metaserver server uris<br />`Config Param: METASERVER_URLS`<br />`Since Version: 0.13.0` |
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [hoodie.database.name](#hoodiedatabasename) | (N/A) | Database name. If different databases have the same table name during incremental query, we can set it to limit the table name under a specific database<br />`Config Param: DATABASE_NAME`<br />`Since Version: 0.13.0` |
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with Hive. Needs to be same across runs.<br />`Config Param: TABLE_NAME`<br />`Since Version: 0.13.0` |
+| [hoodie.metaserver.connect.retries](#hoodiemetaserverconnectretries) | 3 | Number of retries while opening a connection to metaserver<br />`Config Param: METASERVER_CONNECTION_RETRIES`<br />`Since Version: 0.13.0` |
+| [hoodie.metaserver.connect.retry.delay](#hoodiemetaserverconnectretrydelay) | 1 | Number of seconds for the client to wait between consecutive connection attempts<br />`Config Param: METASERVER_CONNECTION_RETRY_DELAY`<br />`Since Version: 0.13.0` |
+| [hoodie.metaserver.enabled](#hoodiemetaserverenabled) | false | Enable Hudi metaserver for storing Hudi tables' metadata.<br />`Config Param: METASERVER_ENABLE`<br />`Since Version: 0.13.0` |
+| [hoodie.metaserver.uris](#hoodiemetaserveruris) | thrift://localhost:9090 | Metaserver server uris<br />`Config Param: METASERVER_URLS`<br />`Since Version: 0.13.0` |
---
@@ -543,6 +568,7 @@ Configurations that control aspects around writing, sizing, reading base and log
| Config Name | Default | Description [...]
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| [hoodie.logfile.data.block.format](#hoodielogfiledatablockformat) | (N/A) | Format of the data block within delta logs. Following formats are currently supported "avro", "hfile", "parquet"<br />`Config Param: LOGFILE_DATA_BLOCK_FORMAT` [...]
+| [hoodie.parquet.writelegacyformat.enabled](#hoodieparquetwritelegacyformatenabled) | (N/A) | Sets spark.sql.parquet.writeLegacyFormat. If true, data will be written in a way of Spark 1.4 and earlier. For example, decimal values will be written in Parquet's fixed-length byte array format which other systems such as Apache Hive and Apache Impala use. If false, the newer format in Parquet will be used. For example, decimals wil [...]
| [hoodie.avro.write.support.class](#hoodieavrowritesupportclass) | org.apache.hudi.avro.HoodieAvroWriteSupport | Provided write support class should extend HoodieAvroWriteSupport class and it is loaded at runtime. This is only required when trying to override the existing write context.<br />`Config Param: HOODIE_AVRO_WRITE_SUPPORT_CLASS`<br />`Since Version: 0.14.0` [...]
| [hoodie.bloom.index.filter.dynamic.max.entries](#hoodiebloomindexfilterdynamicmaxentries) | 100000 | The threshold for the maximum number of keys to record in a dynamic Bloom filter row. Only applies if filter type is BloomFilterTypeCode.DYNAMIC_V0.<br />`Config Param: BLOOM_FILTER_DYNAMIC_MAX_ENTRIES` [...]
| [hoodie.bloom.index.filter.type](#hoodiebloomindexfiltertype) | DYNAMIC_V0 | org.apache.hudi.common.bloom.BloomFilterTypeCode: Filter type used by Bloom filter. SIMPLE: Bloom filter that is based on the configured size. DYNAMIC_V0(default): Bloom filter that is auto sized based on number of keys.<br />`Config Param: BLOOM_FILTER_TYPE` [...]
@@ -567,7 +593,6 @@ Configurations that control aspects around writing, sizing, reading base and log
| [hoodie.parquet.outputtimestamptype](#hoodieparquetoutputtimestamptype) | TIMESTAMP_MICROS | Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.<br />`Config Param: PARQUET_OUTPUT_TIMESTAMP_TYPE` [...]
| [hoodie.parquet.page.size](#hoodieparquetpagesize) | 1048576 | Parquet page size in bytes. Page is the unit of read within a parquet file. Within a block, pages are compressed separately.<br />`Config Param: PARQUET_PAGE_SIZE` [...]
| [hoodie.parquet.spark.row.write.support.class](#hoodieparquetsparkrowwritesupportclass) | org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport | Provided write support class should extend HoodieRowParquetWriteSupport class and it is loaded at runtime. This is only required when trying to override the existing write context when `hoodie.datasource.write.row.writer.enable=true`.<br />`Config Param: HOODIE_PARQUET_SPARK_ROW_WRITE_SUPPORT_CLASS`<br />`Since Version: 0.15.0` [...]
-| [hoodie.parquet.writelegacyformat.enabled](#hoodieparquetwritelegacyformatenabled) | false | Sets spark.sql.parquet.writeLegacyFormat. If true, data will be written in a way of Spark 1.4 and earlier. For example, decimal values will be written in Parquet's fixed-length byte array format which other systems such as Apache Hive and Apache Impala use. If false, the newer format in Parquet will be used. For example, decimals wil [...]
| [hoodie.storage.class](#hoodiestorageclass) | org.apache.hudi.storage.hadoop.HoodieHadoopStorage | The fully-qualified class name of the `HoodieStorage` implementation class to instantiate. The provided class should implement `org.apache.hudi.storage.HoodieStorage`<br />`Config Param: HOODIE_STORAGE_CLASS`<br />`Since Version: 0.15.0` [...]
---
@@ -632,12 +657,12 @@ Configurations that control how file metadata is stored by Hudi, for transaction
| [hoodie.filesystem.view.rocksdb.base.path](#hoodiefilesystemviewrocksdbbasepath) | /tmp/hoodie_timeline_rocksdb | Path on local storage to use, when storing file system view in embedded kv store/rocksdb.<br />`Config Param: ROCKSDB_BASE_PATH` [...]
| [hoodie.filesystem.view.secondary.type](#hoodiefilesystemviewsecondarytype) | MEMORY | Specifies the secondary form of storage for file system view, if the primary (e.g timeline server) is unavailable.<br />`Config Param: SECONDARY_VIEW_TYPE` [...]
| [hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction](#hoodiefilesystemviewspillablebootstrapbasefilememfraction) | 0.05 | Fraction of the file system view memory, to be used for holding mapping to bootstrap base files.<br />`Config Param: BOOTSTRAP_BASE_FILE_MEM_FRACTION` [...]
-| [hoodie.filesystem.view.spillable.clustering.mem.fraction](#hoodiefilesystemviewspillableclusteringmemfraction) | 0.01 | Fraction of the file system view memory, to be used for holding clustering related metadata.<br />`Config Param: SPILLABLE_CLUSTERING_MEM_FRACTION` [...]
-| [hoodie.filesystem.view.spillable.compaction.mem.fraction](#hoodiefilesystemviewspillablecompactionmemfraction) | 0.8 | Fraction of the file system view memory, to be used for holding compaction related metadata.<br />`Config Param: SPILLABLE_COMPACTION_MEM_FRACTION` [...]
+| [hoodie.filesystem.view.spillable.clustering.mem.fraction](#hoodiefilesystemviewspillableclusteringmemfraction) | 0.02 | Fraction of the file system view memory, to be used for holding clustering related metadata.<br />`Config Param: SPILLABLE_CLUSTERING_MEM_FRACTION` [...]
+| [hoodie.filesystem.view.spillable.compaction.mem.fraction](#hoodiefilesystemviewspillablecompactionmemfraction) | 0.1 | Fraction of the file system view memory, to be used for holding compaction related metadata.<br />`Config Param: SPILLABLE_COMPACTION_MEM_FRACTION` [...]
| [hoodie.filesystem.view.spillable.dir](#hoodiefilesystemviewspillabledir) | /tmp/ | Path on local storage to use, when file system view is held in a spillable map.<br />`Config Param: SPILLABLE_DIR` [...]
-| [hoodie.filesystem.view.spillable.log.compaction.mem.fraction](#hoodiefilesystemviewspillablelogcompactionmemfraction) | 0.8 | Fraction of the file system view memory, to be used for holding log compaction related metadata.<br />`Config Param: SPILLABLE_LOG_COMPACTION_MEM_FRACTION`<br />`Since Version: 0.13.0` [...]
+| [hoodie.filesystem.view.spillable.log.compaction.mem.fraction](#hoodiefilesystemviewspillablelogcompactionmemfraction) | 0.02 | Fraction of the file system view memory, to be used for holding log compaction related metadata.<br />`Config Param: SPILLABLE_LOG_COMPACTION_MEM_FRACTION`<br />`Since Version: 0.13.0` [...]
| [hoodie.filesystem.view.spillable.mem](#hoodiefilesystemviewspillablemem) | 104857600 | Amount of memory to be used in bytes for holding file system view, before spilling to disk.<br />`Config Param: SPILLABLE_MEMORY` [...]
-| [hoodie.filesystem.view.spillable.replaced.mem.fraction](#hoodiefilesystemviewspillablereplacedmemfraction) | 0.01 | Fraction of the file system view memory, to be used for holding replace commit related metadata.<br />`Config Param: SPILLABLE_REPLACED_MEM_FRACTION` [...]
+| [hoodie.filesystem.view.spillable.replaced.mem.fraction](#hoodiefilesystemviewspillablereplacedmemfraction) | 0.05 | Fraction of the file system view memory, to be used for holding replace commit related metadata.<br />`Config Param: SPILLABLE_REPLACED_MEM_FRACTION` [...]
| [hoodie.filesystem.view.type](#hoodiefilesystemviewtype) | MEMORY | File system view provides APIs for viewing the files on the underlying lake storage, as file groups and file slices. This config controls how such a view is held. Options include MEMORY,SPILLABLE_DISK,EMBEDDED_KV_STORE,REMOTE_ONLY,REMOTE_FIRST which provide different trade offs for memory usage and API request performance.<br [...]
---
@@ -658,16 +683,14 @@ Configurations that control archival.
[**Advanced Configs**](#Archival-Configs-advanced-configs)
-| Config Name | Default | Description [...]
-| ------------------------------------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
-| [hoodie.archive.async](#hoodiearchiveasync) | false | Only applies when hoodie.archive.automatic is turned on. When turned on runs archiver async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_ARCHIVE`<br />`Since Version: 0.11.0` [...]
-| [hoodie.archive.automatic](#hoodiearchiveautomatic) | true | When enabled, the archival table service is invoked immediately after each commit, to archive commits if we cross a maximum value of commits. It's recommended to enable this, to ensure number of active commits is bounded.<br />`Config Param: AUTO_ARCHIVE` [...]
-| [hoodie.archive.beyond.savepoint](#hoodiearchivebeyondsavepoint) | false | If enabled, archival will proceed beyond savepoint, skipping savepoint commits. If disabled, archival will stop at the earliest savepoint commit.<br />`Config Param: ARCHIVE_BEYOND_SAVEPOINT`<br />`Since Version: 0.12.0` [...]
-| [hoodie.archive.delete.parallelism](#hoodiearchivedeleteparallelism) | 100 | When performing archival operation, Hudi needs to delete the files of the archived instants in the active timeline in .hoodie folder. The file deletion also happens after merging small archived files into larger ones if enabled. This config limits the Spark parallelism for deleting files in both cases, i.e., parallelism of deleting files does not go above the configured value and the par [...]
-| [hoodie.archive.merge.enable](#hoodiearchivemergeenable) | false | When enable, hoodie will auto merge several small archive files into larger one. It's useful when storage scheme doesn't support append operation.<br />`Config Param: ARCHIVE_MERGE_ENABLE` [...]
-| [hoodie.archive.merge.files.batch.size](#hoodiearchivemergefilesbatchsize) | 10 | The number of small archive files to be merged at once.<br />`Config Param: ARCHIVE_MERGE_FILES_BATCH_SIZE` [...]
-| [hoodie.archive.merge.small.file.limit.bytes](#hoodiearchivemergesmallfilelimitbytes) | 20971520 | This config sets the archive file size limit below which an archive file becomes a candidate to be selected as such a small file.<br />`Config Param: ARCHIVE_MERGE_SMALL_FILE_LIMIT_BYTES` [...]
-| [hoodie.commits.archival.batch](#hoodiecommitsarchivalbatch) | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size.<br />`Config Param: COMMITS_ARCHIVAL_BATCH_SIZE` [...]
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
+| [hoodie.archive.async](#hoodiearchiveasync) | false | Only applies when hoodie.archive.automatic is turned on. When turned on runs archiver async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_ARCHIVE`<br />`Since Version: 0.11.0` [...]
+| [hoodie.archive.automatic](#hoodiearchiveautomatic) | true | When enabled, the archival table service is invoked immediately after each commit, to archive commits if we cross a maximum value of commits. It's recommended to enable this, to ensure number of active commits is bounded.<br />`Config Param: AUTO_ARCHIVE` [...]
+| [hoodie.archive.beyond.savepoint](#hoodiearchivebeyondsavepoint) | false | If enabled, archival will proceed beyond savepoint, skipping savepoint commits. If disabled, archival will stop at the earliest savepoint commit.<br />`Config Param: ARCHIVE_BEYOND_SAVEPOINT`<br />`Since Version: 0.12.0` [...]
+| [hoodie.archive.delete.parallelism](#hoodiearchivedeleteparallelism) | 100 | When performing archival operation, Hudi needs to delete the files of the archived instants in the active timeline in .hoodie folder. The file deletion also happens after merging small archived files into larger ones if enabled. This config limits the Spark parallelism for deleting files in both cases, i.e., parallelism of deleting files does not go above the configured value and the parallelism is [...]
+| [hoodie.commits.archival.batch](#hoodiecommitsarchivalbatch) | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size.<br />`Config Param: COMMITS_ARCHIVAL_BATCH_SIZE` [...]
+| [hoodie.timeline.compaction.batch.size](#hoodietimelinecompactionbatchsize) | 10 | The number of small files to compact at once.<br />`Config Param: TIMELINE_COMPACTION_BATCH_SIZE` [...]
---
@@ -707,27 +730,27 @@ Cleaning (reclamation of older/unused file groups/slices).
[**Basic Configs**](#Clean-Configs-basic-configs)
-| Config Name | Default | Description |
-| ---------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.clean.async](#hoodiecleanasync) | false | Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_CLEAN` |
-| [hoodie.cleaner.commits.retained](#hoodiecleanercommitsretained) | 10 | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br />`Config Param: CLEANER_COMMITS_RETAINED` |
+| Config Name | Default | Description |
+| ------------------------------------------------------------ | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.clean.async.enabled](#hoodiecleanasyncenabled) | false | Only applies when hoodie.clean.automatic is turned on. When turned on runs cleaner async with writing, which can speed up overall write performance.<br />`Config Param: ASYNC_CLEAN` |
+| [hoodie.clean.commits.retained](#hoodiecleancommitsretained) | 10 | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries.<br />`Config Param: CLEANER_COMMITS_RETAINED` |
[**Advanced Configs**](#Clean-Configs-advanced-configs)
-| Config Name | Default | Description [...]
-| ---------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.clean.allow.multiple](#hoodiecleanallowmultiple) | false | Allows scheduling/executing multiple cleans by enabling this config. If users prefer to strictly ensure clean requests should be mutually exclusive, .i.e. a 2nd clean will not be scheduled if another clean is not yet completed to avoid repeat cleaning of same files, they might want to disable this config.<br />`Config Param: ALLOW_MULTIPLE_CLEANS`<br />`Since Version: 0.11.0`<br [...]
-| [hoodie.clean.automatic](#hoodiecleanautomatic) | true | When enabled, the cleaner table service is invoked immediately after each commit, to delete older file slices. It's recommended to enable this, to ensure metadata and data storage growth is bounded.<br />`Config Param: AUTO_CLEAN` [...]
-| [hoodie.clean.max.commits](#hoodiecleanmaxcommits) | 1 | Number of commits after the last clean operation, before scheduling of a new clean is attempted.<br />`Config Param: CLEAN_MAX_COMMITS` [...]
-| [hoodie.clean.trigger.strategy](#hoodiecleantriggerstrategy) | NUM_COMMITS | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits`.<br />`Config Param: CLEAN_TRIGGER_STRATEGY` [...]
-| [hoodie.cleaner.delete.bootstrap.base.file](#hoodiecleanerdeletebootstrapbasefile) | false | When set to true, cleaner also deletes the bootstrap base file when it's skeleton base file is cleaned. Turn this to true, if you want to ensure the bootstrap dataset storage is reclaimed over time, as the table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap base files are also physically deleted, to comply with data privac [...]
-| [hoodie.cleaner.fileversions.retained](#hoodiecleanerfileversionsretained) | 3 | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning.<br />`Config Param: CLEANER_FILE_VERSIONS_RETAINED` [...]
-| [hoodie.cleaner.hours.retained](#hoodiecleanerhoursretained) | 24 | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of hours to [...]
-| [hoodie.cleaner.incremental.mode](#hoodiecleanerincrementalmode) | true | When enabled, the plans for each cleaner service run is computed incrementally off the events in the timeline, since the last cleaner run. This is much more efficient than obtaining listings for the full table for each planning (even with a metadata table).<br />`Config Param: CLEANER_INCREMENTAL_MODE_ENABLE` [...]
-| [hoodie.cleaner.parallelism](#hoodiecleanerparallelism) | 200 | This config controls the behavior of both the cleaning plan and cleaning execution. Deriving the cleaning plan is parallelized at the table partition level, i.e., each table partition is processed by one Spark task to figure out the files to clean. The cleaner picks the configured parallelism if the number of table partitions is larger than this configured value. The parallelism [...]
-| [hoodie.cleaner.policy](#hoodiecleanerpolicy) | KEEP_LATEST_COMMITS | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query execu [...]
-| [hoodie.cleaner.policy.failed.writes](#hoodiecleanerpolicyfailedwrites) | EAGER | org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy: Policy that controls how to clean up failed writes. Hudi will delete any files written by failed writes to re-claim space. EAGER(default): Clean failed writes inline after every write operation. LAZY: Clean failed writes lazily after heartbeat timeout when the cleaning service runs. This policy is required whe [...]
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------ | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.clean.automatic](#hoodiecleanautomatic) | true | When enabled, the cleaner table service is invoked immediately after each commit, to delete older file slices. It's recommended to enable this, to ensure metadata and data storage growth is bounded.<br />`Config Param: AUTO_CLEAN` [...]
+| [hoodie.clean.delete.bootstrap.base.file](#hoodiecleandeletebootstrapbasefile) | false | When set to true, cleaner also deletes the bootstrap base file when it's skeleton base file is cleaned. Turn this to true, if you want to ensure the bootstrap dataset storage is reclaimed over time, as the table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap base files are also physically deleted, to comply with data privacy en [...]
+| [hoodie.clean.failed.writes.policy](#hoodiecleanfailedwritespolicy) | EAGER | org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy: Policy that controls how to clean up failed writes. Hudi will delete any files written by failed writes to re-claim space. EAGER(default): Clean failed writes inline after every write operation. LAZY: Clean failed writes lazily after heartbeat timeout when the cleaning service runs. This policy is required when mu [...]
+| [hoodie.clean.fileversions.retained](#hoodiecleanfileversionsretained) | 3 | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning.<br />`Config Param: CLEANER_FILE_VERSIONS_RETAINED` [...]
+| [hoodie.clean.hours.retained](#hoodiecleanhoursretained) | 24 | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of hours to be [...]
+| [hoodie.clean.incremental.enabled](#hoodiecleanincrementalenabled) | true | When enabled, the plans for each cleaner service run is computed incrementally off the events in the timeline, since the last cleaner run. This is much more efficient than obtaining listings for the full table for each planning (even with a metadata table).<br />`Config Param: CLEANER_INCREMENTAL_MODE_ENABLE` [...]
+| [hoodie.clean.multiple.enabled](#hoodiecleanmultipleenabled) | false | Allows scheduling/executing multiple cleans by enabling this config. If users prefer to strictly ensure clean requests should be mutually exclusive, .i.e. a 2nd clean will not be scheduled if another clean is not yet completed to avoid repeat cleaning of same files, they might want to disable this config.<br />`Config Param: ALLOW_MULTIPLE_CLEANS`<br />`Since Version: 0.11.0`<br />`D [...]
+| [hoodie.clean.parallelism](#hoodiecleanparallelism) | 200 | This config controls the behavior of both the cleaning plan and cleaning execution. Deriving the cleaning plan is parallelized at the table partition level, i.e., each table partition is processed by one Spark task to figure out the files to clean. The cleaner picks the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is a [...]
+| [hoodie.clean.policy](#hoodiecleanpolicy) | KEEP_LATEST_COMMITS | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query execution [...]
+| [hoodie.clean.trigger.max.commits](#hoodiecleantriggermaxcommits) | 1 | Number of commits after the last clean operation, before scheduling of a new clean is attempted.<br />`Config Param: CLEAN_MAX_COMMITS` [...]
+| [hoodie.clean.trigger.strategy](#hoodiecleantriggerstrategy) | NUM_COMMITS | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.trigger.max.commits`.<br />`Config Param: CLEAN_TRIGGER_STRATEGY` [...]
---
@@ -758,6 +781,7 @@ Configurations that control the clustering table service in hudi, which optimize
| [hoodie.clustering.plan.strategy.sort.columns](#hoodieclusteringplanstrategysortcolumns) | (N/A) | Columns to sort the data by when clustering<br />`Config Param: PLAN_STRATEGY_SORT_COLUMNS`<br />`Since Version: 0.7.0` [...]
| [hoodie.clustering.async.max.commits](#hoodieclusteringasyncmaxcommits) | 4 | Config to control frequency of async clustering<br />`Config Param: ASYNC_CLUSTERING_MAX_COMMITS`<br />`Since Version: 0.9.0` [...]
| [hoodie.clustering.execution.strategy.class](#hoodieclusteringexecutionstrategyclass) | org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy | Config to provide a strategy class (subclass of RunClusteringStrategy) to define how the clustering plan is executed. By default, we sort the file groups in th plan by the specified columns, while meeting the configured target file sizes.<br />`Config Param: EXECUTI [...]
+| [hoodie.clustering.group.read.parallelism](#hoodieclusteringgroupreadparallelism) | 20 | Maximum number of parallelism when Spark read records from clustering group.<br />`Config Param: CLUSTERING_GROUP_READ_PARALLELISM`<br />`Since Version: 1.0.0` [...]
| [hoodie.clustering.inline.max.commits](#hoodieclusteringinlinemaxcommits) | 4 | Config to control frequency of clustering planning<br />`Config Param: INLINE_CLUSTERING_MAX_COMMITS`<br />`Since Version: 0.7.0` [...]
| [hoodie.clustering.max.parallelism](#hoodieclusteringmaxparallelism) | 15 | Maximum number of parallelism jobs submitted in clustering operation. If the resource is sufficient(Like Spark engine has enough idle executors), increasing this value will let the clustering job run faster, while it will give additional pressure to the execution engi [...]
| [hoodie.clustering.plan.partition.filter.mode](#hoodieclusteringplanpartitionfiltermode) | NONE | org.apache.hudi.table.action.cluster.ClusteringPlanPartitionFilterMode: Partition filter mode used in the creation of clustering plan. NONE(default): Do not filter partitions. The clustering plan will include all partitions that have clustering candidates. REC [...]
@@ -796,15 +820,14 @@ Configurations that control compaction (merging of log files onto a new base fil
| Config Name | Default | Description [...]
| ----------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.compaction.partition.path.regex](#hoodiecompactionpartitionpathregex) | (N/A) | Used to specify the partition path regex for compaction. Only partitions that match the regex will be compacted. Only be used when configure PartitionRegexBasedCompactionStrategy.<br />`Config Param: COMPACTION_SPECIFY_PARTITION_PATH_REGEX` [...]
| [hoodie.compact.inline.max.delta.seconds](#hoodiecompactinlinemaxdeltaseconds) | 3600 | Number of elapsed seconds after the last compaction, before scheduling a new one. This config takes effect only for the compaction triggering strategy based on the elapsed time, i.e., TIME_ELAPSED, NUM_AND_TIME, and NUM_OR_TIME.<br />`Config Param: INLINE_COMPACT_TIME_DELTA_SECONDS` [...]
| [hoodie.compact.inline.trigger.strategy](#hoodiecompactinlinetriggerstrategy) | NUM_COMMITS | org.apache.hudi.table.action.compact.CompactionTriggerStrategy: Controls when compaction is scheduled. NUM_COMMITS(default): triggers compaction when there are at least N delta commits after last completed compaction. NUM_COMMITS_AFTER_LAST_REQUEST: triggers compaction when there are at least N delta comm [...]
| [hoodie.compact.schedule.inline](#hoodiecompactscheduleinline) | false | When set to true, compaction service will be attempted for inline scheduling after each write. Users have to ensure they have a separate job to run async compaction(execution) for the one scheduled by this writer. Users can choose to set both `hoodie.compact.inline` and `hoodie.compact.schedule.inline` to false a [...]
| [hoodie.compaction.daybased.target.partitions](#hoodiecompactiondaybasedtargetpartitions) | 10 | Used by org.apache.hudi.io.compact.strategy.DayBasedCompactionStrategy to denote the number of latest partitions to compact during a compaction run.<br />`Config Param: TARGET_PARTITIONS_PER_DAYBASED_COMPACTION` [...]
-| [hoodie.compaction.lazy.block.read](#hoodiecompactionlazyblockread) | true | When merging the delta log files, this config helps to choose whether the log blocks should be read lazily or not. Choose true to use lazy block reading (low memory usage, but incurs seeks to each block header) or false for immediate block read (higher memory usage)<br />`Config Param: COMPACTION_LAZY_BLOCK_READ_ [...]
| [hoodie.compaction.logfile.num.threshold](#hoodiecompactionlogfilenumthreshold) | 0 | Only if the log file num is greater than the threshold, the file group will be compacted.<br />`Config Param: COMPACTION_LOG_FILE_NUM_THRESHOLD`<br />`Since Version: 0.13.0` [...]
| [hoodie.compaction.logfile.size.threshold](#hoodiecompactionlogfilesizethreshold) | 0 | Only if the log file size is greater than the threshold in bytes, the file group will be compacted.<br />`Config Param: COMPACTION_LOG_FILE_SIZE_THRESHOLD` [...]
-| [hoodie.compaction.reverse.log.read](#hoodiecompactionreverselogread) | false | HoodieLogFormatReader reads a logfile in the forward direction starting from pos=0 to pos=file_length. If this config is set to true, the reader reads the logfile in reverse direction, from pos=file_length to pos=0<br />`Config Param: COMPACTION_REVERSE_LOG_READ_ENABLE` [...]
-| [hoodie.compaction.strategy](#hoodiecompactionstrategy) | org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy | Compaction strategy decides which file groups are picked up for compaction during each compaction run. By default. Hudi picks the log file with most accumulated unmerged data<br />`Config Param: COMPACTION_STRATEGY` [...]
+| [hoodie.compaction.strategy](#hoodiecompactionstrategy) | org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy | Compaction strategy decides which file groups are picked up for compaction during each compaction run. By default. Hudi picks the log file with most accumulated unmerged data. The strategy can be composed with multiple strategies by concatenating the class names with ','.<br />`Config Param: COMPACTION_STRATEGY` [...]
| [hoodie.compaction.target.io](#hoodiecompactiontargetio) | 512000 | Amount of MBs to spend during compaction run for the LogFileSizeBasedCompactionStrategy. This value helps bound ingestion latency while compaction is run inline mode.<br />`Config Param: TARGET_IO_PER_COMPACTION_IN_MB` [...]
| [hoodie.copyonwrite.insert.auto.split](#hoodiecopyonwriteinsertautosplit) | true | Config to control whether we control insert split sizes automatically based on average record sizes. It's recommended to keep this turned on, since hand tuning is otherwise extremely cumbersome.<br />`Config Param: COPY_ON_WRITE_AUTO_SPLIT_INSERTS` [...]
| [hoodie.copyonwrite.insert.split.size](#hoodiecopyonwriteinsertsplitsize) | 500000 | Number of inserts assigned for each partition/bucket for writing. We based the default on writing out 100MB files, with at least 1kb records (100K records per file), and over provision to 500K. As long as auto-tuning of splits is turned on, this only affects the first write, where there is no history to learn [...]
@@ -812,7 +835,6 @@ Configurations that control compaction (merging of log files onto a new base fil
| [hoodie.log.compaction.blocks.threshold](#hoodielogcompactionblocksthreshold) | 5 | Log compaction can be scheduled if the no. of log blocks crosses this threshold value. This is effective only when log compaction is enabled via hoodie.log.compaction.inline<br />`Config Param: LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` [...]
| [hoodie.log.compaction.enable](#hoodielogcompactionenable) | false | By enabling log compaction through this config, log compaction will also get enabled for the metadata table.<br />`Config Param: ENABLE_LOG_COMPACTION`<br />`Since Version: 0.14.0` [...]
| [hoodie.log.compaction.inline](#hoodielogcompactioninline) | false | When set to true, logcompaction service is triggered after each write. While being simpler operationally, this adds extra latency on the write path.<br />`Config Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` [...]
-| [hoodie.optimized.log.blocks.scan.enable](#hoodieoptimizedlogblocksscanenable) | false | New optimized scan for log blocks that handles all multi-writer use-cases while appending to log files. It also differentiates original blocks written by ingestion writers and compacted blocks written log compaction.<br />`Config Param: ENABLE_OPTIMIZED_LOG_BLOCKS_SCAN`<br />`Since Version: 0.13.0` [...]
| [hoodie.parquet.small.file.limit](#hoodieparquetsmallfilelimit) | 104857600 | During upsert operation, we opportunistically expand existing small files on storage, instead of writing new files, to keep number of files to an optimum. This config sets the file size limit below which a file on storage becomes a candidate to be selected as such a `small file`. By default, treat any file <= [...]
| [hoodie.record.size.estimation.threshold](#hoodierecordsizeestimationthreshold) | 1.0 | We use the previous commits' metadata to calculate the estimated record size and use it to bin pack records into partitions. If the previous commit is too small to make an accurate estimation, Hudi will search commits in the reverse order, until we find a commit that has totalBytesWritten larger than (PARQUET_ [...]
---
@@ -855,26 +877,6 @@ Configurations that control storage layout and data distribution, which defines
---
-### Memory Configurations {#Memory-Configurations}
-Controls memory usage for compaction and merges, performed internally by Hudi.
-
-
-
-[**Advanced Configs**](#Memory-Configurations-advanced-configs)
-
-
-| Config Name | Default | Description [...]
-| ------------------------------------------------------------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.memory.compaction.max.size](#hoodiememorycompactionmaxsize) | (N/A) | Maximum amount of memory used in bytes for compaction operations in bytes , before spilling to local storage.<br />`Config Param: MAX_MEMORY_FOR_COMPACTION` [...]
-| [hoodie.memory.spillable.map.path](#hoodiememoryspillablemappath) | (N/A) | Default file path for spillable map<br />`Config Param: SPILLABLE_MAP_BASE_PATH` [...]
-| [hoodie.memory.compaction.fraction](#hoodiememorycompactionfraction) | 0.6 | HoodieCompactedLogScanner reads logblocks, converts records to HoodieRecords and then merges these log blocks and records. At any point, the number of entries in a log block can be less than or equal to the number of entries in the corresponding parquet file. This can lead to OOM in the Scanner. Hence, a spillable map helps alleviate the memory pressure. Use this config to set the max [...]
-| [hoodie.memory.dfs.buffer.max.size](#hoodiememorydfsbuffermaxsize) | 16777216 | Property to control the max memory in bytes for dfs input stream buffer size<br />`Config Param: MAX_DFS_STREAM_BUFFER_SIZE` [...]
-| [hoodie.memory.merge.fraction](#hoodiememorymergefraction) | 0.6 | This fraction is multiplied with the user memory fraction (1 - spark.memory.fraction) to get a final fraction of heap space to use during merge<br />`Config Param: MAX_MEMORY_FRACTION_FOR_MERGE` [...]
-| [hoodie.memory.merge.max.size](#hoodiememorymergemaxsize) | 1073741824 | Maximum amount of memory used in bytes for merge operations, before spilling to local storage.<br />`Config Param: MAX_MEMORY_FOR_MERGE` [...]
-| [hoodie.memory.writestatus.failure.fraction](#hoodiememorywritestatusfailurefraction) | 0.1 | Property to control how what fraction of the failed record, exceptions we report back to driver. Default is 10%. If set to 100%, with lot of failures, this can cause memory pressure, cause OOMs and mask actual data errors.<br />`Config Param: WRITESTATUS_FAILURE_FRACTION` [...]
----
-
-
### TTL management Configs {#TTL-management-Configs}
Data ttl management
@@ -902,93 +904,100 @@ Configurations that control write behavior on Hudi tables. These can be directly
[**Basic Configs**](#Write-Configurations-basic-configs)
-| Config Name | Default | Description [...]
-| --------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` [...]
-| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br />`Config Param: TBL_NAME` [...]
-| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD_NAME` [...]
-| [hoodie.write.concurrency.mode](#hoodiewriteconcurrencymode) | SINGLE_WRITER | org.apache.hudi.common.model.WriteConcurrencyMode: Concurrency modes for write operations. SINGLE_WRITER(default): Only one active writer to the table. Maximizes throughput. OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table with lazy conflict resolution using locks. This means that only one writer succeeds if multiple writers write to the same file group. [...]
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.base.path](#hoodiebasepath) | (N/A) | Base path on lake storage, under which all the table data is stored. Always prefix it explicitly with the storage scheme (e.g hdfs://, s3:// etc). Hudi stores all the main meta-data about commits, savepoints, cleaning audit logs etc in .hoodie directory under this base path directory.<br />`Config Param: BASE_PATH` [...]
+| [hoodie.table.name](#hoodietablename) | (N/A) | Table name that will be used for registering with metastores like HMS. Needs to be same across runs.<br />`Config Param: TBL_NAME` [...]
+| [hoodie.datasource.write.precombine.field](#hoodiedatasourcewriteprecombinefield) | ts | Field used in preCombining before actual write. When two records have the same key value, we will pick the one with the largest value for the precombine field, determined by Object.compareTo(..)<br />`Config Param: PRECOMBINE_FIELD_NAME` [...]
+| [hoodie.instant_state.timeline_server_based.enabled](#hoodieinstant_statetimeline_server_basedenabled) | false | If enabled, writers get instant state from timeline server rather than requesting DFS directly<br />`Config Param: INSTANT_STATE_TIMELINE_SERVER_BASED`<br />`Since Version: 1.0.0` [...]
+| [hoodie.instant_state.timeline_server_based.force_refresh.request.number](#hoodieinstant_statetimeline_server_basedforce_refreshrequestnumber) | 100 | Number of requests to trigger instant state cache refreshing<br />`Config Param: INSTANT_STATE_TIMELINE_SERVER_BASED_FORCE_REFRESH_REQUEST_NUMBER`<br />`Since Version: 1.0.0` [...]
+| [hoodie.write.auto.upgrade](#hoodiewriteautoupgrade) | true | If enabled, writers automatically migrate the table to the specified write table version if the current table version is lower.<br />`Config Param: AUTO_UPGRADE_VERSION`<br />`Since Version: 1.0.0` [...]
+| [hoodie.write.concurrency.mode](#hoodiewriteconcurrencymode) | SINGLE_WRITER | org.apache.hudi.common.model.WriteConcurrencyMode: Concurrency modes for write operations. SINGLE_WRITER(default): Only one active writer to the table. Maximizes throughput. OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table with lazy conflict resolution using locks. This means that only on [...]
+| [hoodie.write.record.merge.mode](#hoodiewriterecordmergemode) | EVENT_TIME_ORDERING | org.apache.hudi.common.config.RecordMergeMode: Determines the logic of merging updates COMMIT_TIME_ORDERING: Using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key. EVENT_TIME_ORDERING(default): Using event time as the ordering to merge rec [...]
+| [hoodie.write.table.version](#hoodiewritetableversion) | 8 | The table version this writer is storing the table in. This should match the current table version.<br />`Config Param: WRITE_TABLE_VERSION`<br />`Since Version: 1.0.0` [...]
[**Advanced Configs**](#Write-Configurations-advanced-configs)
-| Config Name | Default | Description [...]
-| --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.avro.schema](#hoodieavroschema) | (N/A) | Schema string representing the current write schema of the table. Hudi passes this to implementations of HoodieRecordPayload to convert incoming records to avro. This is also used as the write schema evolving records during an update.<br />`Config Param: AVRO_SCHEMA_STRING` [...]
-| [hoodie.bulkinsert.user.defined.partitioner.class](#hoodiebulkinsertuserdefinedpartitionerclass) | (N/A) | If specified, this class will be used to re-partition records before they are bulk inserted. This can be used to sort, pack, cluster data optimally for common query patterns. For now we support a build-in user defined bulkinsert partitioner org.apache.hudi.execution.bulkinsert.RDDCustomColumnsS [...]
-| [hoodie.bulkinsert.user.defined.partitioner.sort.columns](#hoodiebulkinsertuserdefinedpartitionersortcolumns) | (N/A) | Columns to sort the data by when use org.apache.hudi.execution.bulkinsert.RDDCustomColumnsSortPartitioner as user defined partitioner during bulk_insert. For example 'column1,column2'<br />`Config Param: BULKINSERT_USER_DEFINED_PARTITIONER_SORT_COLUMNS` [...]
-| [hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass) | (N/A) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records.<br />`Config Param: KEYGENERATOR_CLASS_NAME` [...]
-| [hoodie.internal.schema](#hoodieinternalschema) | (N/A) | Schema string representing the latest schema of the table. Hudi passes this to implementations of evolution of schema<br />`Config Param: INTERNAL_SCHEMA_STRING` [...]
-| [hoodie.write.schema](#hoodiewriteschema) | (N/A) | Config allowing to override writer's schema. This might be necessary in cases when writer's schema derived from the incoming dataset might actually be different from the schema we actually want to use when writing. This, for ex, could be the case for'partial-update' use-cases (like `MERGE INTO` [...]
-| [_.hoodie.allow.multi.write.on.same.instant](#_hoodieallowmultiwriteonsameinstant) | false | <br />`Config Param: ALLOW_MULTI_WRITE_ON_SAME_INSTANT_ENABLE` [...]
-| [hoodie.allow.empty.commit](#hoodieallowemptycommit) | true | Whether to allow generation of empty commits, even if no data was written in the commit. It's useful in cases where extra metadata needs to be published regardless e.g tracking source offsets when ingesting data<br />`Config Param: ALLOW_EMPTY_COMMIT` [...]
-| [hoodie.allow.operation.metadata.field](#hoodieallowoperationmetadatafield) | false | Whether to include '_hoodie_operation' in the metadata fields. Once enabled, all the changes of a record are persisted to the delta log directly without merge<br />`Config Param: ALLOW_OPERATION_METADATA_FIELD`<br />`Since Version: 0.9.0` [...]
-| [hoodie.auto.adjust.lock.configs](#hoodieautoadjustlockconfigs) | false | Auto adjust lock configurations when metadata table is enabled and for async table services.<br />`Config Param: AUTO_ADJUST_LOCK_CONFIGS`<br />`Since Version: 0.11.0` [...]
-| [hoodie.auto.commit](#hoodieautocommit) | true | Controls whether a write operation should auto commit. This can be turned off to perform inspection of the uncommitted write before deciding to commit.<br />`Config Param: AUTO_COMMIT_ENABLE` [...]
-| [hoodie.avro.schema.external.transformation](#hoodieavroschemaexternaltransformation) | false | When enabled, records in older schema are rewritten into newer schema during upsert,delete and background compaction,clustering operations.<br />`Config Param: AVRO_EXTERNAL_SCHEMA_TRANSFORMATION_ENABLE` [...]
-| [hoodie.avro.schema.validate](#hoodieavroschemavalidate) | false | Validate the schema used for the write against the latest schema, for backwards compatibility.<br />`Config Param: AVRO_SCHEMA_VALIDATE_ENABLE` [...]
-| [hoodie.bulkinsert.shuffle.parallelism](#hoodiebulkinsertshuffleparallelism) | 0 | For large initial imports using bulk_insert operation, controls the parallelism to use for sort modes or custom partitioning done before writing records to the table. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards [...]
-| [hoodie.bulkinsert.sort.mode](#hoodiebulkinsertsortmode) | NONE | org.apache.hudi.execution.bulkinsert.BulkInsertSortMode: Modes for sorting records during bulk insert. NONE(default): No sorting. Fastest and matches `spark.write.parquet()` in number of files and overhead. GLOBAL_SORT: This ensures best file sizes, with lowest memory overhead at cost o [...]
-| [hoodie.client.heartbeat.interval_in_ms](#hoodieclientheartbeatinterval_in_ms) | 60000 | Writers perform heartbeats to indicate liveness. Controls how often (in ms), such heartbeats are registered to lake storage.<br />`Config Param: CLIENT_HEARTBEAT_INTERVAL_IN_MS` [...]
-| [hoodie.client.heartbeat.tolerable.misses](#hoodieclientheartbeattolerablemisses) | 2 | Number of heartbeat misses, before a writer is deemed not alive and all pending writes are aborted.<br />`Config Param: CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES` [...]
-| [hoodie.client.init.callback.classes](#hoodieclientinitcallbackclasses) | | Fully-qualified class names of the Hudi client init callbacks to run at the initialization of the Hudi client. The class names are separated by `,`. The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`.By default, no Hudi client init callback is executed.<br />`C [...]
-| [hoodie.combine.before.delete](#hoodiecombinebeforedelete) | true | During delete operations, controls whether we should combine deletes (and potentially also upserts) before writing to storage.<br />`Config Param: COMBINE_BEFORE_DELETE` [...]
-| [hoodie.combine.before.insert](#hoodiecombinebeforeinsert) | false | When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage.<br />`Config Param: COMBINE_BEFORE_INSERT` [...]
-| [hoodie.combine.before.upsert](#hoodiecombinebeforeupsert) | true | When upserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage. This should be turned off only if you are absolutely certain that there are no duplicates incoming, otherwise it can lead to duplicate keys and violate the unique [...]
-| [hoodie.consistency.check.initial_interval_ms](#hoodieconsistencycheckinitial_interval_ms) | 2000 | Initial time between successive attempts to ensure written data's metadata is consistent on storage. Grows with exponential backoff after the initial value.<br />`Config Param: INITIAL_CONSISTENCY_CHECK_INTERVAL_MS` [...]
-| [hoodie.consistency.check.max_checks](#hoodieconsistencycheckmax_checks) | 7 | Maximum number of checks, for consistency of written data.<br />`Config Param: MAX_CONSISTENCY_CHECKS` [...]
-| [hoodie.consistency.check.max_interval_ms](#hoodieconsistencycheckmax_interval_ms) | 300000 | Max time to wait between successive attempts at performing consistency checks<br />`Config Param: MAX_CONSISTENCY_CHECK_INTERVAL_MS` [...]
-| [hoodie.datasource.write.keygenerator.type](#hoodiedatasourcewritekeygeneratortype) | SIMPLE | **Note** This is being actively worked on. Please use `hoodie.datasource.write.keygenerator.class` instead. org.apache.hudi.keygen.constant.KeyGeneratorType: Key generator type, indicating the key generator class to use, that implements `org.apache.hudi.keygen.KeyGenerator`. SIMPLE(default) [...]
-| [hoodie.datasource.write.payload.class](#hoodiedatasourcewritepayloadclass) | org.apache.hudi.common.model.DefaultHoodieRecordPayload | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br />`Config Param: WRITE_PAYLOAD_CLASS_NAME` [...]
-| [hoodie.datasource.write.payload.type](#hoodiedatasourcewritepayloadtype) | HOODIE_AVRO_DEFAULT | org.apache.hudi.common.model.RecordPayloadType: Payload to use for merging records AWS_DMS_AVRO: Provides support for seamlessly applying changes captured via Amazon Database Migration Service onto S3. HOODIE_AVRO: A payload to wrap a existing Hoodie Avro Record. Useful to create a HoodieRe [...]
-| [hoodie.datasource.write.record.merger.impls](#hoodiedatasourcewriterecordmergerimpls) | org.apache.hudi.common.model.HoodieAvroRecordMerger | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These merger impls will filter by hoodie.datasource.write.record.merger.strategy Hudi will pick most efficient implementation to perform merging/combining of the records (during update, readin [...]
-| [hoodie.datasource.write.record.merger.strategy](#hoodiedatasourcewriterecordmergerstrategy) | eeb8d96f-b1e4-49fd-bbf8-28ac514178e5 | Id of merger strategy. Hudi will pick HoodieRecordMerger implementations in hoodie.datasource.write.record.merger.impls which has the same merger strategy id<br />`Config Param: RECORD_MERGER_STRATEGY`<br />`Since Version: 0.13.0` [...]
-| [hoodie.datasource.write.schema.allow.auto.evolution.column.drop](#hoodiedatasourcewriteschemaallowautoevolutioncolumndrop) | false | Controls whether table's schema is allowed to automatically evolve when incoming batch's schema can have any of the columns dropped. By default, Hudi will not allow this kind of (auto) schema evolution. Set this config to true to allow table's schema to be updated automatically when columns are [...]
-| [hoodie.delete.shuffle.parallelism](#hoodiedeleteshuffleparallelism) | 0 | Parallelism used for delete operation. Delete operations also performs shuffles, similar to upsert operation. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards Hudi by default automatically uses the parallelism deduce [...]
-| [hoodie.embed.timeline.server](#hoodieembedtimelineserver) | true | When true, spins up an instance of the timeline server (meta server that serves cached file listings, statistics),running on each writer's driver process, accepting requests during the write from executors.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_ENABLE` [...]
-| [hoodie.embed.timeline.server.async](#hoodieembedtimelineserverasync) | false | Controls whether or not, the requests to the timeline server are processed in asynchronous fashion, potentially improving throughput.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_USE_ASYNC_ENABLE` [...]
-| [hoodie.embed.timeline.server.gzip](#hoodieembedtimelineservergzip) | true | Controls whether gzip compression is used, for large responses from the timeline server, to improve latency.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_COMPRESS_ENABLE` [...]
-| [hoodie.embed.timeline.server.port](#hoodieembedtimelineserverport) | 0 | Port at which the timeline server listens for requests. When running embedded in each writer, it picks a free port and communicates to all the executors. This should rarely be changed.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_PORT_NUM` [...]
-| [hoodie.embed.timeline.server.reuse.enabled](#hoodieembedtimelineserverreuseenabled) | false | Controls whether the timeline server instance should be cached and reused across the tablesto avoid startup costs and server overhead. This should only be used if you are running multiple writers in the same JVM.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_REUSE_ENABLED` [...]
-| [hoodie.embed.timeline.server.threads](#hoodieembedtimelineserverthreads) | -1 | Number of threads to serve requests in the timeline server. By default, auto configured based on the number of underlying cores.<br />`Config Param: EMBEDDED_TIMELINE_NUM_SERVER_THREADS` [...]
-| [hoodie.fail.on.timeline.archiving](#hoodiefailontimelinearchiving) | true | Timeline archiving removes older instants from the timeline, after each write operation, to minimize metadata overhead. Controls whether or not, the write should be failed as well, if such archiving fails.<br />`Config Param: FAIL_ON_TIMELINE_ARCHIVING_ENABLE` [...]
-| [hoodie.fail.writes.on.inline.table.service.exception](#hoodiefailwritesoninlinetableserviceexception) | true | Table services such as compaction and clustering can fail and prevent syncing to the metaclient. Set this to true to fail writes when table services fail<br />`Config Param: FAIL_ON_INLINE_TABLE_SERVICE_EXCEPTION`<br />`Since Version: 0.13.0` [...]
-| [hoodie.fileid.prefix.provider.class](#hoodiefileidprefixproviderclass) | org.apache.hudi.table.RandomFileIdPrefixProvider | File Id Prefix provider class, that implements `org.apache.hudi.fileid.FileIdPrefixProvider`<br />`Config Param: FILEID_PREFIX_PROVIDER_CLASS`<br />`Since Version: 0.10.0` [...]
-| [hoodie.finalize.write.parallelism](#hoodiefinalizewriteparallelism) | 200 | Parallelism for the write finalization internal operation, which involves removing any partially written files from lake storage, before committing the write. Reduce this value, if the high number of tasks incur delays for smaller tables or low latency writes.<br />`Config Param: FINALIZE_WRITE [...]
-| [hoodie.insert.shuffle.parallelism](#hoodieinsertshuffleparallelism) | 0 | Parallelism for inserting records into the table. Inserts can shuffle data before writing to tune file sizes and optimize the storage layout. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards Hudi by default automatic [...]
-| [hoodie.markers.delete.parallelism](#hoodiemarkersdeleteparallelism) | 100 | Determines the parallelism for deleting marker files, which are used to track all files (valid or invalid/partial) written during a write operation. Increase this value if delays are observed, with large batch writes.<br />`Config Param: MARKERS_DELETE_PARALLELISM_VALUE` [...]
-| [hoodie.markers.timeline_server_based.batch.interval_ms](#hoodiemarkerstimeline_server_basedbatchinterval_ms) | 50 | The batch interval in milliseconds for marker creation batch processing<br />`Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_INTERVAL_MS`<br />`Since Version: 0.9.0` [...]
-| [hoodie.markers.timeline_server_based.batch.num_threads](#hoodiemarkerstimeline_server_basedbatchnum_threads) | 20 | Number of threads to use for batch processing marker creation requests at the timeline server<br />`Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_NUM_THREADS`<br />`Since Version: 0.9.0` [...]
-| [hoodie.merge.allow.duplicate.on.inserts](#hoodiemergeallowduplicateoninserts) | true | When enabled, we allow duplicate keys even if inserts are routed to merge with an existing file (for ensuring file sizing). This is only relevant for insert operation, since upsert, delete operations will ensure unique key constraints are maintained.<br />`Config Param: MERGE_ALLOW_DUPLICATE_ON [...]
-| [hoodie.merge.data.validation.enabled](#hoodiemergedatavalidationenabled) | false | When enabled, data validation checks are performed during merges to ensure expected number of records after merge operation.<br />`Config Param: MERGE_DATA_VALIDATION_CHECK_ENABLE` [...]
-| [hoodie.merge.small.file.group.candidates.limit](#hoodiemergesmallfilegroupcandidateslimit) | 1 | Limits number of file groups, whose base file satisfies small-file limit, to consider for appending records during upsert operation. Only applicable to MOR tables<br />`Config Param: MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT` [...]
-| [hoodie.release.resource.on.completion.enable](#hoodiereleaseresourceoncompletionenable) | true | Control to enable release all persist rdds when the spark job finish.<br />`Config Param: RELEASE_RESOURCE_ENABLE`<br />`Since Version: 0.11.0` [...]
-| [hoodie.rollback.instant.backup.dir](#hoodierollbackinstantbackupdir) | .rollback_backup | Path where instants being rolled back are copied. If not absolute path then a directory relative to .hoodie folder is created.<br />`Config Param: ROLLBACK_INSTANT_BACKUP_DIRECTORY` [...]
-| [hoodie.rollback.instant.backup.enabled](#hoodierollbackinstantbackupenabled) | false | Backup instants removed during rollback and restore (useful for debugging)<br />`Config Param: ROLLBACK_INSTANT_BACKUP_ENABLED` [...]
-| [hoodie.rollback.parallelism](#hoodierollbackparallelism) | 100 | This config controls the parallelism for rollback of commits. Rollbacks perform deletion of files or logging delete blocks to file groups on storage in parallel. The configure value limits the parallelism so that the number of Spark tasks do not exceed the value. If rollback is slow due to the [...]
-| [hoodie.rollback.using.markers](#hoodierollbackusingmarkers) | true | Enables a more efficient mechanism for rollbacks based on the marker files generated during the writes. Turned on by default.<br />`Config Param: ROLLBACK_USING_MARKERS_ENABLE` [...]
-| [hoodie.sensitive.config.keys](#hoodiesensitiveconfigkeys) | ssl,tls,sasl,auth,credentials | Comma separated list of filters for sensitive config keys. Hudi Streamer will not print any configuration which contains the configured filter. For example with a configured filter `ssl`, value for config `ssl.trustore.location` would be masked.<br />`Config Param: SENSITIVE_CONFIG_KEYS_FILTER` [...]
-| [hoodie.skip.default.partition.validation](#hoodieskipdefaultpartitionvalidation) | false | When table is upgraded from pre 0.12 to 0.12, we check for "default" partition and fail if found one. Users are expected to rewrite the data in those partitions. Enabling this config will bypass this validation<br />`Config Param: SKIP_DEFAULT_PARTITION_VALIDATION`<br />`Since Version: 0.12.0` [...]
-| [hoodie.table.base.file.format](#hoodietablebasefileformat) | PARQUET | File format to store all the base file data. org.apache.hudi.common.model.HoodieFileFormat: Hoodie file formats. PARQUET(default): Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data compression and [...]
-| [hoodie.table.services.enabled](#hoodietableservicesenabled) | true | Master control to disable all table services including archive, clean, compact, cluster, etc.<br />`Config Param: TABLE_SERVICES_ENABLED`<br />`Since Version: 0.11.0` [...]
-| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | 1 | Controls the layout of the timeline. Version 0 relied on renames, Version 1 (default) models the timeline as an immutable log relying only on atomic writes for object storage.<br />`Config Param: TIMELINE_LAYOUT_VERSION_NUM`<br />`Since Version: 0.5.1` [...]
-| [hoodie.upsert.shuffle.parallelism](#hoodieupsertshuffleparallelism) | 0 | Parallelism to use for upsert operation on the table. Upserts can shuffle data to perform index lookups, file sizing, bin packing records optimally into file groups. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards H [...]
-| [hoodie.write.buffer.limit.bytes](#hoodiewritebufferlimitbytes) | 4194304 | Size of in-memory buffer used for parallelizing network reads and lake storage writes.<br />`Config Param: WRITE_BUFFER_LIMIT_BYTES_VALUE` [...]
-| [hoodie.write.buffer.record.cache.limit](#hoodiewritebufferrecordcachelimit) | 131072 | Maximum queue size of in-memory buffer for parallelizing network reads and lake storage writes.<br />`Config Param: WRITE_BUFFER_RECORD_CACHE_LIMIT`<br />`Since Version: 0.15.0` [...]
-| [hoodie.write.buffer.record.sampling.rate](#hoodiewritebufferrecordsamplingrate) | 64 | Sampling rate of in-memory buffer used to estimate object size. Higher value lead to lower CPU usage.<br />`Config Param: WRITE_BUFFER_RECORD_SAMPLING_RATE`<br />`Since Version: 0.15.0` [...]
-| [hoodie.write.concurrency.async.conflict.detector.initial_delay_ms](#hoodiewriteconcurrencyasyncconflictdetectorinitial_delay_ms) | 0 | Used for timeline-server-based markers with `AsyncTimelineServerBasedDetectionStrategy`. The time in milliseconds to delay the first execution of async marker-based conflict detection.<br />`Config Param: ASYNC_CONFLICT_DETECTOR_INITIAL_DELAY_MS`<br />`Since Version: 0.13.0` [...]
-| [hoodie.write.concurrency.async.conflict.detector.period_ms](#hoodiewriteconcurrencyasyncconflictdetectorperiod_ms) | 30000 | Used for timeline-server-based markers with `AsyncTimelineServerBasedDetectionStrategy`. The period in milliseconds between successive executions of async marker-based conflict detection.<br />`Config Param: ASYNC_CONFLICT_DETECTOR_PERIOD_MS`<br />`Since Version: 0.13.0` [...]
-| [hoodie.write.concurrency.early.conflict.check.commit.conflict](#hoodiewriteconcurrencyearlyconflictcheckcommitconflict) | false | Whether to enable commit conflict checking or not during early conflict detection.<br />`Config Param: EARLY_CONFLICT_DETECTION_CHECK_COMMIT_CONFLICT`<br />`Since Version: 0.13.0` [...]
-| [hoodie.write.concurrency.early.conflict.detection.enable](#hoodiewriteconcurrencyearlyconflictdetectionenable) | false | Whether to enable early conflict detection based on markers. It eagerly detects writing conflict before create markers and fails fast if a conflict is detected, to release cluster compute resources as soon as possible.<br />`Config Param: EARLY_CONFLICT_DETECTION_ENABLE`<br />`Since Version: 0. [...]
-| [hoodie.write.concurrency.early.conflict.detection.strategy](#hoodiewriteconcurrencyearlyconflictdetectionstrategy) | | The class name of the early conflict detection strategy to use. This should be a subclass of `org.apache.hudi.common.conflict.detection.EarlyConflictDetectionStrategy`.<br />`Config Param: EARLY_CONFLICT_DETECTION_STRATEGY_CLASS_NAME`<br />`Since Version: 0.13.0` [...]
-| [hoodie.write.executor.disruptor.buffer.limit.bytes](#hoodiewriteexecutordisruptorbufferlimitbytes) | 1024 | The size of the Disruptor Executor ring buffer, must be power of 2<br />`Config Param: WRITE_EXECUTOR_DISRUPTOR_BUFFER_LIMIT_BYTES`<br />`Since Version: 0.13.0` [...]
-| [hoodie.write.executor.disruptor.wait.strategy](#hoodiewriteexecutordisruptorwaitstrategy) | BLOCKING_WAIT | org.apache.hudi.common.util.queue.DisruptorWaitStrategyType: Strategy employed for making Disruptor Executor wait on a cursor. BLOCKING_WAIT(default): The slowest of the available wait strategies. However, it is the most conservative with the respect to CPU usage and will give the most cons [...]
-| [hoodie.write.executor.type](#hoodiewriteexecutortype) | SIMPLE | org.apache.hudi.common.util.queue.ExecutorType: Types of executor that implements org.apache.hudi.common.util.queue.HoodieExecutor. The executor orchestrates concurrent producers and consumers communicating through a message queue. BOUNDED_IN_MEMORY: Executor which orchestrates concurrent p [...]
-| [hoodie.write.markers.type](#hoodiewritemarkerstype) | TIMELINE_SERVER_BASED | org.apache.hudi.common.table.marker.MarkerType: Marker type indicating how markers are stored in the file system, used for identifying the files written and cleaning up files not committed which should be deleted. DIRECT: Individual marker file corresponding to each data file is directly cr [...]
-| [hoodie.write.num.retries.on.conflict.failures](#hoodiewritenumretriesonconflictfailures) | 0 | Maximum number of times to retry a batch on conflict failure.<br />`Config Param: NUM_RETRIES_ON_CONFLICT_FAILURES`<br />`Since Version: 0.14.0` [...]
-| [hoodie.write.status.storage.level](#hoodiewritestatusstoragelevel) | MEMORY_AND_DISK_SER | Write status objects hold metadata about a write (stats, errors), that is not yet committed to storage. This controls the how that information is cached for inspection by clients. We rarely expect this to be changed.<br />`Config Param: WRITE_STATUS_STORAGE_LEVEL_VALUE` [...]
-| [hoodie.write.tagged.record.storage.level](#hoodiewritetaggedrecordstoragelevel) | MEMORY_AND_DISK_SER | Determine what level of persistence is used to cache write RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: TAGGED_RECORD_STORAGE_LEVEL_VALUE` [...]
-| [hoodie.writestatus.class](#hoodiewritestatusclass) | org.apache.hudi.client.WriteStatus | Subclass of org.apache.hudi.client.WriteStatus to be used to collect information about a write. Can be overridden to collection additional metrics/statistics about the data if needed.<br />`Config Param: WRITE_STATUS_CLASS_NAME` [...]
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
+| [hoodie.avro.schema](#hoodieavroschema) | (N/A) | Schema string representing the current write schema of the table. Hudi passes this to implementations of HoodieRecordPayload to convert incoming records to avro. This is also used as the write schema evolving records during an update.<br />`Config Param: AVRO_SCHEMA_STRING` [...]
+| [hoodie.bulkinsert.user.defined.partitioner.class](#hoodiebulkinsertuserdefinedpartitionerclass) | (N/A) | If specified, this class will be used to re-partition records before they are bulk inserted. This can be used to sort, pack, cluster data optimally for common query patterns. For now we support a build-in user defined bulkinsert partitioner org.apache.hudi.execution.bulkinsert.RDDCustomColumnsSortPartitio [...]
+| [hoodie.bulkinsert.user.defined.partitioner.sort.columns](#hoodiebulkinsertuserdefinedpartitionersortcolumns) | (N/A) | Columns to sort the data by when use org.apache.hudi.execution.bulkinsert.RDDCustomColumnsSortPartitioner as user defined partitioner during bulk_insert. For example 'column1,column2'<br />`Config Param: BULKINSERT_USER_DEFINED_PARTITIONER_SORT_COLUMNS` [...]
+| [hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass) | (N/A) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records.<br />`Config Param: KEYGENERATOR_CLASS_NAME` [...]
+| [hoodie.datasource.write.payload.class](#hoodiedatasourcewritepayloadclass) | (N/A) | Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective<br />`Config Param: WRITE_PAYLOAD_CLASS_NAME` [...]
+| [hoodie.internal.schema](#hoodieinternalschema) | (N/A) | Schema string representing the latest schema of the table. Hudi passes this to implementations of evolution of schema<br />`Config Param: INTERNAL_SCHEMA_STRING` [...]
+| [hoodie.write.record.merge.custom.implementation.classes](#hoodiewriterecordmergecustomimplementationclasses) | (N/A) | List of HoodieMerger implementations constituting Hudi's merging strategy -- based on the engine used. These record merge impls will filter by hoodie.write.record.merge.strategy.idHudi will pick most efficient implementation to perform merging/combining of the records (during update, reading MOR table, et [...]
+| [hoodie.write.record.merge.strategy.id](#hoodiewriterecordmergestrategyid) | (N/A) | ID of record merge strategy. Hudi will pick HoodieRecordMerger implementations in `hoodie.write.record.merge.custom.implementation.classes` which has the same merge strategy id<br />`Config Param: RECORD_MERGE_STRATEGY_ID`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.schema](#hoodiewriteschema) | (N/A) | Config allowing to override writer's schema. This might be necessary in cases when writer's schema derived from the incoming dataset might actually be different from the schema we actually want to use when writing. This, for ex, could be the case for'partial-update' use-cases (like `MERGE INTO` Spark SQL [...]
+| [_.hoodie.allow.multi.write.on.same.instant](#_hoodieallowmultiwriteonsameinstant) | false | <br />`Config Param: ALLOW_MULTI_WRITE_ON_SAME_INSTANT_ENABLE` [...]
+| [hoodie.allow.empty.commit](#hoodieallowemptycommit) | true | Whether to allow generation of empty commits, even if no data was written in the commit. It's useful in cases where extra metadata needs to be published regardless e.g tracking source offsets when ingesting data<br />`Config Param: ALLOW_EMPTY_COMMIT` [...]
+| [hoodie.allow.operation.metadata.field](#hoodieallowoperationmetadatafield) | false | Whether to include '_hoodie_operation' in the metadata fields. Once enabled, all the changes of a record are persisted to the delta log directly without merge<br />`Config Param: ALLOW_OPERATION_METADATA_FIELD`<br />`Since Version: 0.9.0` [...]
+| [hoodie.auto.adjust.lock.configs](#hoodieautoadjustlockconfigs) | false | Auto adjust lock configurations when metadata table is enabled and for async table services.<br />`Config Param: AUTO_ADJUST_LOCK_CONFIGS`<br />`Since Version: 0.11.0` [...]
+| [hoodie.auto.commit](#hoodieautocommit) | true | Controls whether a write operation should auto commit. This can be turned off to perform inspection of the uncommitted write before deciding to commit.<br />`Config Param: AUTO_COMMIT_ENABLE` [...]
+| [hoodie.avro.schema.external.transformation](#hoodieavroschemaexternaltransformation) | false | When enabled, records in older schema are rewritten into newer schema during upsert,delete and background compaction,clustering operations.<br />`Config Param: AVRO_EXTERNAL_SCHEMA_TRANSFORMATION_ENABLE` [...]
+| [hoodie.avro.schema.validate](#hoodieavroschemavalidate) | false | Validate the schema used for the write against the latest schema, for backwards compatibility.<br />`Config Param: AVRO_SCHEMA_VALIDATE_ENABLE` [...]
+| [hoodie.base.file.format](#hoodiebasefileformat) | PARQUET | File format to store all the base file data. org.apache.hudi.common.model.HoodieFileFormat: Hoodie file formats. PARQUET(default): Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data compression and encoding s [...]
+| [hoodie.bulkinsert.shuffle.parallelism](#hoodiebulkinsertshuffleparallelism) | 0 | For large initial imports using bulk_insert operation, controls the parallelism to use for sort modes or custom partitioning done before writing records to the table. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards Hudi by def [...]
+| [hoodie.bulkinsert.sort.mode](#hoodiebulkinsertsortmode) | NONE | org.apache.hudi.execution.bulkinsert.BulkInsertSortMode: Modes for sorting records during bulk insert. NONE(default): No sorting. Fastest and matches `spark.write.parquet()` in number of files and overhead. GLOBAL_SORT: This ensures best file sizes, with lowest memory overhead at cost of sorting. [...]
+| [hoodie.bulkinsert.sort.suffix.record_key](#hoodiebulkinsertsortsuffixrecord_key) | false | When using user defined sort columns there can be possibility of skew because spark's RangePartitioner used in sort can reduce the number of outputSparkPartitionsif the sampled dataset has a low cardinality on the provided sort columns. This can cause an increase in commit durations as we are not leveragi [...]
+| [hoodie.client.heartbeat.interval_in_ms](#hoodieclientheartbeatinterval_in_ms) | 60000 | Writers perform heartbeats to indicate liveness. Controls how often (in ms), such heartbeats are registered to lake storage.<br />`Config Param: CLIENT_HEARTBEAT_INTERVAL_IN_MS` [...]
+| [hoodie.client.heartbeat.tolerable.misses](#hoodieclientheartbeattolerablemisses) | 2 | Number of heartbeat misses, before a writer is deemed not alive and all pending writes are aborted.<br />`Config Param: CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES` [...]
+| [hoodie.client.init.callback.classes](#hoodieclientinitcallbackclasses) | | Fully-qualified class names of the Hudi client init callbacks to run at the initialization of the Hudi client. The class names are separated by `,`. The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`.By default, no Hudi client init callback is executed.<br />`Config Param [...]
+| [hoodie.combine.before.delete](#hoodiecombinebeforedelete) | true | During delete operations, controls whether we should combine deletes (and potentially also upserts) before writing to storage.<br />`Config Param: COMBINE_BEFORE_DELETE` [...]
+| [hoodie.combine.before.insert](#hoodiecombinebeforeinsert) | false | When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage.<br />`Config Param: COMBINE_BEFORE_INSERT` [...]
+| [hoodie.combine.before.upsert](#hoodiecombinebeforeupsert) | true | When upserted records share same key, controls whether they should be first combined (i.e de-duplicated) before writing to storage. This should be turned off only if you are absolutely certain that there are no duplicates incoming, otherwise it can lead to duplicate keys and violate the uniqueness guaran [...]
+| [hoodie.consistency.check.initial_interval_ms](#hoodieconsistencycheckinitial_interval_ms) | 2000 | Initial time between successive attempts to ensure written data's metadata is consistent on storage. Grows with exponential backoff after the initial value.<br />`Config Param: INITIAL_CONSISTENCY_CHECK_INTERVAL_MS` [...]
+| [hoodie.consistency.check.max_checks](#hoodieconsistencycheckmax_checks) | 7 | Maximum number of checks, for consistency of written data.<br />`Config Param: MAX_CONSISTENCY_CHECKS` [...]
+| [hoodie.consistency.check.max_interval_ms](#hoodieconsistencycheckmax_interval_ms) | 300000 | Max time to wait between successive attempts at performing consistency checks<br />`Config Param: MAX_CONSISTENCY_CHECK_INTERVAL_MS` [...]
+| [hoodie.datasource.write.keygenerator.type](#hoodiedatasourcewritekeygeneratortype) | SIMPLE | **Note** This is being actively worked on. Please use `hoodie.datasource.write.keygenerator.class` instead. org.apache.hudi.keygen.constant.KeyGeneratorType: Key generator type, indicating the key generator class to use, that implements `org.apache.hudi.keygen.KeyGenerator`. SIMPLE(default): Simple ke [...]
+| [hoodie.datasource.write.schema.allow.auto.evolution.column.drop](#hoodiedatasourcewriteschemaallowautoevolutioncolumndrop) | false | Controls whether table's schema is allowed to automatically evolve when incoming batch's schema can have any of the columns dropped. By default, Hudi will not allow this kind of (auto) schema evolution. Set this config to true to allow table's schema to be updated automatically when columns are dropped fr [...]
+| [hoodie.delete.shuffle.parallelism](#hoodiedeleteshuffleparallelism) | 0 | Parallelism used for delete operation. Delete operations also performs shuffles, similar to upsert operation. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards Hudi by default automatically uses the parallelism deduced by Spark [...]
+| [hoodie.embed.timeline.server](#hoodieembedtimelineserver) | true | When true, spins up an instance of the timeline server (meta server that serves cached file listings, statistics),running on each writer's driver process, accepting requests during the write from executors.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_ENABLE` [...]
+| [hoodie.embed.timeline.server.async](#hoodieembedtimelineserverasync) | false | Controls whether or not, the requests to the timeline server are processed in asynchronous fashion, potentially improving throughput.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_USE_ASYNC_ENABLE` [...]
+| [hoodie.embed.timeline.server.gzip](#hoodieembedtimelineservergzip) | true | Controls whether gzip compression is used, for large responses from the timeline server, to improve latency.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_COMPRESS_ENABLE` [...]
+| [hoodie.embed.timeline.server.port](#hoodieembedtimelineserverport) | 0 | Port at which the timeline server listens for requests. When running embedded in each writer, it picks a free port and communicates to all the executors. This should rarely be changed.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_PORT_NUM` [...]
+| [hoodie.embed.timeline.server.reuse.enabled](#hoodieembedtimelineserverreuseenabled) | false | Controls whether the timeline server instance should be cached and reused across the tablesto avoid startup costs and server overhead. This should only be used if you are running multiple writers in the same JVM.<br />`Config Param: EMBEDDED_TIMELINE_SERVER_REUSE_ENABLED` [...]
+| [hoodie.embed.timeline.server.threads](#hoodieembedtimelineserverthreads) | -1 | Number of threads to serve requests in the timeline server. By default, auto configured based on the number of underlying cores.<br />`Config Param: EMBEDDED_TIMELINE_NUM_SERVER_THREADS` [...]
+| [hoodie.fail.on.timeline.archiving](#hoodiefailontimelinearchiving) | true | Timeline archiving removes older instants from the timeline, after each write operation, to minimize metadata overhead. Controls whether or not, the write should be failed as well, if such archiving fails.<br />`Config Param: FAIL_ON_TIMELINE_ARCHIVING_ENABLE` [...]
+| [hoodie.fail.writes.on.inline.table.service.exception](#hoodiefailwritesoninlinetableserviceexception) | true | Table services such as compaction and clustering can fail and prevent syncing to the metaclient. Set this to true to fail writes when table services fail<br />`Config Param: FAIL_ON_INLINE_TABLE_SERVICE_EXCEPTION`<br />`Since Version: 0.13.0` [...]
+| [hoodie.fileid.prefix.provider.class](#hoodiefileidprefixproviderclass) | org.apache.hudi.table.RandomFileIdPrefixProvider | File Id Prefix provider class, that implements `org.apache.hudi.fileid.FileIdPrefixProvider`<br />`Config Param: FILEID_PREFIX_PROVIDER_CLASS`<br />`Since Version: 0.10.0` [...]
+| [hoodie.finalize.write.parallelism](#hoodiefinalizewriteparallelism) | 200 | Parallelism for the write finalization internal operation, which involves removing any partially written files from lake storage, before committing the write. Reduce this value, if the high number of tasks incur delays for smaller tables or low latency writes.<br />`Config Param: FINALIZE_WRITE_PARALLELIS [...]
+| [hoodie.insert.shuffle.parallelism](#hoodieinsertshuffleparallelism) | 0 | Parallelism for inserting records into the table. Inserts can shuffle data before writing to tune file sizes and optimize the storage layout. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards Hudi by default automatically uses t [...]
+| [hoodie.markers.delete.parallelism](#hoodiemarkersdeleteparallelism) | 100 | Determines the parallelism for deleting marker files, which are used to track all files (valid or invalid/partial) written during a write operation. Increase this value if delays are observed, with large batch writes.<br />`Config Param: MARKERS_DELETE_PARALLELISM_VALUE` [...]
+| [hoodie.markers.timeline_server_based.batch.interval_ms](#hoodiemarkerstimeline_server_basedbatchinterval_ms) | 50 | The batch interval in milliseconds for marker creation batch processing<br />`Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_INTERVAL_MS`<br />`Since Version: 0.9.0` [...]
+| [hoodie.markers.timeline_server_based.batch.num_threads](#hoodiemarkerstimeline_server_basedbatchnum_threads) | 20 | Number of threads to use for batch processing marker creation requests at the timeline server<br />`Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_NUM_THREADS`<br />`Since Version: 0.9.0` [...]
+| [hoodie.merge.allow.duplicate.on.inserts](#hoodiemergeallowduplicateoninserts) | true | When enabled, we allow duplicate keys even if inserts are routed to merge with an existing file (for ensuring file sizing). This is only relevant for insert operation, since upsert, delete operations will ensure unique key constraints are maintained.<br />`Config Param: MERGE_ALLOW_DUPLICATE_ON_INSERTS_EN [...]
+| [hoodie.merge.data.validation.enabled](#hoodiemergedatavalidationenabled) | false | When enabled, data validation checks are performed during merges to ensure expected number of records after merge operation.<br />`Config Param: MERGE_DATA_VALIDATION_CHECK_ENABLE` [...]
+| [hoodie.merge.small.file.group.candidates.limit](#hoodiemergesmallfilegroupcandidateslimit) | 1 | Limits number of file groups, whose base file satisfies small-file limit, to consider for appending records during upsert operation. Only applicable to MOR tables<br />`Config Param: MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT` [...]
+| [hoodie.release.resource.on.completion.enable](#hoodiereleaseresourceoncompletionenable) | true | Control to enable release all persist rdds when the spark job finish.<br />`Config Param: RELEASE_RESOURCE_ENABLE`<br />`Since Version: 0.11.0` [...]
+| [hoodie.rollback.instant.backup.dir](#hoodierollbackinstantbackupdir) | .rollback_backup | Path where instants being rolled back are copied. If not absolute path then a directory relative to .hoodie folder is created.<br />`Config Param: ROLLBACK_INSTANT_BACKUP_DIRECTORY` [...]
+| [hoodie.rollback.instant.backup.enabled](#hoodierollbackinstantbackupenabled) | false | Backup instants removed during rollback and restore (useful for debugging)<br />`Config Param: ROLLBACK_INSTANT_BACKUP_ENABLED` [...]
+| [hoodie.rollback.parallelism](#hoodierollbackparallelism) | 100 | This config controls the parallelism for rollback of commits. Rollbacks perform deletion of files or logging delete blocks to file groups on storage in parallel. The configure value limits the parallelism so that the number of Spark tasks do not exceed the value. If rollback is slow due to the limited par [...]
+| [hoodie.rollback.using.markers](#hoodierollbackusingmarkers) | true | Enables a more efficient mechanism for rollbacks based on the marker files generated during the writes. Turned on by default.<br />`Config Param: ROLLBACK_USING_MARKERS_ENABLE` [...]
+| [hoodie.sensitive.config.keys](#hoodiesensitiveconfigkeys) | ssl,tls,sasl,auth,credentials | Comma separated list of filters for sensitive config keys. Hudi Streamer will not print any configuration which contains the configured filter. For example with a configured filter `ssl`, value for config `ssl.trustore.location` would be masked.<br />`Config Param: SENSITIVE_CONFIG_KEYS_FILTER`<br />`Sinc [...]
+| [hoodie.skip.default.partition.validation](#hoodieskipdefaultpartitionvalidation) | false | When table is upgraded from pre 0.12 to 0.12, we check for "default" partition and fail if found one. Users are expected to rewrite the data in those partitions. Enabling this config will bypass this validation<br />`Config Param: SKIP_DEFAULT_PARTITION_VALIDATION`<br />`Since Version: 0.12.0` [...]
+| [hoodie.table.services.enabled](#hoodietableservicesenabled) | true | Master control to disable all table services including archive, clean, compact, cluster, etc.<br />`Config Param: TABLE_SERVICES_ENABLED`<br />`Since Version: 0.11.0` [...]
+| [hoodie.timeline.layout.version](#hoodietimelinelayoutversion) | 2 | Controls the layout of the timeline. Version 0 relied on renames, Version 1 (default) models the timeline as an immutable log relying only on atomic writes for object storage.<br />`Config Param: TIMELINE_LAYOUT_VERSION_NUM`<br />`Since Version: 0.5.1` [...]
+| [hoodie.upsert.shuffle.parallelism](#hoodieupsertshuffleparallelism) | 0 | Parallelism to use for upsert operation on the table. Upserts can shuffle data to perform index lookups, file sizing, bin packing records optimally into file groups. Before 0.13.0 release, if users do not configure it, Hudi would use 200 as the default shuffle parallelism. From 0.13.0 onwards Hudi by defa [...]
+| [hoodie.write.buffer.limit.bytes](#hoodiewritebufferlimitbytes) | 4194304 | Size of in-memory buffer used for parallelizing network reads and lake storage writes.<br />`Config Param: WRITE_BUFFER_LIMIT_BYTES_VALUE` [...]
+| [hoodie.write.buffer.record.cache.limit](#hoodiewritebufferrecordcachelimit) | 131072 | Maximum queue size of in-memory buffer for parallelizing network reads and lake storage writes.<br />`Config Param: WRITE_BUFFER_RECORD_CACHE_LIMIT`<br />`Since Version: 0.15.0` [...]
+| [hoodie.write.buffer.record.sampling.rate](#hoodiewritebufferrecordsamplingrate) | 64 | Sampling rate of in-memory buffer used to estimate object size. Higher value lead to lower CPU usage.<br />`Config Param: WRITE_BUFFER_RECORD_SAMPLING_RATE`<br />`Since Version: 0.15.0` [...]
+| [hoodie.write.concurrency.async.conflict.detector.initial_delay_ms](#hoodiewriteconcurrencyasyncconflictdetectorinitial_delay_ms) | 0 | Used for timeline-server-based markers with `AsyncTimelineServerBasedDetectionStrategy`. The time in milliseconds to delay the first execution of async marker-based conflict detection.<br />`Config Param: ASYNC_CONFLICT_DETECTOR_INITIAL_DELAY_MS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.concurrency.async.conflict.detector.period_ms](#hoodiewriteconcurrencyasyncconflictdetectorperiod_ms) | 30000 | Used for timeline-server-based markers with `AsyncTimelineServerBasedDetectionStrategy`. The period in milliseconds between successive executions of async marker-based conflict detection.<br />`Config Param: ASYNC_CONFLICT_DETECTOR_PERIOD_MS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.concurrency.early.conflict.check.commit.conflict](#hoodiewriteconcurrencyearlyconflictcheckcommitconflict) | false | Whether to enable commit conflict checking or not during early conflict detection.<br />`Config Param: EARLY_CONFLICT_DETECTION_CHECK_COMMIT_CONFLICT`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.concurrency.early.conflict.detection.enable](#hoodiewriteconcurrencyearlyconflictdetectionenable) | false | Whether to enable early conflict detection based on markers. It eagerly detects writing conflict before create markers and fails fast if a conflict is detected, to release cluster compute resources as soon as possible.<br />`Config Param: EARLY_CONFLICT_DETECTION_ENABLE`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.concurrency.early.conflict.detection.strategy](#hoodiewriteconcurrencyearlyconflictdetectionstrategy) | | The class name of the early conflict detection strategy to use. This should be a subclass of `org.apache.hudi.common.conflict.detection.EarlyConflictDetectionStrategy`.<br />`Config Param: EARLY_CONFLICT_DETECTION_STRATEGY_CLASS_NAME`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.executor.disruptor.buffer.limit.bytes](#hoodiewriteexecutordisruptorbufferlimitbytes) | 1024 | The size of the Disruptor Executor ring buffer, must be power of 2<br />`Config Param: WRITE_EXECUTOR_DISRUPTOR_BUFFER_LIMIT_BYTES`<br />`Since Version: 0.13.0` [...]
+| [hoodie.write.executor.disruptor.wait.strategy](#hoodiewriteexecutordisruptorwaitstrategy) | BLOCKING_WAIT | org.apache.hudi.common.util.queue.DisruptorWaitStrategyType: Strategy employed for making Disruptor Executor wait on a cursor. BLOCKING_WAIT(default): The slowest of the available wait strategies. However, it is the most conservative with the respect to CPU usage and will give the most consistent beha [...]
+| [hoodie.write.executor.type](#hoodiewriteexecutortype) | SIMPLE | org.apache.hudi.common.util.queue.ExecutorType: Types of executor that implements org.apache.hudi.common.util.queue.HoodieExecutor. The executor orchestrates concurrent producers and consumers communicating through a message queue. BOUNDED_IN_MEMORY: Executor which orchestrates concurrent producers an [...]
+| [hoodie.write.markers.type](#hoodiewritemarkerstype) | TIMELINE_SERVER_BASED | org.apache.hudi.common.table.marker.MarkerType: Marker type indicating how markers are stored in the file system, used for identifying the files written and cleaning up files not committed which should be deleted. DIRECT: Individual marker file corresponding to each data file is directly created by th [...]
+| [hoodie.write.num.retries.on.conflict.failures](#hoodiewritenumretriesonconflictfailures) | 0 | Maximum number of times to retry a batch on conflict failure.<br />`Config Param: NUM_RETRIES_ON_CONFLICT_FAILURES`<br />`Since Version: 0.14.0` [...]
+| [hoodie.write.partial.update.schema](#hoodiewritepartialupdateschema) | | Avro schema of the partial updates. This is automatically set by the Hudi write client and user is not expected to manually change the value.<br />`Config Param: WRITE_PARTIAL_UPDATE_SCHEMA`<br />`Since Version: 1.0.0` [...]
+| [hoodie.write.record.positions](#hoodiewriterecordpositions) | true | Whether to write record positions to the block header for data blocks containing updates and delete blocks. The record positions can be used to improve the performance of merging records from base and log files.<br />`Config Param: WRITE_RECORD_POSITIONS`<br />`Since Version: 1.0.0` [...]
+| [hoodie.write.status.storage.level](#hoodiewritestatusstoragelevel) | MEMORY_AND_DISK_SER | Write status objects hold metadata about a write (stats, errors), that is not yet committed to storage. This controls the how that information is cached for inspection by clients. We rarely expect this to be changed.<br />`Config Param: WRITE_STATUS_STORAGE_LEVEL_VALUE` [...]
+| [hoodie.write.tagged.record.storage.level](#hoodiewritetaggedrecordstoragelevel) | MEMORY_AND_DISK_SER | Determine what level of persistence is used to cache write RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: TAGGED_RECORD_STORAGE_LEVEL_VALUE` [...]
+| [hoodie.writestatus.class](#hoodiewritestatusclass) | org.apache.hudi.client.WriteStatus | Subclass of org.apache.hudi.client.WriteStatus to be used to collect information about a write. Can be overridden to collection additional metrics/statistics about the data if needed.<br />`Config Param: WRITE_STATUS_CLASS_NAME` [...]
---
@@ -1091,7 +1100,7 @@ Configurations that control locking mechanisms required for concurrency control
| [hoodie.write.lock.filesystem.expire](#hoodiewritelockfilesystemexpire) | 0 | For DFS based lock providers, expire time in minutes, must be a non-negative number, default means no expire<br />`Config Param: FILESYSTEM_LOCK_EXPIRE`<br />`Since Version: 0.12.0` |
| [hoodie.write.lock.max_wait_time_ms_between_retry](#hoodiewritelockmax_wait_time_ms_between_retry) | 16000 | Maximum amount of time to wait between retries by lock provider client. This bounds the maximum delay from the exponential backoff. Currently used by ZK based lock provider only.<br />`Config Param: LOCK_ACQUIRE_RETRY_MAX_WAIT_TIME_IN_MILLIS`<br />`Since Version: 0.8.0` |
| [hoodie.write.lock.num_retries](#hoodiewritelocknum_retries) | 15 | Maximum number of times to retry lock acquire, at each lock provider<br />`Config Param: LOCK_ACQUIRE_NUM_RETRIES`<br />`Since Version: 0.8.0` |
-| [hoodie.write.lock.provider](#hoodiewritelockprovider) | org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider | Lock provider class name, user can provide their own implementation of LockProvider which should be subclass of org.apache.hudi.common.lock.LockProvider<br />`Config Param: LOCK_PROVIDER_CLASS_NAME`<br />`Since Version: 0.8.0` |
+| [hoodie.write.lock.provider](#hoodiewritelockprovider) | org.apache.hudi.client.transaction.lock.InProcessLockProvider | Lock provider class name, user can provide their own implementation of LockProvider which should be subclass of org.apache.hudi.common.lock.LockProvider<br />`Config Param: LOCK_PROVIDER_CLASS_NAME`<br />`Since Version: 0.8.0` |
| [hoodie.write.lock.wait_time_ms](#hoodiewritelockwait_time_ms) | 60000 | Timeout in ms, to wait on an individual lock acquire() call, at the lock provider.<br />`Config Param: LOCK_ACQUIRE_WAIT_TIMEOUT_MS`<br />`Since Version: 0.8.0` |
| [hoodie.write.lock.wait_time_ms_between_retry](#hoodiewritelockwait_time_ms_between_retry) | 1000 | Initial amount of time to wait between retries to acquire locks, subsequent retries will exponentially backoff.<br />`Config Param: LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS`<br />`Since Version: 0.8.0` |
| [hoodie.write.lock.zookeeper.connection_timeout_ms](#hoodiewritelockzookeeperconnection_timeout_ms) | 15000 | Timeout in ms, to wait for establishing connection with Zookeeper.<br />`Config Param: ZK_CONNECTION_TIMEOUT_MS`<br />`Since Version: 0.8.0` |
@@ -1138,6 +1147,7 @@ Hudi maintains keys (record key + partition path) for uniquely identifying a par
| ------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [hoodie.datasource.write.partitionpath.field](#hoodiedatasourcewritepartitionpathfield) | (N/A) | Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
| [hoodie.datasource.write.recordkey.field](#hoodiedatasourcewriterecordkeyfield) | (N/A) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: RECORDKEY_FIELD_NAME` |
+| [hoodie.datasource.write.secondarykey.column](#hoodiedatasourcewritesecondarykeycolumn) | (N/A) | Columns that constitute the secondary key component. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: SECONDARYKEY_COLUMN_NAME` |
| [hoodie.datasource.write.hive_style_partitioning](#hoodiedatasourcewritehive_style_partitioning) | false | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` |
[**Advanced Configs**](#Key-Generator-Options-advanced-configs)
@@ -1160,7 +1170,7 @@ Configs used for TimestampBasedKeyGenerator which relies on timestamps for the p
| Config Name | Default | Description |
| ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.keygen.timebased.timestamp.type](#hoodiekeygentimebasedtimestamptype) | (N/A) | Timestamp type of the field, which should be one of the timestamp types supported: `UNIX_TIMESTAMP`, `DATE_STRING`, `MIXED`, `EPOCHMILLISECONDS`, `SCALAR`.<br />`Config Param: TIMESTAMP_TYPE_FIELD` |
+| [hoodie.keygen.timebased.timestamp.type](#hoodiekeygentimebasedtimestamptype) | (N/A) | Timestamp type of the field, which should be one of the timestamp types supported: `UNIX_TIMESTAMP`, `DATE_STRING`, `MIXED`, `EPOCHMILLISECONDS`, `EPOCHMICROSECONDS`, `SCALAR`.<br />`Config Param: TIMESTAMP_TYPE_FIELD` |
| [hoodie.keygen.datetime.parser.class](#hoodiekeygendatetimeparserclass) | org.apache.hudi.keygen.parser.HoodieDateTimeParser | Date time parser class name.<br />`Config Param: DATE_TIME_PARSER` |
| [hoodie.keygen.timebased.input.dateformat](#hoodiekeygentimebasedinputdateformat) | | Input date format such as `yyyy-MM-dd'T'HH:mm:ss.SSSZ`.<br />`Config Param: TIMESTAMP_INPUT_DATE_FORMAT` |
| [hoodie.keygen.timebased.input.dateformat.list.delimiter.regex](#hoodiekeygentimebasedinputdateformatlistdelimiterregex) | , | The delimiter for allowed input date format list, usually `,`.<br />`Config Param: TIMESTAMP_INPUT_DATE_FORMAT_LIST_DELIMITER_REGEX` |
@@ -1184,41 +1194,59 @@ Configurations that control indexing behavior, which tags incoming records as ei
[**Basic Configs**](#Common-Index-Configs-basic-configs)
-| Config Name | Default | Description [...]
-| ------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.index.type](#hoodieindextype) | (N/A) | org.apache.hudi.index.HoodieIndex$IndexType: Determines how input records are indexed, i.e., looked up based on the key for the location in the existing table. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java engines. HBASE: uses an external managed Apache HBase table to store record key to location mapping. HBase index is a global index, enforcing key uniqueness across all partitions in the table. INMEMORY: Use [...]
+| Config Name | Default | Description |
+| ------------------------------------------------------------------ | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.expression.index.function](#hoodieexpressionindexfunction) | (N/A) | Function to be used for building the expression index.<br />`Config Param: INDEX_FUNCTION`<br />`Since Version: 1.0.0` |
+| [hoodie.expression.index.name](#hoodieexpressionindexname) | (N/A) | Name of the expression index. This is also used for the partition name in the metadata table.<br />`Config Param: INDEX_NAME`<br />`Since Version: 1.0.0` |
+| [hoodie.table.checksum](#hoodietablechecksum) | (N/A) | Index definition checksum is used to guard against partial writes in HDFS. It is added as the last entry in index.properties and then used to validate while reading table config.<br />`Config Param: INDEX_DEFINITION_CHECKSUM`<br />`Since Version: 1.0.0` |
+| [hoodie.expression.index.type](#hoodieexpressionindextype) | COLUMN_STATS | Type of the expression index. Default is `column_stats` if there are no functions and expressions in the command. Valid options could be BITMAP, COLUMN_STATS, LUCENE, etc. If index_type is not provided, and there are functions or expressions in the command then a expression index using column stats will be created.<br />`Config Param: INDEX_TYPE`<br />`Since Version: 1.0.0` |
+---
+
+
+#### Common Index Configs {#Common-Index-Configs}
+
+
+
+
+[**Basic Configs**](#Common-Index-Configs-basic-configs)
+
+
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.index.type](#hoodieindextype) | (N/A) | org.apache.hudi.index.HoodieIndex$IndexType: Determines how input records are indexed, i.e., looked up based on the key for the location in the existing table. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java engines. HBASE: uses an external managed Apache HBase table to store record key to location mapping. HBase index is a global index, enforcing key uniqueness across all partitions i [...]
+| [hoodie.bucket.index.query.pruning](#hoodiebucketindexquerypruning) | true | Control if table with bucket index use bucket query or not<br />`Config Param: BUCKET_QUERY_INDEX` [...]
[**Advanced Configs**](#Common-Index-Configs-advanced-configs)
-| Config Name | Default | Description [...]
-| ----------------------------------------------------------------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.bucket.index.hash.field](#hoodiebucketindexhashfield) | (N/A) | Index key. It is used to index the record and find its file group. If not set, use record key field as default<br />`Config Param: BUCKET_INDEX_HASH_FIELD` [...]
-| [hoodie.bucket.index.max.num.buckets](#hoodiebucketindexmaxnumbuckets) | (N/A) | Only applies if bucket index engine is consistent hashing. Determine the upper bound of the number of buckets in the hudi table. Bucket resizing cannot be done higher than this max limit.<br />`Config Param: BUCKET_INDEX_MAX_NUM_BUCKETS`<br />`Since Version: 0.13.0` [...]
-| [hoodie.bucket.index.min.num.buckets](#hoodiebucketindexminnumbuckets) | (N/A) | Only applies if bucket index engine is consistent hashing. Determine the lower bound of the number of buckets in the hudi table. Bucket resizing cannot be done lower than this min limit.<br />`Config Param: BUCKET_INDEX_MIN_NUM_BUCKETS`<br />`Since Version: 0.13.0` [...]
-| [hoodie.bloom.index.bucketized.checking](#hoodiebloomindexbucketizedchecking) | true | Only applies if index type is BLOOM. When true, bucketized bloom filtering is enabled. This reduces skew seen in sort based bloom index lookup<br />`Config Param: BLOOM_INDEX_BUCKETIZED_CHECKING` [...]
-| [hoodie.bloom.index.input.storage.level](#hoodiebloomindexinputstoragelevel) | MEMORY_AND_DISK_SER | Only applies when #bloomIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: BLOOM_INDEX_INPUT_STORAGE_LEVEL_VALUE` [...]
-| [hoodie.bloom.index.keys.per.bucket](#hoodiebloomindexkeysperbucket) | 10000000 | Only applies if bloomIndexBucketizedChecking is enabled and index type is bloom. This configuration controls the “bucket” size which tracks the number of record-key checks made against a single file and is the unit of work allocated to each partition performing bloom filter lookup. A higher value would amortize the fixed cost of reading a bloom filter to memory.<br />`Config Par [...]
-| [hoodie.bloom.index.parallelism](#hoodiebloomindexparallelism) | 0 | Only applies if index type is BLOOM. This is the amount of parallelism for index lookup, which involves a shuffle. By default, this is auto computed based on input workload characteristics. If the parallelism is explicitly configured by the user, the user-configured value is used in defining the actual parallelism. If the indexing stage is slow due to the limited parallelism, yo [...]
-| [hoodie.bloom.index.prune.by.ranges](#hoodiebloomindexprunebyranges) | true | Only applies if index type is BLOOM. When true, range information from files to leveraged speed up index lookups. Particularly helpful, if the key has a monotonously increasing prefix, such as timestamp. If the record key is completely random, it is better to turn this off, since range pruning will only add extra overhead to the index lookup.<br />`Config Param: BLOOM_INDEX_PRU [...]
-| [hoodie.bloom.index.update.partition.path](#hoodiebloomindexupdatepartitionpath) | true | Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition<br />`Config Param: BLOOM_INDEX_UPDATE_PARTI [...]
-| [hoodie.bloom.index.use.caching](#hoodiebloomindexusecaching) | true | Only applies if index type is BLOOM.When true, the input RDD will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br />`Config Param: BLOOM_INDEX_USE_CACHING` [...]
-| [hoodie.bloom.index.use.metadata](#hoodiebloomindexusemetadata) | false | Only applies if index type is BLOOM.When true, the index lookup uses bloom filters and column stats from metadata table when available to speed up the process.<br />`Config Param: BLOOM_INDEX_USE_METADATA`<br />`Since Version: 0.11.0` [...]
-| [hoodie.bloom.index.use.treebased.filter](#hoodiebloomindexusetreebasedfilter) | true | Only applies if index type is BLOOM. When true, interval tree based file pruning optimization is enabled. This mode speeds-up file-pruning based on key ranges when compared with the brute-force mode<br />`Config Param: BLOOM_INDEX_TREE_BASED_FILTER` [...]
-| [hoodie.bucket.index.merge.threshold](#hoodiebucketindexmergethreshold) | 0.2 | Control if buckets should be merged when using consistent hashing bucket indexSpecifically, if a file slice size is smaller than `hoodie.xxxx.max.file.size` * threshold, then it will be consideredas a merge candidate.<br />`Config Param: BUCKET_MERGE_THRESHOLD`<br />`Since Version: 0.13.0` [...]
-| [hoodie.bucket.index.num.buckets](#hoodiebucketindexnumbuckets) | 256 | Only applies if index type is BUCKET. Determine the number of buckets in the hudi table, and each partition is divided to N buckets.<br />`Config Param: BUCKET_INDEX_NUM_BUCKETS` [...]
-| [hoodie.bucket.index.split.threshold](#hoodiebucketindexsplitthreshold) | 2.0 | Control if the bucket should be split when using consistent hashing bucket index.Specifically, if a file slice size reaches `hoodie.xxxx.max.file.size` * threshold, then split will be carried out.<br />`Config Param: BUCKET_SPLIT_THRESHOLD`<br />`Since Version: 0.13.0` [...]
-| [hoodie.global.index.reconcile.parallelism](#hoodieglobalindexreconcileparallelism) | 60 | Only applies if index type is GLOBAL_BLOOM or GLOBAL_SIMPLE. This controls the parallelism for deduplication during indexing where more than 1 record could be tagged due to partition update.<br />`Config Param: GLOBAL_INDEX_RECONCILE_PARALLELISM` [...]
-| [hoodie.global.simple.index.parallelism](#hoodieglobalsimpleindexparallelism) | 100 | Only applies if index type is GLOBAL_SIMPLE. This limits the parallelism of fetching records from the base files of all table partitions. The index picks the configured parallelism if the number of base files is larger than this configured value; otherwise, the number of base files is used as the parallelism. If the indexing stage is slow due to the limited parallelism, you can [...]
-| [hoodie.index.bucket.engine](#hoodieindexbucketengine) | SIMPLE | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. SIMPLE(default): Uses a fixed number of buckets for file groups which cannot shrink or expand. This works for both COW and MOR tables. CONSISTENT_HASHING: Supports dynamic number of buckets with bucket resizing to proper [...]
-| [hoodie.index.class](#hoodieindexclass) | | Full path of user-defined index class and must be a subclass of HoodieIndex class. It will take precedence over the hoodie.index.type configuration if specified<br />`Config Param: INDEX_CLASS_NAME` [...]
-| [hoodie.record.index.input.storage.level](#hoodierecordindexinputstoragelevel) | MEMORY_AND_DISK_SER | Only applies when #recordIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: RECORD_INDEX_INPUT_STORAGE_LEVEL_VALUE`<br />`Since Version: 0.14.0` [...]
-| [hoodie.record.index.update.partition.path](#hoodierecordindexupdatepartitionpath) | false | Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: true , isAdvanced: true , description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to fals [...]
-| [hoodie.record.index.use.caching](#hoodierecordindexusecaching) | true | Only applies if index type is RECORD_INDEX.When true, the input RDD will be cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br />`Config Param: RECORD_INDEX_USE_CACHING`<br />`Since Version: 0.14.0` [...]
-| [hoodie.simple.index.input.storage.level](#hoodiesimpleindexinputstoragelevel) | MEMORY_AND_DISK_SER | Only applies when #simpleIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: SIMPLE_INDEX_INPUT_STORAGE_LEVEL_VALUE` [...]
-| [hoodie.simple.index.parallelism](#hoodiesimpleindexparallelism) | 0 | Only applies if index type is SIMPLE. This limits the parallelism of fetching records from the base files of affected partitions. By default, this is auto computed based on input workload characteristics. If the parallelism is explicitly configured by the user, the user-configured value is used in defining the actual parallelism. If the indexing stage is slow due to the limited [...]
-| [hoodie.simple.index.update.partition.path](#hoodiesimpleindexupdatepartitionpath) | true | Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: true , isAdvanced: true , description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to fals [...]
-| [hoodie.simple.index.use.caching](#hoodiesimpleindexusecaching) | true | Only applies if index type is SIMPLE. When true, the incoming writes will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br />`Config Param: SIMPLE_INDEX_USE_CACHING` [...]
+| Config Name | Default | Description [...]
+| ----------------------------------------------------------------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.bucket.index.hash.field](#hoodiebucketindexhashfield) | (N/A) | Index key. It is used to index the record and find its file group. If not set, use record key field as default<br />`Config Param: BUCKET_INDEX_HASH_FIELD` [...]
+| [hoodie.bucket.index.max.num.buckets](#hoodiebucketindexmaxnumbuckets) | (N/A) | Only applies if bucket index engine is consistent hashing. Determine the upper bound of the number of buckets in the hudi table. Bucket resizing cannot be done higher than this max limit.<br />`Config Param: BUCKET_INDEX_MAX_NUM_BUCKETS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.bucket.index.min.num.buckets](#hoodiebucketindexminnumbuckets) | (N/A) | Only applies if bucket index engine is consistent hashing. Determine the lower bound of the number of buckets in the hudi table. Bucket resizing cannot be done lower than this min limit.<br />`Config Param: BUCKET_INDEX_MIN_NUM_BUCKETS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.bloom.index.bucketized.checking](#hoodiebloomindexbucketizedchecking) | true | Only applies if index type is BLOOM. When true, bucketized bloom filtering is enabled. This reduces skew seen in sort based bloom index lookup<br />`Config Param: BLOOM_INDEX_BUCKETIZED_CHECKING` [...]
+| [hoodie.bloom.index.input.storage.level](#hoodiebloomindexinputstoragelevel) | MEMORY_AND_DISK_SER | Only applies when #bloomIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: BLOOM_INDEX_INPUT_STORAGE_LEVEL_VALUE` [...]
+| [hoodie.bloom.index.keys.per.bucket](#hoodiebloomindexkeysperbucket) | 10000000 | Only applies if bloomIndexBucketizedChecking is enabled and index type is bloom. This configuration controls the “bucket” size which tracks the number of record-key checks made against a single file and is the unit of work allocated to each partition performing bloom filter lookup. A higher value would amortize the fixed cost of reading a bloom filter to memory.<br />`Config Par [...]
+| [hoodie.bloom.index.parallelism](#hoodiebloomindexparallelism) | 0 | Only applies if index type is BLOOM. This is the amount of parallelism for index lookup, which involves a shuffle. By default, this is auto computed based on input workload characteristics. If the parallelism is explicitly configured by the user, the user-configured value is used in defining the actual parallelism. If the indexing stage is slow due to the limited parallelism, yo [...]
+| [hoodie.bloom.index.prune.by.ranges](#hoodiebloomindexprunebyranges) | true | Only applies if index type is BLOOM. When true, range information from files to leveraged speed up index lookups. Particularly helpful, if the key has a monotonously increasing prefix, such as timestamp. If the record key is completely random, it is better to turn this off, since range pruning will only add extra overhead to the index lookup.<br />`Config Param: BLOOM_INDEX_PRU [...]
+| [hoodie.bloom.index.update.partition.path](#hoodiebloomindexupdatepartitionpath) | true | Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition<br />`Config Param: BLOOM_INDEX_UPDATE_PARTI [...]
+| [hoodie.bloom.index.use.caching](#hoodiebloomindexusecaching) | true | Only applies if index type is BLOOM.When true, the input RDD will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br />`Config Param: BLOOM_INDEX_USE_CACHING` [...]
+| [hoodie.bloom.index.use.metadata](#hoodiebloomindexusemetadata) | false | Only applies if index type is BLOOM.When true, the index lookup uses bloom filters and column stats from metadata table when available to speed up the process.<br />`Config Param: BLOOM_INDEX_USE_METADATA`<br />`Since Version: 0.11.0` [...]
+| [hoodie.bloom.index.use.treebased.filter](#hoodiebloomindexusetreebasedfilter) | true | Only applies if index type is BLOOM. When true, interval tree based file pruning optimization is enabled. This mode speeds-up file-pruning based on key ranges when compared with the brute-force mode<br />`Config Param: BLOOM_INDEX_TREE_BASED_FILTER` [...]
+| [hoodie.bucket.index.merge.threshold](#hoodiebucketindexmergethreshold) | 0.2 | Control if buckets should be merged when using consistent hashing bucket indexSpecifically, if a file slice size is smaller than `hoodie.xxxx.max.file.size` * threshold, then it will be consideredas a merge candidate.<br />`Config Param: BUCKET_MERGE_THRESHOLD`<br />`Since Version: 0.13.0` [...]
+| [hoodie.bucket.index.num.buckets](#hoodiebucketindexnumbuckets) | 256 | Only applies if index type is BUCKET. Determine the number of buckets in the hudi table, and each partition is divided to N buckets.<br />`Config Param: BUCKET_INDEX_NUM_BUCKETS` [...]
+| [hoodie.bucket.index.split.threshold](#hoodiebucketindexsplitthreshold) | 2.0 | Control if the bucket should be split when using consistent hashing bucket index.Specifically, if a file slice size reaches `hoodie.xxxx.max.file.size` * threshold, then split will be carried out.<br />`Config Param: BUCKET_SPLIT_THRESHOLD`<br />`Since Version: 0.13.0` [...]
+| [hoodie.global.index.reconcile.parallelism](#hoodieglobalindexreconcileparallelism) | 60 | Only applies if index type is GLOBAL_BLOOM or GLOBAL_SIMPLE. This controls the parallelism for deduplication during indexing where more than 1 record could be tagged due to partition update.<br />`Config Param: GLOBAL_INDEX_RECONCILE_PARALLELISM` [...]
+| [hoodie.global.simple.index.parallelism](#hoodieglobalsimpleindexparallelism) | 0 | Only applies if index type is GLOBAL_SIMPLE. This limits the parallelism of fetching records from the base files of all table partitions. The index picks the configured parallelism if the number of base files is larger than this configured value; otherwise, the number of base files is used as the parallelism. If the indexing stage is slow due to the limited parallelism, you can [...]
+| [hoodie.index.bucket.engine](#hoodieindexbucketengine) | SIMPLE | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. SIMPLE(default): Uses a fixed number of buckets for file groups which cannot shrink or expand. This works for both COW and MOR tables. CONSISTENT_HASHING: Supports dynamic number of buckets with bucket resizing to proper [...]
+| [hoodie.index.class](#hoodieindexclass) | | Full path of user-defined index class and must be a subclass of HoodieIndex class. It will take precedence over the hoodie.index.type configuration if specified<br />`Config Param: INDEX_CLASS_NAME` [...]
+| [hoodie.record.index.input.storage.level](#hoodierecordindexinputstoragelevel) | MEMORY_AND_DISK_SER | Only applies when #recordIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: RECORD_INDEX_INPUT_STORAGE_LEVEL_VALUE`<br />`Since Version: 0.14.0` [...]
+| [hoodie.record.index.update.partition.path](#hoodierecordindexupdatepartitionpath) | false | Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: true , isAdvanced: true , description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to fals [...]
+| [hoodie.record.index.use.caching](#hoodierecordindexusecaching) | true | Only applies if index type is RECORD_INDEX.When true, the input RDD will be cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br />`Config Param: RECORD_INDEX_USE_CACHING`<br />`Since Version: 0.14.0` [...]
+| [hoodie.simple.index.input.storage.level](#hoodiesimpleindexinputstoragelevel) | MEMORY_AND_DISK_SER | Only applies when #simpleIndexUseCaching is set. Determine what level of persistence is used to cache input RDDs. Refer to org.apache.spark.storage.StorageLevel for different values<br />`Config Param: SIMPLE_INDEX_INPUT_STORAGE_LEVEL_VALUE` [...]
+| [hoodie.simple.index.parallelism](#hoodiesimpleindexparallelism) | 0 | Only applies if index type is SIMPLE. This limits the parallelism of fetching records from the base files of affected partitions. By default, this is auto computed based on input workload characteristics. If the parallelism is explicitly configured by the user, the user-configured value is used in defining the actual parallelism. If the indexing stage is slow due to the limited [...]
+| [hoodie.simple.index.update.partition.path](#hoodiesimpleindexupdatepartitionpath) | true | Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: true , isAdvanced: true , description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to fals [...]
+| [hoodie.simple.index.use.caching](#hoodiesimpleindexusecaching) | true | Only applies if index type is SIMPLE. When true, the incoming writes will cached to speed up index lookup by reducing IO for computing parallelism or affected partitions<br />`Config Param: SIMPLE_INDEX_USE_CACHING` [...]
---
@@ -1273,9 +1301,15 @@ Configurations that control file group reading.
[**Advanced Configs**](#Reader-Configs-advanced-configs)
-| Config Name | Default | Description |
-| --------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [_hoodie.hfile.use.native.reader](#_hoodiehfileusenativereader) | false | When enabled, the native HFile reader is used to read HFiles. This is an internal config.<br />`Config Param: USE_NATIVE_HFILE_READER`<br />`Since Version: 0.15.0` |
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
+| [_hoodie.hfile.use.native.reader](#_hoodiehfileusenativereader) | true | When enabled, the native HFile reader is used to read HFiles. This is an internal config.<br />`Config Param: USE_NATIVE_HFILE_READER`<br />`Since Version: 1.0.0` [...]
+| [hoodie.compaction.lazy.block.read](#hoodiecompactionlazyblockread) | true | When merging the delta log files, this config helps to choose whether the log blocks should be read lazily or not. Choose true to use lazy block reading (low memory usage, but incurs seeks to each block header) or false for immediate block read (higher memory usage)<br />`Config Param: COMPACTION_LAZY_BLOCK_READ_ENABLE` [...]
+| [hoodie.compaction.reverse.log.read](#hoodiecompactionreverselogread) | false | HoodieLogFormatReader reads a logfile in the forward direction starting from pos=0 to pos=file_length. If this config is set to true, the reader reads the logfile in reverse direction, from pos=file_length to pos=0<br />`Config Param: COMPACTION_REVERSE_LOG_READ_ENABLE` [...]
+| [hoodie.datasource.merge.type](#hoodiedatasourcemergetype) | payload_combine | For Snapshot query on merge on read table. Use this key to define how the payloads are merged, in 1) skip_merge: read the base file records plus the log file records without merging; 2) payload_combine: read the base file records first, for each record in base file, checks whether the key is in the log file records (combines the two records with same key for base and log file records) [...]
+| [hoodie.file.group.reader.enabled](#hoodiefilegroupreaderenabled) | true | Use engine agnostic file group reader if enabled<br />`Config Param: FILE_GROUP_READER_ENABLED`<br />`Since Version: 1.0.0` [...]
+| [hoodie.merge.use.record.positions](#hoodiemergeuserecordpositions) | false | Whether to use positions in the block header for data blocks containing updates and delete blocks for merging.<br />`Config Param: MERGE_USE_RECORD_POSITIONS`<br />`Since Version: 1.0.0` [...]
+| [hoodie.optimized.log.blocks.scan.enable](#hoodieoptimizedlogblocksscanenable) | false | New optimized scan for log blocks that handles all multi-writer use-cases while appending to log files. It also differentiates original blocks written by ingestion writers and compacted blocks written log compaction.<br />`Config Param: ENABLE_OPTIMIZED_LOG_BLOCKS_SCAN`<br />`Since Version: 0.13.0` [...]
---
## Metastore and Catalog Sync Configs {#META_SYNC}
@@ -1297,21 +1331,21 @@ Configurations used by the Hudi to sync metadata to external metastores and cata
[**Advanced Configs**](#Common-Metadata-Sync-Configs-advanced-configs)
-| Config Name | Default | Description |
-| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` |
-| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` |
-| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` |
-| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` |
-| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` |
-| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` |
-| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` |
-| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` |
-| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` |
-| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` |
-| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` |
-| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` |
-| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` |
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` [...]
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` [...]
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` [...]
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` [...]
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` [...]
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` [...]
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` [...]
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` [...]
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | true | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
+| [hoodie.meta.sync.no_partition_metadata](#hoodiemetasyncno_partition_metadata) | false | If true, the partition metadata will not be synced to the metastore. This is useful when the partition metadata is large, and the partition info can be obtained from Hudi's internal metadata table. Note, Key: 'hoodie.metadata.enable' , default: true , isAdvanced: false , description: Enable the internal metadata table which s [...]
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` [...]
---
@@ -1331,13 +1365,14 @@ Configs that control Glue catalog sync based client.
[**Advanced Configs**](#Glue-catalog-sync-based-client-Configurations-advanced-configs)
-| Config Name | Default | Description |
-| ---------------------------------------------------------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.datasource.meta.sync.glue.all_partitions_read_parallelism](#hoodiedatasourcemetasyncglueall_partitions_read_parallelism) | 1 | Parallelism for listing all partitions(first time sync). Should be in interval [1, 10].<br />`Config Param: ALL_PARTITIONS_READ_PARALLELISM`<br />`Since Version: 0.15.0` |
-| [hoodie.datasource.meta.sync.glue.changed_partitions_read_parallelism](#hoodiedatasourcemetasyncgluechanged_partitions_read_parallelism) | 1 | Parallelism for listing changed partitions(second and subsequent syncs).<br />`Config Param: CHANGED_PARTITIONS_READ_PARALLELISM`<br />`Since Version: 0.15.0` |
-| [hoodie.datasource.meta.sync.glue.metadata_file_listing](#hoodiedatasourcemetasyncgluemetadata_file_listing) | false | Makes athena use the metadata table to list partitions and files. Currently it won't benefit from other features such stats indexes<br />`Config Param: GLUE_METADATA_FILE_LISTING`<br />`Since Version: 0.14.0` |
-| [hoodie.datasource.meta.sync.glue.partition_change_parallelism](#hoodiedatasourcemetasyncgluepartition_change_parallelism) | 1 | Parallelism for change operations - such as create/update/delete.<br />`Config Param: PARTITION_CHANGE_PARALLELISM`<br />`Since Version: 0.15.0` |
-| [hoodie.datasource.meta.sync.glue.skip_table_archive](#hoodiedatasourcemetasyncglueskip_table_archive) | true | Glue catalog sync based client will skip archiving the table version if this config is set to true<br />`Config Param: GLUE_SKIP_TABLE_ARCHIVE`<br />`Since Version: 0.14.0` |
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.datasource.meta.sync.glue.all_partitions_read_parallelism](#hoodiedatasourcemetasyncglueall_partitions_read_parallelism) | 1 | Parallelism for listing all partitions(first time sync). Should be in interval [1, 10].<br />`Config Param: ALL_PARTITIONS_READ_PARALLELISM`<br />`Since Version: 0.15.0` |
+| [hoodie.datasource.meta.sync.glue.changed_partitions_read_parallelism](#hoodiedatasourcemetasyncgluechanged_partitions_read_parallelism) | 1 | Parallelism for listing changed partitions(second and subsequent syncs).<br />`Config Param: CHANGED_PARTITIONS_READ_PARALLELISM`<br />`Since Version: 0.15.0` |
+| [hoodie.datasource.meta.sync.glue.metadata_file_listing](#hoodiedatasourcemetasyncgluemetadata_file_listing) | false | Makes athena use the metadata table to list partitions and files. Currently it won't benefit from other features such stats indexes<br />`Config Param: GLUE_METADATA_FILE_LISTING`<br />`Since Version: 0.14.0` |
+| [hoodie.datasource.meta.sync.glue.partition_change_parallelism](#hoodiedatasourcemetasyncgluepartition_change_parallelism) | 1 | Parallelism for change operations - such as create/update/delete.<br />`Config Param: PARTITION_CHANGE_PARALLELISM`<br />`Since Version: 0.15.0` |
+| [hoodie.datasource.meta.sync.glue.recreate_table_on_error](#hoodiedatasourcemetasyncgluerecreate_table_on_error) | false | Glue sync may fail if the Glue table exists with partitions differing from the Hoodie table or if schema evolution is not supported by Glue.Enabling this configuration will drop and create the table to match the Hoodie config<br />`Config Param: RECREATE_GLUE_TABLE_ON_ERROR`<br />`Since Version: 0.14.0` |
+| [hoodie.datasource.meta.sync.glue.skip_table_archive](#hoodiedatasourcemetasyncglueskip_table_archive) | true | Glue catalog sync based client will skip archiving the table version if this config is set to true<br />`Config Param: GLUE_SKIP_TABLE_ARCHIVE`<br />`Since Version: 0.14.0` |
---
@@ -1356,33 +1391,33 @@ Configurations used by the Hudi to sync metadata to Google BigQuery.
[**Advanced Configs**](#BigQuery-Sync-Configs-advanced-configs)
-| Config Name | Default | Description [...]
-| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
-| [hoodie.gcp.bigquery.sync.big_lake_connection_id](#hoodiegcpbigquerysyncbig_lake_connection_id) | (N/A) | The Big Lake connection ID to use<br />`Config Param: BIGQUERY_SYNC_BIG_LAKE_CONNECTION_ID`<br />`Since Version: 0.14.1` [...]
-| [hoodie.gcp.bigquery.sync.dataset_location](#hoodiegcpbigquerysyncdataset_location) | (N/A) | Location of the target dataset in BigQuery<br />`Config Param: BIGQUERY_SYNC_DATASET_LOCATION` [...]
-| [hoodie.gcp.bigquery.sync.project_id](#hoodiegcpbigquerysyncproject_id) | (N/A) | Name of the target project in BigQuery<br />`Config Param: BIGQUERY_SYNC_PROJECT_ID` [...]
-| [hoodie.gcp.bigquery.sync.source_uri](#hoodiegcpbigquerysyncsource_uri) | (N/A) | Name of the source uri gcs path of the table<br />`Config Param: BIGQUERY_SYNC_SOURCE_URI` [...]
-| [hoodie.gcp.bigquery.sync.source_uri_prefix](#hoodiegcpbigquerysyncsource_uri_prefix) | (N/A) | Name of the source uri gcs path prefix of the table<br />`Config Param: BIGQUERY_SYNC_SOURCE_URI_PREFIX` [...]
-| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` [...]
-| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` [...]
-| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` [...]
-| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` [...]
-| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` [...]
-| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` [...]
-| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` [...]
-| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` [...]
-| [hoodie.gcp.bigquery.sync.base_path](#hoodiegcpbigquerysyncbase_path) | | Base path of the hoodie table to sync<br />`Config Param: BIGQUERY_SYNC_SYNC_BASE_PATH` [...]
-| [hoodie.gcp.bigquery.sync.dataset_name](#hoodiegcpbigquerysyncdataset_name) | | Name of the target dataset in BigQuery<br />`Config Param: BIGQUERY_SYNC_DATASET_NAME` [...]
-| [hoodie.gcp.bigquery.sync.partition_fields](#hoodiegcpbigquerysyncpartition_fields) | | Comma-delimited partition fields. Default to non-partitioned.<br />`Config Param: BIGQUERY_SYNC_PARTITION_FIELDS` [...]
-| [hoodie.gcp.bigquery.sync.require_partition_filter](#hoodiegcpbigquerysyncrequire_partition_filter) | false | If true, configure table to require a partition filter to be specified when querying the table<br />`Config Param: BIGQUERY_SYNC_REQUIRE_PARTITION_FILTER`<br />`Since Version: 0.14.1` [...]
-| [hoodie.gcp.bigquery.sync.table_name](#hoodiegcpbigquerysynctable_name) | | Name of the target table in BigQuery<br />`Config Param: BIGQUERY_SYNC_TABLE_NAME` [...]
-| [hoodie.gcp.bigquery.sync.use_bq_manifest_file](#hoodiegcpbigquerysyncuse_bq_manifest_file) | false | If true, generate a manifest file with data file absolute paths and use BigQuery manifest file support to directly create one external table over the Hudi table. If false (default), generate a manifest file with data file names and create two external tables and one view in BigQuery. Query the view for the same results as [...]
-| [hoodie.gcp.bigquery.sync.use_file_listing_from_metadata](#hoodiegcpbigquerysyncuse_file_listing_from_metadata) | false | Fetch file listing from Hudi's metadata<br />`Config Param: BIGQUERY_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
-| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` [...]
-| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` [...]
-| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
-| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` [...]
-| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` [...]
+| Config Name | Default | Description [...]
+| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
+| [hoodie.gcp.bigquery.sync.big_lake_connection_id](#hoodiegcpbigquerysyncbig_lake_connection_id) | (N/A) | The Big Lake connection ID to use<br />`Config Param: BIGQUERY_SYNC_BIG_LAKE_CONNECTION_ID`<br />`Since Version: 0.14.1` [...]
+| [hoodie.gcp.bigquery.sync.billing.project.id](#hoodiegcpbigquerysyncbillingprojectid) | (N/A) | Name of the billing project id in BigQuery. By default it uses the configuration from `hoodie.gcp.bigquery.sync.project_id` if this configuration is not set. This can only be used with manifest file based approach<br />`Config Param: BIGQUERY_SYNC_BILLING_PROJECT_ID`<br />`Since Version: 1.0.0` [...]
+| [hoodie.gcp.bigquery.sync.dataset_location](#hoodiegcpbigquerysyncdataset_location) | (N/A) | Location of the target dataset in BigQuery<br />`Config Param: BIGQUERY_SYNC_DATASET_LOCATION` [...]
+| [hoodie.gcp.bigquery.sync.project_id](#hoodiegcpbigquerysyncproject_id) | (N/A) | Name of the target project in BigQuery<br />`Config Param: BIGQUERY_SYNC_PROJECT_ID` [...]
+| [hoodie.gcp.bigquery.sync.source_uri](#hoodiegcpbigquerysyncsource_uri) | (N/A) | Name of the source uri gcs path of the table<br />`Config Param: BIGQUERY_SYNC_SOURCE_URI` [...]
+| [hoodie.gcp.bigquery.sync.source_uri_prefix](#hoodiegcpbigquerysyncsource_uri_prefix) | (N/A) | Name of the source uri gcs path prefix of the table<br />`Config Param: BIGQUERY_SYNC_SOURCE_URI_PREFIX` [...]
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` [...]
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` [...]
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` [...]
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` [...]
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` [...]
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` [...]
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` [...]
+| [hoodie.gcp.bigquery.sync.dataset_name](#hoodiegcpbigquerysyncdataset_name) | | Name of the target dataset in BigQuery<br />`Config Param: BIGQUERY_SYNC_DATASET_NAME` [...]
+| [hoodie.gcp.bigquery.sync.partition_fields](#hoodiegcpbigquerysyncpartition_fields) | | Comma-delimited partition fields. Default to non-partitioned.<br />`Config Param: BIGQUERY_SYNC_PARTITION_FIELDS` [...]
+| [hoodie.gcp.bigquery.sync.require_partition_filter](#hoodiegcpbigquerysyncrequire_partition_filter) | false | If true, configure table to require a partition filter to be specified when querying the table<br />`Config Param: BIGQUERY_SYNC_REQUIRE_PARTITION_FILTER`<br />`Since Version: 0.14.1` [...]
+| [hoodie.gcp.bigquery.sync.table_name](#hoodiegcpbigquerysynctable_name) | | Name of the target table in BigQuery<br />`Config Param: BIGQUERY_SYNC_TABLE_NAME` [...]
+| [hoodie.gcp.bigquery.sync.use_bq_manifest_file](#hoodiegcpbigquerysyncuse_bq_manifest_file) | false | If true, generate a manifest file with data file absolute paths and use BigQuery manifest file support to directly create one external table over the Hudi table. If false (default), generate a manifest file with data file names and create two external tables and one view in BigQuery. Query the view for the same results as [...]
+| [hoodie.gcp.bigquery.sync.use_file_listing_from_metadata](#hoodiegcpbigquerysyncuse_file_listing_from_metadata) | true | Fetch file listing from Hudi's metadata<br />`Config Param: BIGQUERY_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` [...]
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | true | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
+| [hoodie.meta.sync.no_partition_metadata](#hoodiemetasyncno_partition_metadata) | false | If true, the partition metadata will not be synced to the metastore. This is useful when the partition metadata is large, and the partition info can be obtained from Hudi's internal metadata table. Note, Key: 'hoodie.metadata.enable' , default: true , isAdvanced: false , description: Enable the internal metadata table whic [...]
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` [...]
---
@@ -1405,42 +1440,43 @@ Configurations used by the Hudi to sync metadata to Hive Metastore.
[**Advanced Configs**](#Hive-Sync-Configs-advanced-configs)
-| Config Name | Default | Description |
-| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` |
-| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` |
-| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` |
-| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` |
-| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` |
-| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` |
-| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` |
-| [hoodie.datasource.hive_sync.bucket_sync_spec](#hoodiedatasourcehive_syncbucket_sync_spec) | | The hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC_SPEC` |
-| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` |
-| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` |
-| [hoodie.datasource.hive_sync.filter_pushdown_enabled](#hoodiedatasourcehive_syncfilter_pushdown_enabled) | false | Whether to enable push down partitions by filter<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_ENABLED` |
-| [hoodie.datasource.hive_sync.filter_pushdown_max_size](#hoodiedatasourcehive_syncfilter_pushdown_max_size) | 1000 | Max size limit to push down partition filters, if the estimate push down filters exceed this size, will directly try to fetch all partitions between the min/max.In case of glue metastore, this value should be reduced because it has a filter length limit.<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_MAX_SIZE` |
-| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` |
-| [hoodie.datasource.hive_sync.omit_metadata_fields](#hoodiedatasourcehive_syncomit_metadata_fields) | false | Whether to omit the hoodie metadata fields in the target table.<br />`Config Param: HIVE_SYNC_OMIT_METADATA_FIELDS`<br />`Since Version: 0.13.0` |
-| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` |
-| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` |
-| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` |
-| [hoodie.datasource.hive_sync.schema_string_length_thresh](#hoodiedatasourcehive_syncschema_string_length_thresh) | 4000 | <br />`Config Param: HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD` |
-| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` |
-| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` |
-| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` |
-| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` |
-| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` |
-| [hoodie.datasource.hive_sync.table.strategy](#hoodiedatasourcehive_synctablestrategy) | ALL | Hive table synchronization strategy. Available option: RO, RT, ALL.<br />`Config Param: HIVE_SYNC_TABLE_STRATEGY`<br />`Since Version: 0.13.0` |
-| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` |
-| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMAT` |
-| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` |
-| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` |
-| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` |
-| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` |
-| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` |
-| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` |
-| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` |
-| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` |
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` [...]
+| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` [...]
+| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` [...]
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` [...]
+| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` [...]
+| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` [...]
+| [hoodie.datasource.hive_sync.bucket_sync_spec](#hoodiedatasourcehive_syncbucket_sync_spec) | | The hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC_SPEC` [...]
+| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` [...]
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` [...]
+| [hoodie.datasource.hive_sync.filter_pushdown_enabled](#hoodiedatasourcehive_syncfilter_pushdown_enabled) | false | Whether to enable push down partitions by filter<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_ENABLED` [...]
+| [hoodie.datasource.hive_sync.filter_pushdown_max_size](#hoodiedatasourcehive_syncfilter_pushdown_max_size) | 1000 | Max size limit to push down partition filters, if the estimate push down filters exceed this size, will directly try to fetch all partitions between the min/max.In case of glue metastore, this value should be reduced because it has a filter length limit.<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_MAX_SIZE` [...]
+| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` [...]
+| [hoodie.datasource.hive_sync.omit_metadata_fields](#hoodiedatasourcehive_syncomit_metadata_fields) | false | Whether to omit the hoodie metadata fields in the target table.<br />`Config Param: HIVE_SYNC_OMIT_METADATA_FIELDS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` [...]
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` [...]
+| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` [...]
+| [hoodie.datasource.hive_sync.recreate_table_on_error](#hoodiedatasourcehive_syncrecreate_table_on_error) | false | Hive sync may fail if the Hive table exists with partitions differing from the Hoodie table or if schema evolution if not supported by Hive.Enabling this configuration will drop and create the table to match the Hoodie config<br />`Config Param: RECREATE_HIVE_TABLE_ON_ERROR`<br />`Since Version: 0.14.0` [...]
+| [hoodie.datasource.hive_sync.schema_string_length_thresh](#hoodiedatasourcehive_syncschema_string_length_thresh) | 4000 | <br />`Config Param: HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD` [...]
+| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` [...]
+| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` [...]
+| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` [...]
+| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` [...]
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` [...]
+| [hoodie.datasource.hive_sync.table.strategy](#hoodiedatasourcehive_synctablestrategy) | ALL | Hive table synchronization strategy. Available option: RO, RT, ALL.<br />`Config Param: HIVE_SYNC_TABLE_STRATEGY`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` [...]
+| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMA [...]
+| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` [...]
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` [...]
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` [...]
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` [...]
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | true | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
+| [hoodie.meta.sync.no_partition_metadata](#hoodiemetasyncno_partition_metadata) | false | If true, the partition metadata will not be synced to the metastore. This is useful when the partition metadata is large, and the partition info can be obtained from Hudi's internal metadata table. Note, Key: 'hoodie.metadata.enable' , default: true , isAdvanced: false , description: Enable the internal metadata table whi [...]
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` [...]
---
@@ -1463,43 +1499,44 @@ Global replication configurations used by the Hudi to sync metadata to Hive Meta
[**Advanced Configs**](#Global-Hive-Sync-Configs-advanced-configs)
-| Config Name | Default | Description |
-| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` |
-| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` |
-| [hoodie.meta_sync.global.replicate.timestamp](#hoodiemeta_syncglobalreplicatetimestamp) | (N/A) | <br />`Config Param: META_SYNC_GLOBAL_REPLICATE_TIMESTAMP` |
-| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` |
-| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` |
-| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` |
-| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` |
-| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` |
-| [hoodie.datasource.hive_sync.bucket_sync_spec](#hoodiedatasourcehive_syncbucket_sync_spec) | | The hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC_SPEC` |
-| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` |
-| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` |
-| [hoodie.datasource.hive_sync.filter_pushdown_enabled](#hoodiedatasourcehive_syncfilter_pushdown_enabled) | false | Whether to enable push down partitions by filter<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_ENABLED` |
-| [hoodie.datasource.hive_sync.filter_pushdown_max_size](#hoodiedatasourcehive_syncfilter_pushdown_max_size) | 1000 | Max size limit to push down partition filters, if the estimate push down filters exceed this size, will directly try to fetch all partitions between the min/max.In case of glue metastore, this value should be reduced because it has a filter length limit.<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_MAX_SIZE` |
-| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` |
-| [hoodie.datasource.hive_sync.omit_metadata_fields](#hoodiedatasourcehive_syncomit_metadata_fields) | false | Whether to omit the hoodie metadata fields in the target table.<br />`Config Param: HIVE_SYNC_OMIT_METADATA_FIELDS`<br />`Since Version: 0.13.0` |
-| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` |
-| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` |
-| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` |
-| [hoodie.datasource.hive_sync.schema_string_length_thresh](#hoodiedatasourcehive_syncschema_string_length_thresh) | 4000 | <br />`Config Param: HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD` |
-| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` |
-| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` |
-| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` |
-| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` |
-| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` |
-| [hoodie.datasource.hive_sync.table.strategy](#hoodiedatasourcehive_synctablestrategy) | ALL | Hive table synchronization strategy. Available option: RO, RT, ALL.<br />`Config Param: HIVE_SYNC_TABLE_STRATEGY`<br />`Since Version: 0.13.0` |
-| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` |
-| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMAT` |
-| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` |
-| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` |
-| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` |
-| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` |
-| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` |
-| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` |
-| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` |
-| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` |
+| Config Name | Default | Description [...]
+| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.datasource.hive_sync.serde_properties](#hoodiedatasourcehive_syncserde_properties) | (N/A) | Serde properties to hive table.<br />`Config Param: HIVE_TABLE_SERDE_PROPERTIES` [...]
+| [hoodie.datasource.hive_sync.table_properties](#hoodiedatasourcehive_synctable_properties) | (N/A) | Additional properties to store with table.<br />`Config Param: HIVE_TABLE_PROPERTIES` [...]
+| [hoodie.meta_sync.global.replicate.timestamp](#hoodiemeta_syncglobalreplicatetimestamp) | (N/A) | <br />`Config Param: META_SYNC_GLOBAL_REPLICATE_TIMESTAMP` [...]
+| [hoodie.datasource.hive_sync.auto_create_database](#hoodiedatasourcehive_syncauto_create_database) | true | Auto create hive database if does not exists<br />`Config Param: HIVE_AUTO_CREATE_DATABASE` [...]
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` [...]
+| [hoodie.datasource.hive_sync.batch_num](#hoodiedatasourcehive_syncbatch_num) | 1000 | The number of partitions one batch when synchronous partitions to hive.<br />`Config Param: HIVE_BATCH_SYNC_PARTITION_NUM` [...]
+| [hoodie.datasource.hive_sync.bucket_sync](#hoodiedatasourcehive_syncbucket_sync) | false | Whether sync hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC` [...]
+| [hoodie.datasource.hive_sync.bucket_sync_spec](#hoodiedatasourcehive_syncbucket_sync_spec) | | The hive metastore bucket specification when using bucket index.The specification is 'CLUSTERED BY (trace_id) SORTED BY (trace_id ASC) INTO 65536 BUCKETS'<br />`Config Param: HIVE_SYNC_BUCKET_SYNC_SPEC` [...]
+| [hoodie.datasource.hive_sync.create_managed_table](#hoodiedatasourcehive_synccreate_managed_table) | false | Whether to sync the table as managed table.<br />`Config Param: HIVE_CREATE_MANAGED_TABLE` [...]
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` [...]
+| [hoodie.datasource.hive_sync.filter_pushdown_enabled](#hoodiedatasourcehive_syncfilter_pushdown_enabled) | false | Whether to enable push down partitions by filter<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_ENABLED` [...]
+| [hoodie.datasource.hive_sync.filter_pushdown_max_size](#hoodiedatasourcehive_syncfilter_pushdown_max_size) | 1000 | Max size limit to push down partition filters, if the estimate push down filters exceed this size, will directly try to fetch all partitions between the min/max.In case of glue metastore, this value should be reduced because it has a filter length limit.<br />`Config Param: HIVE_SYNC_FILTER_PUSHDOWN_MAX_SIZE` [...]
+| [hoodie.datasource.hive_sync.ignore_exceptions](#hoodiedatasourcehive_syncignore_exceptions) | false | Ignore exceptions when syncing with Hive.<br />`Config Param: HIVE_IGNORE_EXCEPTIONS` [...]
+| [hoodie.datasource.hive_sync.omit_metadata_fields](#hoodiedatasourcehive_syncomit_metadata_fields) | false | Whether to omit the hoodie metadata fields in the target table.<br />`Config Param: HIVE_SYNC_OMIT_METADATA_FIELDS`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` [...]
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` [...]
+| [hoodie.datasource.hive_sync.password](#hoodiedatasourcehive_syncpassword) | hive | hive password to use<br />`Config Param: HIVE_PASS` [...]
+| [hoodie.datasource.hive_sync.recreate_table_on_error](#hoodiedatasourcehive_syncrecreate_table_on_error) | false | Hive sync may fail if the Hive table exists with partitions differing from the Hoodie table or if schema evolution if not supported by Hive.Enabling this configuration will drop and create the table to match the Hoodie config<br />`Config Param: RECREATE_HIVE_TABLE_ON_ERROR`<br />`Since Version: 0.14.0` [...]
+| [hoodie.datasource.hive_sync.schema_string_length_thresh](#hoodiedatasourcehive_syncschema_string_length_thresh) | 4000 | <br />`Config Param: HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD` [...]
+| [hoodie.datasource.hive_sync.skip_ro_suffix](#hoodiedatasourcehive_syncskip_ro_suffix) | false | Skip the _ro suffix for Read optimized table, when registering<br />`Config Param: HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE` [...]
+| [hoodie.datasource.hive_sync.support_timestamp](#hoodiedatasourcehive_syncsupport_timestamp) | false | ‘INT64’ with original type TIMESTAMP_MICROS is converted to hive ‘timestamp’ type. Disabled by default for backward compatibility. NOTE: On Spark entrypoints, this is defaulted to TRUE<br />`Config Param: HIVE_SUPPORT_TIMESTAMP_TYPE` [...]
+| [hoodie.datasource.hive_sync.sync_as_datasource](#hoodiedatasourcehive_syncsync_as_datasource) | true | <br />`Config Param: HIVE_SYNC_AS_DATA_SOURCE_TABLE` [...]
+| [hoodie.datasource.hive_sync.sync_comment](#hoodiedatasourcehive_syncsync_comment) | false | Whether to sync the table column comments while syncing the table.<br />`Config Param: HIVE_SYNC_COMMENT` [...]
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` [...]
+| [hoodie.datasource.hive_sync.table.strategy](#hoodiedatasourcehive_synctablestrategy) | ALL | Hive table synchronization strategy. Available option: RO, RT, ALL.<br />`Config Param: HIVE_SYNC_TABLE_STRATEGY`<br />`Since Version: 0.13.0` [...]
+| [hoodie.datasource.hive_sync.use_jdbc](#hoodiedatasourcehive_syncuse_jdbc) | true | Use JDBC when hive synchronization is enabled<br />`Config Param: HIVE_USE_JDBC` [...]
+| [hoodie.datasource.hive_sync.use_pre_apache_input_format](#hoodiedatasourcehive_syncuse_pre_apache_input_format) | false | Flag to choose InputFormat under com.uber.hoodie package instead of org.apache.hudi package. Use this when you are in the process of migrating from com.uber.hoodie to org.apache.hudi. Stop using this after you migrated the table definition to org.apache.hudi input format<br />`Config Param: HIVE_USE_PRE_APACHE_INPUT_FORMA [...]
+| [hoodie.datasource.hive_sync.username](#hoodiedatasourcehive_syncusername) | hive | hive user name to use<br />`Config Param: HIVE_USER` [...]
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` [...]
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` [...]
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` [...]
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | true | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
+| [hoodie.meta.sync.no_partition_metadata](#hoodiemetasyncno_partition_metadata) | false | If true, the partition metadata will not be synced to the metastore. This is useful when the partition metadata is large, and the partition info can be obtained from Hudi's internal metadata table. Note, Key: 'hoodie.metadata.enable' , default: true , isAdvanced: false , description: Enable the internal metadata table whi [...]
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` [...]
---
@@ -1518,27 +1555,27 @@ Configurations used by the Hudi to sync metadata to DataHub.
[**Advanced Configs**](#DataHub-Sync-Configs-advanced-configs)
-| Config Name | Default | Description |
-| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.meta.sync.datahub.emitter.server](#hoodiemetasyncdatahubemitterserver) | (N/A) | Server URL of the DataHub instance.<br />`Config Param: META_SYNC_DATAHUB_EMITTER_SERVER` |
-| [hoodie.meta.sync.datahub.emitter.supplier.class](#hoodiemetasyncdatahubemittersupplierclass) | (N/A) | Pluggable class to supply a DataHub REST emitter to connect to the DataHub instance. This overwrites other emitter configs.<br />`Config Param: META_SYNC_DATAHUB_EMITTER_SUPPLIER_CLASS` |
-| [hoodie.meta.sync.datahub.emitter.token](#hoodiemetasyncdatahubemittertoken) | (N/A) | Auth token to connect to the DataHub instance.<br />`Config Param: META_SYNC_DATAHUB_EMITTER_TOKEN` |
-| [hoodie.datasource.hive_sync.assume_date_partitioning](#hoodiedatasourcehive_syncassume_date_partitioning) | false | Assume partitioning is yyyy/MM/dd<br />`Config Param: META_SYNC_ASSUME_DATE_PARTITION` |
-| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` |
-| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` |
-| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` |
-| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` |
-| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` |
-| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` |
-| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` |
-| [hoodie.meta.sync.datahub.dataplatform.name](#hoodiemetasyncdatahubdataplatformname) | hudi | String used to represent Hudi when creating its corresponding DataPlatform entity within Datahub<br />`Config Param: META_SYNC_DATAHUB_DATAPLATFORM_NAME` |
-| [hoodie.meta.sync.datahub.dataset.env](#hoodiemetasyncdatahubdatasetenv) | DEV | Environment to use when pushing entities to Datahub<br />`Config Param: META_SYNC_DATAHUB_DATASET_ENV` |
-| [hoodie.meta.sync.datahub.dataset.identifier.class](#hoodiemetasyncdatahubdatasetidentifierclass) | org.apache.hudi.sync.datahub.config.HoodieDataHubDatasetIdentifier | Pluggable class to help provide info to identify a DataHub Dataset.<br />`Config Param: META_SYNC_DATAHUB_DATASET_IDENTIFIER_CLASS` |
-| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` |
-| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0.14.0` |
-| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | false | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` |
-| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` |
-| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` |
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.meta.sync.datahub.emitter.server](#hoodiemetasyncdatahubemitterserver) | (N/A) | Server URL of the DataHub instance.<br />`Config Param: META_SYNC_DATAHUB_EMITTER_SERVER` [...]
+| [hoodie.meta.sync.datahub.emitter.supplier.class](#hoodiemetasyncdatahubemittersupplierclass) | (N/A) | Pluggable class to supply a DataHub REST emitter to connect to the DataHub instance. This overwrites other emitter configs.<br />`Config Param: META_SYNC_DATAHUB_EMITTER_SUPPLIER_CLASS` [...]
+| [hoodie.meta.sync.datahub.emitter.token](#hoodiemetasyncdatahubemittertoken) | (N/A) | Auth token to connect to the DataHub instance.<br />`Config Param: META_SYNC_DATAHUB_EMITTER_TOKEN` [...]
+| [hoodie.datasource.hive_sync.base_file_format](#hoodiedatasourcehive_syncbase_file_format) | PARQUET | Base file format for the sync.<br />`Config Param: META_SYNC_BASE_FILE_FORMAT` [...]
+| [hoodie.datasource.hive_sync.database](#hoodiedatasourcehive_syncdatabase) | default | The name of the destination database that we should sync the hudi table to.<br />`Config Param: META_SYNC_DATABASE_NAME` [...]
+| [hoodie.datasource.hive_sync.partition_extractor_class](#hoodiedatasourcehive_syncpartition_extractor_class) | org.apache.hudi.hive.MultiPartKeysValueExtractor | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'.<br />`Config Param: META_SYNC_PARTITION_EXTRACTOR_CLASS` [...]
+| [hoodie.datasource.hive_sync.partition_fields](#hoodiedatasourcehive_syncpartition_fields) | | Field in the table to use for determining hive partition columns.<br />`Config Param: META_SYNC_PARTITION_FIELDS` [...]
+| [hoodie.datasource.hive_sync.table](#hoodiedatasourcehive_synctable) | unknown | The name of the destination table that we should sync the hudi table to.<br />`Config Param: META_SYNC_TABLE_NAME` [...]
+| [hoodie.datasource.meta.sync.base.path](#hoodiedatasourcemetasyncbasepath) | | Base path of the hoodie table to sync<br />`Config Param: META_SYNC_BASE_PATH` [...]
+| [hoodie.datasource.meta_sync.condition.sync](#hoodiedatasourcemeta_syncconditionsync) | false | If true, only sync on conditions like schema change or partition change.<br />`Config Param: META_SYNC_CONDITIONAL_SYNC` [...]
+| [hoodie.meta.sync.datahub.dataplatform.name](#hoodiemetasyncdatahubdataplatformname) | hudi | String used to represent Hudi when creating its corresponding DataPlatform entity within Datahub<br />`Config Param: META_SYNC_DATAHUB_DATAPLATFORM_NAME` [...]
+| [hoodie.meta.sync.datahub.dataset.env](#hoodiemetasyncdatahubdatasetenv) | DEV | Environment to use when pushing entities to Datahub<br />`Config Param: META_SYNC_DATAHUB_DATASET_ENV` [...]
+| [hoodie.meta.sync.datahub.dataset.identifier.class](#hoodiemetasyncdatahubdatasetidentifierclass) | org.apache.hudi.sync.datahub.config.HoodieDataHubDatasetIdentifier | Pluggable class to help provide info to identify a DataHub Dataset.<br />`Config Param: META_SYNC_DATAHUB_DATASET_IDENTIFIER_CLASS` [...]
+| [hoodie.meta.sync.decode_partition](#hoodiemetasyncdecode_partition) | false | If true, meta sync will url-decode the partition path, as it is deemed as url-encoded. Default to false.<br />`Config Param: META_SYNC_DECODE_PARTITION` [...]
+| [hoodie.meta.sync.incremental](#hoodiemetasyncincremental) | true | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost.<br />`Config Param: META_SYNC_INCREMENTAL`<br />`Since Version: 0. [...]
+| [hoodie.meta.sync.metadata_file_listing](#hoodiemetasyncmetadata_file_listing) | true | Enable the internal metadata table for file listing for syncing with metastores<br />`Config Param: META_SYNC_USE_FILE_LISTING_FROM_METADATA` [...]
+| [hoodie.meta.sync.no_partition_metadata](#hoodiemetasyncno_partition_metadata) | false | If true, the partition metadata will not be synced to the metastore. This is useful when the partition metadata is large, and the partition info can be obtained from Hudi's internal metadata table. Note, Key: 'hoodie.metadata.enable' , default: true , isAdvanced: false , description: Enable the internal meta [...]
+| [hoodie.meta.sync.sync_snapshot_with_table_name](#hoodiemetasyncsync_snapshot_with_table_name) | true | sync meta info to origin table if enable<br />`Config Param: META_SYNC_SNAPSHOT_WITH_TABLE_NAME`<br />`Since Version: 0.14.0` [...]
+| [hoodie.meta_sync.spark.version](#hoodiemeta_syncsparkversion) | | The spark version used when syncing with a metastore.<br />`Config Param: META_SYNC_SPARK_VERSION` [...]
---
## Metrics Configs {#METRICS}
@@ -1693,11 +1730,11 @@ Payload related configs, that can be leveraged to control merges based on specif
[**Advanced Configs**](#Payload-Configurations-advanced-configs)
-| Config Name | Default | Description |
-| ---------------------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | org.apache.hudi.common.model.DefaultHoodieRecordPayload | This needs to be same as class used during insert/upserts. Just like writing, compaction also uses the record payload class to merge records in the log against each other, merge again with the base file and produce the final record to be written after compaction.<br />`Config Param: PAYLOAD_CLASS_NAME` |
-| [hoodie.payload.event.time.field](#hoodiepayloadeventtimefield) | ts | Table column/field name to derive timestamp associated with the records. This canbe useful for e.g, determining the freshness of the table.<br />`Config Param: EVENT_TIME_FIELD` |
-| [hoodie.payload.ordering.field](#hoodiepayloadorderingfield) | ts | Table column/field name to order records that have the same key, before merging and writing to storage.<br />`Config Param: ORDERING_FIELD` |
+| Config Name | Default | Description |
+| ---------------------------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.compaction.payload.class](#hoodiecompactionpayloadclass) | org.apache.hudi.common.model.DefaultHoodieRecordPayload | This needs to be same as class used during insert/upserts. Just like writing, compaction also uses the record payload class to merge records in the log against each other, merge again with the base file and produce the final record to be written after compaction.<br />`Config Param: PAYLOAD_CLASS_NAME` |
+| [hoodie.payload.event.time.field](#hoodiepayloadeventtimefield) | ts | Table column/field name to derive timestamp associated with the records. This canbe useful for e.g, determining the freshness of the table.<br />`Config Param: EVENT_TIME_FIELD` |
+| [hoodie.payload.ordering.field](#hoodiepayloadorderingfield) | ts | Table column/field name to order records that have the same key, before merging and writing to storage.<br />`Config Param: ORDERING_FIELD` |
---
## Kafka Connect Configs {#KAFKA_CONNECT}
@@ -1738,7 +1775,7 @@ Configurations specific to Amazon Web Services.
### Amazon Web Services Configs {#Amazon-Web-Services-Configs}
-Amazon Web Services configurations to access resources like Amazon DynamoDB (for locks), Amazon CloudWatch (metrics).
+Amazon Web Services configurations to access resources like Amazon DynamoDB (for locks), Amazon CloudWatch (metrics) and Amazon Glue (metadata).
@@ -1819,22 +1856,24 @@ Configs that are common during ingestion across different cloud stores
[**Advanced Configs**](#Cloud-Source-Configs-advanced-configs)
-| Config Name | Default | Description [...]
-| ------------------------------------------------------------------------------------------------------------------------------------ | -------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| [hoodie.streamer.source.cloud.data.datasource.options](#hoodiestreamersourceclouddatadatasourceoptions) | (N/A) | A JSON string passed to the Spark DataFrameReader while loading the dataset. Example: `hoodie.streamer.gcp.spark.datasource.options={"header":"true","encoding":"UTF-8"}` <br />`Config Param: SPARK_DATASOURCE_OPTIONS` [...]
-| [hoodie.streamer.source.cloud.data.ignore.relpath.prefix](#hoodiestreamersourceclouddataignorerelpathprefix) | (N/A) | Ignore objects in the bucket whose relative path starts this prefix<br />`Config Param: IGNORE_RELATIVE_PATH_PREFIX` [...]
-| [hoodie.streamer.source.cloud.data.ignore.relpath.substring](#hoodiestreamersourceclouddataignorerelpathsubstring) | (N/A) | Ignore objects in the bucket whose relative path contains this substring<br />`Config Param: IGNORE_RELATIVE_PATH_SUBSTR` [...]
-| [hoodie.streamer.source.cloud.data.partition.fields.from.path](#hoodiestreamersourceclouddatapartitionfieldsfrompath) | (N/A) | A comma delimited list of path-based partition fields in the source file structure.<br />`Config Param: PATH_BASED_PARTITION_FIELDS`<br />`Since Version: 0.14.0` [...]
-| [hoodie.streamer.source.cloud.data.partition.max.size](#hoodiestreamersourceclouddatapartitionmaxsize) | (N/A) | specify this value in bytes, to coalesce partitions of source dataset not greater than specified limit<br />`Config Param: SOURCE_MAX_BYTES_PER_PARTITION`<br />`Since Version: 0.14.1` [...]
-| [hoodie.streamer.source.cloud.data.select.file.extension](#hoodiestreamersourceclouddataselectfileextension) | (N/A) | Only match files with this extension. By default, this is the same as hoodie.streamer.source.hoodieincr.file.format<br />`Config Param: CLOUD_DATAFILE_EXTENSION` [...]
-| [hoodie.streamer.source.cloud.data.select.relpath.prefix](#hoodiestreamersourceclouddataselectrelpathprefix) | (N/A) | Only selects objects in the bucket whose relative path starts with this prefix<br />`Config Param: SELECT_RELATIVE_PATH_PREFIX` [...]
-| [hoodie.streamer.source.cloud.data.check.file.exists](#hoodiestreamersourceclouddatacheckfileexists) | false | If true, checks whether file exists before attempting to pull it<br />`Config Param: ENABLE_EXISTS_CHECK` [...]
-| [hoodie.streamer.source.cloud.data.datafile.format](#hoodiestreamersourceclouddatadatafileformat) | parquet | Format of the data file. By default, this will be the same as hoodie.streamer.source.hoodieincr.file.format<br />`Config Param: DATAFILE_FORMAT` [...]
-| [hoodie.streamer.source.cloud.data.reader.comma.separated.path.format](#hoodiestreamersourceclouddatareadercommaseparatedpathformat) | false | Boolean value for specifying path format in load args of spark.read.format("..").load("a.xml,b.xml,c.xml"), * set true if path format needs to be comma separated string value, if false it's passed as array of strings like spark.read.format("..").load(new String[]\{a.xml,b.xml,c.xml})<br />`Config Param: SPARK_DATASOURCE_READER_COMMA_SEPARATED [...]
-| [hoodie.streamer.source.cloud.meta.ack](#hoodiestreamersourcecloudmetaack) | true | Whether to acknowledge Metadata messages during Cloud Ingestion or not. This is useful during dev and testing. In Prod this should always be true. In case of Cloud Pubsub, not acknowledging means Pubsub will keep redelivering the same messages.<br />`Config Param: ACK_MESSAGES` [...]
-| [hoodie.streamer.source.cloud.meta.batch.size](#hoodiestreamersourcecloudmetabatchsize) | 10 | Number of metadata messages to pull in one API call to the cloud events queue. Multiple API calls with this batch size are sent to cloud events queue, until we consume `hoodie.streamer.source.cloud.meta.max.num.messages.per.sync` from the queue or `hoodie.streamer.source.cloud.meta.max.fetch.time.per.sync.ms` amount of time has passed or queu [...]
-| [hoodie.streamer.source.cloud.meta.max.fetch.time.per.sync.secs](#hoodiestreamersourcecloudmetamaxfetchtimepersyncsecs) | 60 | Max time in secs to consume `hoodie.streamer.source.cloud.meta.max.num.messages.per.sync` messages from cloud queue. Cloud event queues like SQS, PubSub can return empty responses even when messages are available the queue, this config ensures we don't wait forever to consume MAX_MESSAGES_CONF messages, but time out and move on further.<br /> [...]
-| [hoodie.streamer.source.cloud.meta.max.num.messages.per.sync](#hoodiestreamersourcecloudmetamaxnummessagespersync) | 1000 | Maximum number of messages to consume per sync round. Multiple rounds of `hoodie.streamer.source.cloud.meta.batch.size` could be invoked to reach max messages as configured by this config<br />`Config Param: MAX_NUM_MESSAGES_PER_SYNC`<br />`Since Version: 0.14.1` [...]
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------------------------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.streamer.source.cloud.data.datasource.options](#hoodiestreamersourceclouddatadatasourceoptions) | (N/A) | A JSON string passed to the Spark DataFrameReader while loading the dataset. Example: `hoodie.streamer.gcp.spark.datasource.options={"header":"true","encoding":"UTF-8"}` <br />`Config Param: SPARK_DATASOURCE_OPTIONS` [...]
+| [hoodie.streamer.source.cloud.data.ignore.relpath.prefix](#hoodiestreamersourceclouddataignorerelpathprefix) | (N/A) | Ignore objects in the bucket whose relative path starts this prefix<br />`Config Param: IGNORE_RELATIVE_PATH_PREFIX` [...]
+| [hoodie.streamer.source.cloud.data.ignore.relpath.substring](#hoodiestreamersourceclouddataignorerelpathsubstring) | (N/A) | Ignore objects in the bucket whose relative path contains this substring<br />`Config Param: IGNORE_RELATIVE_PATH_SUBSTR` [...]
+| [hoodie.streamer.source.cloud.data.partition.fields.from.path](#hoodiestreamersourceclouddatapartitionfieldsfrompath) | (N/A) | A comma delimited list of path-based partition fields in the source file structure.<br />`Config Param: PATH_BASED_PARTITION_FIELDS`<br />`Since Version: 0.14.0` [...]
+| [hoodie.streamer.source.cloud.data.partition.max.size](#hoodiestreamersourceclouddatapartitionmaxsize) | (N/A) | specify this value in bytes, to coalesce partitions of source dataset not greater than specified limit<br />`Config Param: SOURCE_MAX_BYTES_PER_PARTITION`<br />`Since Version: 0.14.1` [...]
+| [hoodie.streamer.source.cloud.data.select.file.extension](#hoodiestreamersourceclouddataselectfileextension) | (N/A) | Only match files with this extension. By default, this is the same as hoodie.streamer.source.hoodieincr.file.format<br />`Config Param: CLOUD_DATAFILE_EXTENSION` [...]
+| [hoodie.streamer.source.cloud.data.select.relative.path.regex](#hoodiestreamersourceclouddataselectrelativepathregex) | (N/A) | Only selects objects in the bucket whose relative path matches this regex. For example: When hoodie.streamer.source.cloud.data.select.relpath.prefix is set to /path/prefix, and the hoodie.streamer.source.cloud.data.select.relative.path.regex is regex/files[0-9]+, only files located in the /path/prefix/regex directory that match the pattern (e [...]
+| [hoodie.streamer.source.cloud.data.select.relpath.prefix](#hoodiestreamersourceclouddataselectrelpathprefix) | (N/A) | Only selects objects in the bucket whose relative path starts with this prefix<br />`Config Param: SELECT_RELATIVE_PATH_PREFIX` [...]
+| [hoodie.streamer.source.cloud.data.check.file.exists](#hoodiestreamersourceclouddatacheckfileexists) | false | If true, checks whether file exists before attempting to pull it<br />`Config Param: ENABLE_EXISTS_CHECK` [...]
+| [hoodie.streamer.source.cloud.data.datafile.format](#hoodiestreamersourceclouddatadatafileformat) | parquet | Format of the data file. By default, this will be the same as hoodie.streamer.source.hoodieincr.file.format<br />`Config Param: DATAFILE_FORMAT` [...]
+| [hoodie.streamer.source.cloud.data.reader.coalesce.aliases](#hoodiestreamersourceclouddatareadercoalescealiases) | true | Boolean value to allow coalesce alias columns with actual columns while reading from source<br />`Config Param: SPARK_DATASOURCE_READER_COALESCE_ALIAS_COLUMNS`<br />`Since Version: 1.0.0` [...]
+| [hoodie.streamer.source.cloud.data.reader.comma.separated.path.format](#hoodiestreamersourceclouddatareadercommaseparatedpathformat) | false | Boolean value for specifying path format in load args of `spark.read.format("..").load("a.xml,b.xml,c.xml")`, * set true if path format needs to be comma separated string value, if false it's passed as array of strings like * `spark.read.format("..").load(new String[]{a.xml,b.xml,c.xml})`<br />`Config Param: SPARK_DATASOURCE_READER_COMM [...]
+| [hoodie.streamer.source.cloud.meta.ack](#hoodiestreamersourcecloudmetaack) | true | Whether to acknowledge Metadata messages during Cloud Ingestion or not. This is useful during dev and testing. In Prod this should always be true. In case of Cloud Pubsub, not acknowledging means Pubsub will keep redelivering the same messages.<br />`Config Param: ACK_MESSAGES` [...]
+| [hoodie.streamer.source.cloud.meta.batch.size](#hoodiestreamersourcecloudmetabatchsize) | 10 | Number of metadata messages to pull in one API call to the cloud events queue. Multiple API calls with this batch size are sent to cloud events queue, until we consume hoodie.streamer.source.cloud.meta.max.num.messages.per.syncfrom the queue or hoodie.streamer.source.cloud.meta.max.fetch.time.per.sync.ms amount of time has passed or queue is [...]
+| [hoodie.streamer.source.cloud.meta.max.fetch.time.per.sync.secs](#hoodiestreamersourcecloudmetamaxfetchtimepersyncsecs) | 60 | Max time in secs to consume hoodie.streamer.source.cloud.meta.max.num.messages.per.sync messages from cloud queue. Cloud event queues like SQS, PubSub can return empty responses even when messages are available the queue, this config ensures we don't wait forever to consume MAX_MESSAGES_CONF messages, but time out and move on further.<br />`C [...]
+| [hoodie.streamer.source.cloud.meta.max.num.messages.per.sync](#hoodiestreamersourcecloudmetamaxnummessagespersync) | 1000 | Maximum number of messages to consume per sync round. Multiple rounds of hoodie.streamer.source.cloud.meta.batch.size could be invoked to reach max messages as configured by this config<br />`Config Param: MAX_NUM_MESSAGES_PER_SYNC`<br />`Since Version: 0.14.1` [...]
---
@@ -1992,17 +2031,21 @@ Configurations controlling the behavior of Kafka source in Hudi Streamer.
[**Advanced Configs**](#Kafka-Source-Configs-advanced-configs)
-| Config Name | Default | Description [...]
-| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
-| [hoodie.streamer.source.kafka.value.deserializer.schema](#hoodiestreamersourcekafkavaluedeserializerschema) | (N/A) | Schema to deserialize the records.<br />`Config Param: KAFKA_VALUE_DESERIALIZER_SCHEMA` [...]
-| [auto.offset.reset](#autooffsetreset) | LATEST | Kafka consumer strategy for reading data.<br />`Config Param: KAFKA_AUTO_OFFSET_RESET` [...]
-| [hoodie.streamer.kafka.source.maxEvents](#hoodiestreamerkafkasourcemaxEvents) | 5000000 | Maximum number of records obtained in each batch.<br />`Config Param: MAX_EVENTS_FROM_KAFKA_SOURCE` [...]
-| [hoodie.streamer.source.kafka.checkpoint.type](#hoodiestreamersourcekafkacheckpointtype) | string | Kafka checkpoint type. Value must be one of the following: string, timestamp, single_offset. Default type is string. For type string, checkpoint should be provided as: topicName,0:offset0,1:offset1,2:offset2. For type timestamp, checkpoint should be provided as long value of desired timestamp. For type single_offset, we as [...]
-| [hoodie.streamer.source.kafka.enable.commit.offset](#hoodiestreamersourcekafkaenablecommitoffset) | false | Automatically submits offset to kafka.<br />`Config Param: ENABLE_KAFKA_COMMIT_OFFSET` [...]
-| [hoodie.streamer.source.kafka.enable.failOnDataLoss](#hoodiestreamersourcekafkaenablefailOnDataLoss) | false | Fail when checkpoint goes out of bounds instead of seeking to earliest offsets.<br />`Config Param: ENABLE_FAIL_ON_DATA_LOSS` [...]
-| [hoodie.streamer.source.kafka.fetch_partition.time.out](#hoodiestreamersourcekafkafetch_partitiontimeout) | 300000 | Time out for fetching partitions. 5min by default<br />`Config Param: KAFKA_FETCH_PARTITION_TIME_OUT` [...]
-| [hoodie.streamer.source.kafka.minPartitions](#hoodiestreamersourcekafkaminPartitions) | 0 | Desired minimum number of partitions to read from Kafka. By default, Hudi has a 1-1 mapping of topicPartitions to Hudi partitions consuming from Kafka. If set this option to a value greater than topicPartitions, Hudi will divvy up large Kafka partitions to smaller pieces. Please note that this configuration is like a hint: [...]
-| [hoodie.streamer.source.kafka.value.deserializer.class](#hoodiestreamersourcekafkavaluedeserializerclass) | io.confluent.kafka.serializers.KafkaAvroDeserializer | This class is used by kafka client to deserialize the records.<br />`Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.9.0` [...]
+| Config Name | Default | Description [...]
+| ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| [hoodie.streamer.source.kafka.value.deserializer.schema](#hoodiestreamersourcekafkavaluedeserializerschema) | (N/A) | Schema to deserialize the records.<br />`Config Param: KAFKA_VALUE_DESERIALIZER_SCHEMA` [...]
+| [auto.offset.reset](#autooffsetreset) | LATEST | Kafka consumer strategy for reading data.<br />`Config Param: KAFKA_AUTO_OFFSET_RESET` [...]
+| [hoodie.streamer.kafka.source.maxEvents](#hoodiestreamerkafkasourcemaxEvents) | 5000000 | Maximum number of records obtained in each batch.<br />`Config Param: MAX_EVENTS_FROM_KAFKA_SOURCE` [...]
+| [hoodie.streamer.source.kafka.checkpoint.type](#hoodiestreamersourcekafkacheckpointtype) | string | Kafka checkpoint type. Value must be one of the following: string, timestamp, single_offset. Default type is string. For type string, checkpoint should be provided as: topicName,0:offset0,1:offset1,2:offset2. For type timestamp, checkpoint should be provided as long value of desired timestamp. For type single_offset, we a [...]
+| [hoodie.streamer.source.kafka.enable.commit.offset](#hoodiestreamersourcekafkaenablecommitoffset) | false | Automatically submits offset to kafka.<br />`Config Param: ENABLE_KAFKA_COMMIT_OFFSET` [...]
+| [hoodie.streamer.source.kafka.enable.failOnDataLoss](#hoodiestreamersourcekafkaenablefailOnDataLoss) | false | Fail when checkpoint goes out of bounds instead of seeking to earliest offsets.<br />`Config Param: ENABLE_FAIL_ON_DATA_LOSS` [...]
+| [hoodie.streamer.source.kafka.fetch_partition.time.out](#hoodiestreamersourcekafkafetch_partitiontimeout) | 300000 | Time out for fetching partitions. 5min by default<br />`Config Param: KAFKA_FETCH_PARTITION_TIME_OUT` [...]
+| [hoodie.streamer.source.kafka.minPartitions](#hoodiestreamersourcekafkaminPartitions) | 0 | Desired minimum number of partitions to read from Kafka. By default, Hudi has a 1-1 mapping of topicPartitions to Hudi partitions consuming from Kafka. If set this option to a value greater than topicPartitions, Hudi will divvy up large Kafka partitions to smaller pieces. Please note that this configuration is like a hint [...]
+| [hoodie.streamer.source.kafka.retry.exceptions](#hoodiestreamersourcekafkaretryexceptions) | | The class name of the Exception that needs to be retried, separated by commas. Default is empty which means retry all the IOException and RuntimeException from KafkaConsumer<br />`Config Param: RETRY_EXCEPTIONS`<br />`Since Version: 1.1.0` [...]
+| [hoodie.streamer.source.kafka.retry.initial_interval_ms](#hoodiestreamersourcekafkaretryinitial_interval_ms) | 100 | Amount of time (in ms) to wait, before retry to do operations on KafkaConsumer.<br />`Config Param: INITIAL_RETRY_INTERVAL_MS`<br />`Since Version: 1.1.0` [...]
+| [hoodie.streamer.source.kafka.retry.max_count](#hoodiestreamersourcekafkaretrymax_count) | 4 | Maximum number of retry actions to perform, with exponential backoff.<br />`Config Param: MAX_RETRY_COUNT`<br />`Since Version: 1.1.0` [...]
+| [hoodie.streamer.source.kafka.retry.max_interval_ms](#hoodiestreamersourcekafkaretrymax_interval_ms) | 2000 | Maximum amount of time (in ms), to wait for next retry.<br />`Config Param: MAX_RETRY_INTERVAL_MS`<br />`Since Version: 1.1.0` [...]
+| [hoodie.streamer.source.kafka.value.deserializer.class](#hoodiestreamersourcekafkavaluedeserializerclass) | io.confluent.kafka.serializers.KafkaAvroDeserializer | This class is used by kafka client to deserialize the records.<br />`Config Param: KAFKA_AVRO_VALUE_DESERIALIZER_CLASS`<br />`Since Version: 0.9.0` [...]
---
@@ -2052,14 +2095,14 @@ Configurations controlling the behavior of incremental pulling from S3 events me
[**Advanced Configs**](#S3-Event-based-Hudi-Incremental-Source-Configs-advanced-configs)
-| Config Name | Default | Description |
-| ----------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [hoodie.streamer.source.s3incr.ignore.key.prefix](#hoodiestreamersources3incrignorekeyprefix) | (N/A) | Control whether to ignore the s3 objects starting with this prefix<br />`Config Param: S3_IGNORE_KEY_PREFIX` |
-| [hoodie.streamer.source.s3incr.ignore.key.substring](#hoodiestreamersources3incrignorekeysubstring) | (N/A) | Control whether to ignore the s3 objects with this substring<br />`Config Param: S3_IGNORE_KEY_SUBSTRING` |
-| [hoodie.streamer.source.s3incr.key.prefix](#hoodiestreamersources3incrkeyprefix) | (N/A) | Control whether to filter the s3 objects starting with this prefix<br />`Config Param: S3_KEY_PREFIX` |
-| [hoodie.streamer.source.s3incr.spark.datasource.options](#hoodiestreamersources3incrsparkdatasourceoptions) | (N/A) | JSON string, passed to the reader while loading dataset. Example Hudi Streamer conf --hoodie-conf `hoodie.streamer.source.s3incr.spark.datasource.options={"header":"true","encoding":"UTF-8"}`<br />`Config Param: SPARK_DATASOURCE_OPTIONS` |
-| [hoodie.streamer.source.s3incr.check.file.exists](#hoodiestreamersources3incrcheckfileexists) | false | Control whether we do existence check for files before consuming them<br />`Config Param: S3_INCR_ENABLE_EXISTS_CHECK` |
-| [hoodie.streamer.source.s3incr.fs.prefix](#hoodiestreamersources3incrfsprefix) | s3 | The file system prefix.<br />`Config Param: S3_FS_PREFIX` |
+| Config Name | Default | Description |
+| ----------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.source.s3incr.ignore.key.prefix](#hoodiestreamersources3incrignorekeyprefix) | (N/A) | Control whether to ignore the s3 objects starting with this prefix<br />`Config Param: S3_IGNORE_KEY_PREFIX` |
+| [hoodie.streamer.source.s3incr.ignore.key.substring](#hoodiestreamersources3incrignorekeysubstring) | (N/A) | Control whether to ignore the s3 objects with this substring<br />`Config Param: S3_IGNORE_KEY_SUBSTRING` |
+| [hoodie.streamer.source.s3incr.key.prefix](#hoodiestreamersources3incrkeyprefix) | (N/A) | Control whether to filter the s3 objects starting with this prefix<br />`Config Param: S3_KEY_PREFIX` |
+| [hoodie.streamer.source.s3incr.spark.datasource.options](#hoodiestreamersources3incrsparkdatasourceoptions) | (N/A) | Json string, passed to the reader while loading dataset. Example Hudi Streamer conf `--hoodie-conf hoodie.streamer.source.s3incr.spark.datasource.options={"header":"true","encoding":"UTF-8"}`<br />`Config Param: SPARK_DATASOURCE_OPTIONS` |
+| [hoodie.streamer.source.s3incr.check.file.exists](#hoodiestreamersources3incrcheckfileexists) | false | Control whether we do existence check for files before consuming them<br />`Config Param: S3_INCR_ENABLE_EXISTS_CHECK` |
+| [hoodie.streamer.source.s3incr.fs.prefix](#hoodiestreamersources3incrfsprefix) | s3 | The file system prefix.<br />`Config Param: S3_FS_PREFIX` |
---
@@ -2144,14 +2187,13 @@ Configurations that control the schema provider for Hudi Streamer.
[**Advanced Configs**](#Hudi-Streamer-Schema-Provider-Configs-advanced-configs)
-| Config Name | Default | Description |
-| ------------------------------------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------- |
-| [hoodie.streamer.schemaprovider.registry.baseUrl](#hoodiestreamerschemaproviderregistrybaseUrl) | (N/A) | The base URL of the schema registry.<br />`Config Param: SCHEMA_REGISTRY_BASE_URL` |
-| [hoodie.streamer.schemaprovider.registry.schemaconverter](#hoodiestreamerschemaproviderregistryschemaconverter) | (N/A) | The class name of the custom schema converter to use.<br />`Config Param: SCHEMA_CONVERTER` |
-| [hoodie.streamer.schemaprovider.registry.sourceUrlSuffix](#hoodiestreamerschemaproviderregistrysourceUrlSuffix) | (N/A) | The source URL suffix.<br />`Config Param: SCHEMA_REGISTRY_SOURCE_URL_SUFFIX` |
-| [hoodie.streamer.schemaprovider.registry.targetUrlSuffix](#hoodiestreamerschemaproviderregistrytargetUrlSuffix) | (N/A) | The target URL suffix.<br />`Config Param: SCHEMA_REGISTRY_TARGET_URL_SUFFIX` |
-| [hoodie.streamer.schemaprovider.registry.urlSuffix](#hoodiestreamerschemaproviderregistryurlSuffix) | (N/A) | The suffix of the URL for the schema registry.<br />`Config Param: SCHEMA_REGISTRY_URL_SUFFIX` |
-| [hoodie.streamer.schemaprovider.spark_avro_post_processor.enable](#hoodiestreamerschemaproviderspark_avro_post_processorenable) | true | Whether to enable Spark Avro post processor.<br />`Config Param: SPARK_AVRO_POST_PROCESSOR_ENABLE` |
+| Config Name | Default | Description |
+| --------------------------------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------- |
+| [hoodie.streamer.schemaprovider.registry.baseUrl](#hoodiestreamerschemaproviderregistrybaseUrl) | (N/A) | The base URL of the schema registry.<br />`Config Param: SCHEMA_REGISTRY_BASE_URL` |
+| [hoodie.streamer.schemaprovider.registry.schemaconverter](#hoodiestreamerschemaproviderregistryschemaconverter) | (N/A) | The class name of the custom schema converter to use.<br />`Config Param: SCHEMA_CONVERTER` |
+| [hoodie.streamer.schemaprovider.registry.sourceUrlSuffix](#hoodiestreamerschemaproviderregistrysourceUrlSuffix) | (N/A) | The source URL suffix.<br />`Config Param: SCHEMA_REGISTRY_SOURCE_URL_SUFFIX` |
+| [hoodie.streamer.schemaprovider.registry.targetUrlSuffix](#hoodiestreamerschemaproviderregistrytargetUrlSuffix) | (N/A) | The target URL suffix.<br />`Config Param: SCHEMA_REGISTRY_TARGET_URL_SUFFIX` |
+| [hoodie.streamer.schemaprovider.registry.urlSuffix](#hoodiestreamerschemaproviderregistryurlSuffix) | (N/A) | The suffix of the URL for the schema registry.<br />`Config Param: SCHEMA_REGISTRY_URL_SUFFIX` |
---