You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-agent (via GitHub)" <gi...@apache.org> on 2026/05/29 07:46:34 UTC
Re: [PR] docs: Add release 1.2.0 documentation [hudi]
hudi-agent commented on code in PR #18880:
URL: https://github.com/apache/hudi/pull/18880#discussion_r3322900852
##########
website/versioned_docs/version-1.2.0/gcp_bigquery.md:
##########
@@ -0,0 +1,102 @@
+---
+title: Google BigQuery
+keywords: [ hudi, gcp, bigquery ]
+summary: Introduce BigQuery integration in Hudi.
+---
+
+Hudi tables can be queried from [Google Cloud BigQuery](https://cloud.google.com/bigquery) as external tables. As of
+now, the Hudi-BigQuery integration only works for hive-style partitioned Copy-On-Write and Read-Optimized Merge-On-Read tables.
+
+## Sync Modes
+### Manifest File
+As of version 0.14.0, the `BigQuerySyncTool` supports syncing table to BigQuery using [manifests](https://cloud.google.com/blog/products/data-analytics/bigquery-manifest-file-support-for-open-table-format-queries). On the first run, the tool will create a manifest file representing the current base files in the table and a table in BigQuery based on the provided configurations. The tool produces a new manifest file on each subsequent run and will update the schema of the table in BigQuery if the schema changes in your Hudi table.
+#### Benefits of using the new manifest approach:
+<ol>
+ <li>Only the files in the manifest can be scanned leading to less cost and better performance for your queries</li>
+ <li>The schema is now synced from the Hudi commit metadata allowing for proper schema evolution</li>
+ <li>Lists no longer have unnecessary nesting when querying in BigQuery as list inference is enabled by default</li>
+ <li>Partition column no longer needs to be dropped from the files due to new schema handling improvements</li>
+</ol>
+
+To enable this feature, set `hoodie.gcp.bigquery.sync.use_bq_manifest_file` to true.
+
+### View Over Files (Legacy)
+This is the current default behavior to preserve compatibility as users upgrade to 0.14.0 and beyond.
+After run, the sync tool will create 2 tables and 1 view in the target dataset in BigQuery. The tables and the view
+share the same name prefix, which is taken from the Hudi table name. Query the view for the same results as querying the
+Copy-on-Write Hudi table.
+**NOTE:** The view can scan all of the parquet files under your table's base path so it is recommended to upgrade to the manifest based approach for improved cost and performance.
+
+## Configurations
+
+Hudi uses `org.apache.hudi.gcp.bigquery.BigQuerySyncTool` to sync tables. It works with Hudi Streamer via
+setting sync tool class. A few BigQuery-specific configurations are required.
+
+| Config | Notes |
+|:---------------------------------------------|:----------------------------------------------------------------------------------------------------------------|
+| `hoodie.gcp.bigquery.sync.project_id` | The target Google Cloud project |
+| `hoodie.gcp.bigquery.sync.dataset_name` | BigQuery dataset name; create before running the sync tool |
+| `hoodie.gcp.bigquery.sync.dataset_location` | Region info of the dataset; same as the GCS bucket that stores the Hudi table |
+| `hoodie.gcp.bigquery.sync.source_uri` | A wildcard path pattern pointing to the first level partition; partition key can be specified or auto-inferred. Only required for partitioned tables |
+| `hoodie.gcp.bigquery.sync.source_uri_prefix` | The common prefix of the `source_uri`, usually it's the path to the Hudi table, trailing slash does not matter. |
+| `hoodie.gcp.bigquery.sync.base_path` | The usual basepath config for Hudi table. |
+| `hoodie.gcp.bigquery.sync.use_bq_manifest_file` | Set to true to enable the manifest based sync |
+| `hoodie.gcp.bigquery.sync.require_partition_filter` | Introduced in Hudi version 0.14.1, this configuration accepts a BOOLEAN value, with the default being false. When enabled (set to true), you must create a partition filter (a WHERE clause) for all queries, targeting the partitioning column of a partitioned table. Queries lacking such a filter will result in an error. |
+
+
+Refer to `org.apache.hudi.gcp.bigquery.BigQuerySyncConfig` for the complete configuration list.
+### Partition Handling
+In addition to the BigQuery-specific configs, you will need to use hive style partitioning for partition pruning in BigQuery. On top of that, the value in partition path will be the value returned for that field in your query. For example if you partition on a time-millis field, `time`, with an output format of `time=yyyy-MM-dd`, the query will return `time` values with day level granularity instead of the original milliseconds so keep this in mind while setting up your tables.
+
+```
+hoodie.datasource.write.hive_style_partitioning = 'true'
+```
+
+For the view based sync you must also specify the following configurations:
+```
+hoodie.datasource.write.drop.partition.columns = 'true'
+hoodie.partition.metafile.use.base.format = 'true'
+```
+
+## Example
+
+Below shows an example for running `BigQuerySyncTool` with Hudi Streamer.
+
+```shell
+spark-submit --master yarn \
+--packages com.google.cloud:google-cloud-bigquery:2.10.4 \
+--jars "/opt/hudi-gcp-bundle-0.13.0.jar,/opt/hudi-utilities-slim-bundle_2.12-1.2.0.jar,/opt/hudi-spark3.5-bundle_2.12-1.2.0.jar" \
Review Comment:
π€ The `hudi-gcp-bundle-0.13.0.jar` reference appears to have been missed by the version bump β the surrounding utilities/spark bundles in the same `--jars` argument were updated to `1.2.0`. Since the PR description explicitly calls out bumping hardcoded Maven/bundle examples, this should likely be `hudi-gcp-bundle-1.2.0.jar` for consistency with the rest of the 1.2.0 docs.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/flink_tuning.md:
##########
@@ -0,0 +1,164 @@
+---
+title: Flink Tuning Guide
+toc: true
+---
+
+## Global Configurations
+When using Flink, you can set some global configurations in `$FLINK_HOME/conf/flink-conf.yaml`
+
+### Parallelism
+
+| Option Name | Default | Type | Description |
+| ----------- | ------- | ------- | ------- |
+| `taskmanager.numberOfTaskSlots` | `1` | `Integer` | The number of parallel operator or user function instances that a single TaskManager can run. We recommend setting this value > 4, and the actual value needs to be set according to the amount of data |
+| `parallelism.default` | `1` | `Integer` | The default parallelism used when no parallelism is specified anywhere (default: 1). For example, If the value of [`write.bucket_assign.tasks`](#parallelism-1) is not set, this value will be used |
+
+### Memory
+
+| Option Name | Default | Type | Description |
+| ----------- | ------- | ------- | ------- |
+| `jobmanager.memory.process.size` | `(none)` | `MemorySize` | Total Process Memory size for the JobManager. This includes all the memory that a JobManager JVM process consumes, consisting of Total Flink Memory, JVM Metaspace, and JVM Overhead |
+| `taskmanager.memory.task.heap.size` | `(none)` | `MemorySize` | Task Heap Memory size for TaskExecutors. This is the size of JVM heap memory reserved for write cache |
+| `taskmanager.memory.managed.size` | `(none)` | `MemorySize` | Managed Memory size for TaskExecutors. This is the size of off-heap memory managed by the memory manager, reserved for sorting and RocksDB state backend. If you choose RocksDB as the state backend, you need to set this memory |
+
+### Checkpoint
+
+| Option Name | Default | Type | Description |
+| ----------- | ------- | ------- | ------- |
+| `execution.checkpointing.interval` | `(none)` | `Duration` | Setting this value as `execution.checkpointing.interval = 150000ms`, 150000ms = 2.5min. Configuring this parameter is equivalent to enabling the checkpoint |
+| `state.backend` | `(none)` | `String` | The state backend to be used to store state. We recommend setting store state as `rocksdb` : `state.backend: rocksdb` |
+| `state.backend.rocksdb.localdir` | `(none)` | `String` | The local directory (on the TaskManager) where RocksDB puts its files |
+| `state.checkpoints.dir` | `(none)` | `String` | The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem. The storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers), like hdfs and oss path |
+| `state.backend.incremental` | `false` | `Boolean` | Option whether the state backend should create incremental checkpoints, if possible. For an incremental checkpoint, only a diff from the previous checkpoint is stored, rather than the complete checkpoint state. If store state is setting as `rocksdb`, recommending to turn on |
+
+## Table Options
+
+Flink SQL jobs can be configured through options in the `WITH` clause.
+The actual datasource level configs are listed below.
+
+### Memory
+
+:::note
+When optimizing memory, we need to pay attention to the memory configuration
+and the number of taskManagers, parallelism of write tasks (write.tasks : 4) first. After confirm each write task to be
+allocated with enough memory, we can try to set these memory options.
+:::
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.task.max.size` | Maximum memory in MB for a write task, when the threshold hits, it flushes the max size data bucket to avoid OOM. Default `1024MB` | `1024D` | The memory reserved for write buffer is `write.task.max.size` - `compaction.max_memory`. When total buffer of write tasks reach the threshold, the largest buffer in the memory will be flushed |
+| `write.batch.size` | In order to improve the efficiency of writing, Flink write task will cache data in buffer according to the write bucket until the memory reaches the threshold. When reached threshold, the data buffer would be flushed out. Default `64MB` | `64D` | Recommend to use the default settings |
+| `write.log_block.size` | The log writer of Hudi will not flush the data immediately after receiving data. The writer flush data to the disk in the unit of `LogBlock`. Before `LogBlock` reached threshold, records will be buffered in the writer in form of serialized bytes. Default `128MB` | `128` | Recommend to use the default settings |
+| `write.merge.max_memory` | If write type is `COPY_ON_WRITE`, Hudi will merge the incremental data and base file data. The incremental data will be cached and spilled to disk. this threshold controls the max heap size that can be used. Default `100MB` | `100` | Recommend to use the default settings |
+| `compaction.max_memory` | Same as `write.merge.max_memory`, but occurs during compaction. Default `100MB` | `100` | If it is online compaction, it can be turned up when resources are sufficient, such as setting as `1024MB` |
+
+### Parallelism
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.tasks` | The parallelism of writer tasks. Each write task writes 1 to `N` buckets in sequence. Default `4` | `4` | Increases the parallelism has no effect on the number of small files |
+| `write.bucket_assign.tasks` | The parallelism of bucket assigner operators. No default value, using Flink `parallelism.default` | [`parallelism.default`](#parallelism) | Increases the parallelism also increases the number of buckets, thus the number of small files (small buckets) |
+| `write.index_boostrap.tasks` | The parallelism of index bootstrap. Increasing parallelism can speed up the efficiency of the bootstrap stage. The bootstrap stage will block checkpointing. Therefore, it is necessary to set more checkpoint failure tolerance times. Default using Flink `parallelism.default` | [`parallelism.default`](#parallelism) | It only take effect when `index.bootsrap.enabled` is `true` |
+| `read.tasks` | The parallelism of read operators (batch and stream). Default `4` | `4` | |
+| `compaction.tasks` | The parallelism of online compaction. Default `4` | `4` | `Online compaction` will occupy the resources of the write task. It is recommended to use [`offline compaction`](compaction/#flink-offline-compaction) |
+
+### Compaction
+
+:::note
+These are options only for `online compaction`.
+:::
+
+:::note
+Turn off online compaction by setting `compaction.async.enabled` = `false`, but we still recommend turning on `compaction.schedule.enable` for the writing job. You can then execute the compaction plan by [`offline compaction`](compaction.md#flink-offline-compaction).
+:::
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `compaction.schedule.enabled` | Whether to generate compaction plan periodically | `true` | Recommend to turn it on, even if `compaction.async.enabled` = `false` |
+| `compaction.async.enabled` | Async Compaction, enabled by default for MOR | `true` | Turn off `online compaction` by turning off this option |
+| `compaction.trigger.strategy` | Strategy to trigger compaction | `num_commits` | Options are `num_commits`: trigger compaction when reach N delta commits; `time_elapsed`: trigger compaction when time elapsed > N seconds since last compaction; `num_and_time`: trigger compaction when both `NUM_COMMITS` and `TIME_ELAPSED` are satisfied; `num_or_time`: trigger compaction when `NUM_COMMITS` or `TIME_ELAPSED` is satisfied. |
+| `compaction.delta_commits` | Max delta commits needed to trigger compaction, default `5` commits | `5` | -- |
+| `compaction.delta_seconds` | Max delta seconds time needed to trigger compaction, default `1` hour | `3600` | -- |
+| `compaction.max_memory` | Max memory in MB for compaction spillable map, default `100MB` | `100` | If your have sufficient resources, recommend to adjust to `1024MB` |
+| `compaction.target_io` | Target IO per compaction (both read and write), default `500GB`| `512000` | -- |
+
+## Memory Optimization
+
+### MOR
+
+1. [Setting Flink state backend to `rocksdb`](#checkpoint) (the default `in memory` state backend is very memory intensive).
+2. If there is enough memory, `compaction.max_memory` can be set larger (`100MB` by default, and can be adjust to `1024MB`).
+3. Pay attention to the memory allocated to each write task by taskManager to ensure that each write task can be allocated to the
+ desired memory size `write.task.max.size`. For example, taskManager has `4GB` of memory running two streamWriteFunction, so each write task
+ can be allocated with `2GB` memory. Please reserve some buffers because the network buffer and other types of tasks on taskManager (such as bucketAssignFunction) will also consume memory.
+4. Pay attention to the memory changes of compaction. `compaction.max_memory` controls the maximum memory that each task can be used when compaction tasks read
+ logs. `compaction.tasks` controls the parallelism of compaction tasks.
+
+### COW
+
+1. [Setting Flink state backend to `rocksdb`](#checkpoint) (the default `in memory` state backend is very memory intensive).
+2. Increase both `write.task.max.size` and `write.merge.max_memory` (`1024MB` and `100MB` by default, adjust to `2014MB` and `1024MB`).
+3. Pay attention to the memory allocated to each write task by taskManager to ensure that each write task can be allocated to the
+ desired memory size `write.task.max.size`. For example, taskManager has `4GB` of memory running two write tasks, so each write task
+ can be allocated with `2GB` memory. Please reserve some buffers because the network buffer and other types of tasks on taskManager (such as `BucketAssignFunction`) will also consume memory.
+
+
+## Write Rate Limit
+
+In the existing data synchronization, `snapshot data` and `incremental data` are send to kafka first, and then streaming write
+to Hudi by Flink. Because the direct consumption of `snapshot data` will lead to problems such as high throughput and serious
+disorder (writing partition randomly), which will lead to write performance degradation and throughput glitches. At this time,
+the `write.rate.limit` option can be turned on to ensure smooth writing.
+
+### Options
+
+| Option Name | Required | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.rate.limit` | `false` | `0` | Turn off by default |
+
+## Managed-Memory Write Buffer
+
+By default, the Flink write buffer uses JVM heap memory (`ON_HEAP`). In containerized environments where heap memory is tightly budgeted, you can switch to Flink's managed (off-heap) memory pool to reduce GC pressure and avoid OOM errors.
+
+:::note
+When using `MANAGED` memory type, ensure `taskmanager.memory.managed.size` is configured sufficiently in `flink-conf.yaml`.
+:::
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.buffer.memory.type` | Memory type for the write buffer: `ON_HEAP` (default, uses JVM heap) or `MANAGED` (uses Flink managed off-heap memory) | `ON_HEAP` | Switch to `MANAGED` to avoid OOM in memory-constrained deployments |
+| `write.memory.segment.page.size` | Page size in bytes for memory segments used in the write buffer | `32768` (32 KB) | Tune for workload characteristics; larger pages reduce overhead for large records |
+
+## Disruptor Buffer Tuning
+
+When `write.buffer.type=DISRUPTOR` is set in the table options (see [Append Write Buffer](ingestion_flink.md#append-write-buffer)), the following tuning options control the Disruptor ring buffer:
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.buffer.disruptor.ring.size` | Size of the Disruptor ring buffer (must be a power of 2) | `16384` | Larger values absorb write bursts but consume more heap memory |
Review Comment:
π€ The wording "As of Hudi 1.2.0, Flink writers support `TIMELINE_SERVER_BASED` marker type" reads as if this is a brand-new capability in 1.2.0. `TIMELINE_SERVER_BASED` markers themselves have been around since 0.9.0. Is the intent to call out that Flink specifically gained support in 1.2.0, or is this restating an existing feature? @danny0405 could you confirm when Flink writers gained TIMELINE_SERVER_BASED marker support so this can be worded accurately?
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/flink-quick-start-guide.md:
##########
@@ -0,0 +1,472 @@
+---
+title: "Flink Quick Start"
+toc: true
+last_modified_at: 2026-05-27T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This page introduces FlinkβHudi integration and demonstrates how Flink brings the power of streaming to Hudi.
+
+## Setup
+
+### Flink Support Matrix
+
+| Hudi | Supported Flink versions |
+| :----- | :---------------------------------------------------------------------------- |
+| 1.2.x | 1.17.x, 1.18.x, 1.19.x, 1.20.x (default build), 2.0.x, 2.1.x |
+| 1.1.x | 1.17.x, 1.18.x, 1.19.x, 1.20.x (default build), 2.0.x |
+| 1.0.x | 1.14.x, 1.15.x, 1.16.x, 1.17.x, 1.18.x, 1.19.x, 1.20.x (default build) |
+| 0.15.x | 1.14.x, 1.15.x, 1.16.x, 1.17.x, 1.18.x |
+| 0.14.x | 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x |
+
+### Download Flink and Start Flink cluster
+
+- You can follow the [instructions here](https://flink.apache.org/downloads) for setting up Flink
+- Then start a standalone Flink cluster within a Hadoop environment
+
+For local setup, you can download Hadoop binaries and set `HADOOP_HOME` as follows:
+
+```shell
+# HADOOP_HOME is your hadoop root directory after unpack the binary package.
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+
+# Start the Flink standalone cluster
+./bin/start-cluster.sh
+```
+<div className="notice--info">
+ <h4>Please note the following: </h4>
+<ul>
+ <li>We recommend Hadoop 2.9.x+ because some object storage systems have filesystem implementations only from that version onward</li>
+ <li>The flink-parquet and flink-avro formats are already packaged into the hudi-flink-bundle jar</li>
+</ul>
+</div>
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+<TabItem value="flinksql">
+
+We use the [Flink SQL Client](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sqlclient/) because it's a good
Review Comment:
π€ This link points to `flink-docs-release-1.13` for the Flink SQL Client, but the 1.2.0 Flink Support Matrix above only lists Flink 1.17.x+. Worth bumping the doc URL to a currently supported Flink minor (e.g., 1.20) so the SQL Client docs match the supported runtime.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/flink-quick-start-guide.md:
##########
@@ -0,0 +1,472 @@
+---
+title: "Flink Quick Start"
+toc: true
+last_modified_at: 2026-05-27T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This page introduces FlinkβHudi integration and demonstrates how Flink brings the power of streaming to Hudi.
+
+## Setup
+
+### Flink Support Matrix
+
+| Hudi | Supported Flink versions |
+| :----- | :---------------------------------------------------------------------------- |
+| 1.2.x | 1.17.x, 1.18.x, 1.19.x, 1.20.x (default build), 2.0.x, 2.1.x |
+| 1.1.x | 1.17.x, 1.18.x, 1.19.x, 1.20.x (default build), 2.0.x |
+| 1.0.x | 1.14.x, 1.15.x, 1.16.x, 1.17.x, 1.18.x, 1.19.x, 1.20.x (default build) |
+| 0.15.x | 1.14.x, 1.15.x, 1.16.x, 1.17.x, 1.18.x |
+| 0.14.x | 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x |
+
+### Download Flink and Start Flink cluster
+
+- You can follow the [instructions here](https://flink.apache.org/downloads) for setting up Flink
+- Then start a standalone Flink cluster within a Hadoop environment
+
+For local setup, you can download Hadoop binaries and set `HADOOP_HOME` as follows:
+
+```shell
+# HADOOP_HOME is your hadoop root directory after unpack the binary package.
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+
+# Start the Flink standalone cluster
+./bin/start-cluster.sh
+```
+<div className="notice--info">
+ <h4>Please note the following: </h4>
+<ul>
+ <li>We recommend Hadoop 2.9.x+ because some object storage systems have filesystem implementations only from that version onward</li>
+ <li>The flink-parquet and flink-avro formats are already packaged into the hudi-flink-bundle jar</li>
+</ul>
+</div>
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+<TabItem value="flinksql">
+
+We use the [Flink SQL Client](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sqlclient/) because it's a good
+quick-start tool for SQL users.
+
+### Start Flink SQL Client
+
+Hudi supports a packaged bundle jar for Flink, which should be loaded in the Flink SQL Client when it starts up.
+You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore.md#install)), or download it from the
+[Apache Official Repository](https://repo.maven.apache.org/maven2/org/apache/hudi/).
+
+Now start the SQL CLI:
+
+```bash
+# Supported Flink versions for Hudi 1.2.x: 1.17, 1.18, 1.19, 1.20 (default build), 2.0, 2.1
+export FLINK_VERSION=1.20
+export HUDI_VERSION=1.2.0
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-flink${FLINK_VERSION}-bundle/${HUDI_VERSION}/hudi-flink${FLINK_VERSION}-bundle-${HUDI_VERSION}.jar -P /tmp/
+./bin/sql-client.sh embedded -j /tmp/hudi-flink${FLINK_VERSION}-bundle-${HUDI_VERSION}.jar shell
+```
+
+Setup table name, base path and operate using SQL for this guide.
+The SQL CLI only executes the SQL line by line.
+</TabItem>
+
+<TabItem value="dataStream">
+
+Please add the desired dependency to your project:
+```xml
+<!-- Supported Flink versions for Hudi 1.2.x: 1.17, 1.18, 1.19, 1.20 (default build), 2.0, 2.1 -->
+<properties>
+ <flink.version>1.20.1</flink.version>
+ <flink.binary.version>1.20</flink.binary.version>
+ <hudi.version>1.2.0</hudi.version>
+</properties>
+<dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-flink${flink.binary.version}-bundle</artifactId>
+ <version>${hudi.version}</version>
+</dependency>
+```
+
+</TabItem>
+
+</Tabs
+>
+
+## Create Table
+
+First, let's create a Hudi table. Here, we use a partitioned table for illustration, but Hudi also supports non-partitioned tables.
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+Here is an example of creating a Flink Hudi table.
+
+```sql
+-- sets up the result mode to tableau to show the results directly in the CLI
+set sql-client.execution.result-mode = tableau;
+DROP TABLE hudi_table;
+CREATE TABLE hudi_table(
+ ts BIGINT,
+ uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
+ rider VARCHAR(20),
+ driver VARCHAR(20),
+ fare DOUBLE,
+ city VARCHAR(20)
+)
+PARTITIONED BY (`city`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'file:///tmp/hudi_table',
+ 'table.type' = 'MERGE_ON_READ'
+);
+```
+
+</TabItem>
+
+<TabItem value="dataStream">
+
+```java
+// Java
+// The first commit will auto-initialize the table if it does not exist in the specified base path.
+```
+
+</TabItem>
+
+
+</Tabs
+>
+
+## Insert Data
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+Insert data into the Hudi table using SQL `VALUES`.
+
+```sql
+-- insert data using values
+INSERT INTO hudi_table
+VALUES
+(1695159649087,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554788,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462179,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066204,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137016,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo'),
+(1695376420876,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo'),
+(1695173887231,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai'),
+(1695115999911,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+```
+</TabItem>
+
+<TabItem value="dataStream">
+
+Add some streaming source to flink and load the data in hudi table. Since, this is the first write, it will also auto-create the table.
+
+```java
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.data.RowData;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.util.HoodiePipeline;
+
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+String targetTable = "hudi_table";
+String basePath = "file:///tmp/hudi_table";
+
+Map<String, String> options = new HashMap<>();
+options.put("path", basePath);
+options.put("table.type", HoodieTableType.MERGE_ON_READ.name());
+options.put("ordering.fields", "ts");
+
+DataStream<RowData> dataStream = env.addSource(...);
+HoodiePipeline.Builder builder = HoodiePipeline.builder(targetTable)
+ .column("uuid VARCHAR(20)")
+ .column("name VARCHAR(10)")
+ .column("age INT")
+ .column("ts TIMESTAMP(3)")
+ .column("`partition` VARCHAR(20)")
+ .pk("uuid")
+ .partition("partition")
+ .options(options);
+
+builder.sink(dataStream, false); // The second parameter indicating whether the input data stream is bounded
+env.execute("Api_Sink");
+```
+Refer Full Quickstart Example [here](https://github.com/ad1happy2go/hudi-examples/blob/main/flink/src/main/java/com/hudi/flink/quickstart/HudiDataStreamWriter.java)
+
+</TabItem>
+
+</Tabs
+>
+
+## Query Data
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+```sql
+-- query from the Hudi table
+select * from hudi_table;
+```
+</TabItem>
+
+<TabItem value="dataStream">
+
+```java
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.data.RowData;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.util.HoodiePipeline;
+
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+String targetTable = "hudi_table";
+String basePath = "file:///tmp/hudi_table";
+
+Map<String, String> options = new HashMap<>();
+options.put("path", basePath);
+options.put("table.type", HoodieTableType.MERGE_ON_READ.name());
+options.put("read.streaming.enabled", "true"); // this option enable the streaming read
+options.put("read.start-commit", "20210316134557"); // specifies the start commit instant time
+
+HoodiePipeline.Builder builder = HoodiePipeline.builder(targetTable)
+ .column("uuid VARCHAR(20)")
+ .column("name VARCHAR(10)")
+ .column("age INT")
+ .column("ts TIMESTAMP(3)")
+ .column("`partition` VARCHAR(20)")
+ .pk("uuid")
+ .partition("partition")
+ .options(options);
+
+DataStream<RowData> rowDataDataStream = builder.source(env);
+rowDataDataStream.print();
+env.execute("Api_Source");
+```
+Refer Full Streaming Reader Example [here](https://github.com/ad1happy2go/hudi-examples/blob/main/flink/src/main/java/com/hudi/flink/quickstart/HudiDataStreamReader.java)
+
+</TabItem>
+
+</Tabs
+>
+
+This statement queries snapshot view of the dataset.
+Refers to [Table types and queries](concepts.md#table-types--queries) for more info on all table types and query types supported.
+
+## Update Data
+
+This is similar to inserting new data.
+
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+Hudi tables can be updated by either inserting records with the same record key or using a standard UPDATE statement as shown below.
+
+```sql
+-- Update Queries only works with batch execution mode
+SET 'execution.runtime-mode' = 'batch';
+UPDATE hudi_table SET fare = 25.0 WHERE uuid = '334e26e9-8355-45cc-97c6-c31daf0df330';
+```
+
+:::note
+The `UPDATE` statement is supported since Flink 1.17, so only Hudi Flink bundle compiled with Flink 1.17+ supplies this functionality.
+Only **batch** queries on Hudi table with record key work correctly.
+:::
+</TabItem>
+
+<TabItem value="dataStream">
+
+Add a streaming source to Flink and load the data into the Hudi table using the DataStream API as [above](#insert-data).
+When new rows with the same record key arrive in the stream, they will be updated.
+In the insert example, incoming rows with the same record ID will be updated.
+
+Refer Update Example [here](https://github.com/ad1happy2go/hudi-examples/blob/main/flink/src/main/java/com/hudi/flink/quickstart/HudiDataStreamWriter.java)
+
+</TabItem>
+
+</Tabs
+>
+
+[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts.md)
+denoted by the timestamp.
+
+
+## Delete Data {#deletes}
+<Tabs
+defaultValue="flinksql"
+values={[
+{ label: 'Flink SQL', value: 'flinksql', },
+{ label: 'DataStream API', value: 'dataStream', },
+]}
+>
+
+<TabItem value="flinksql">
+
+### RowβLevel Delete
+
+When consuming data in a streaming query, the Hudi Flink source can also accept change logs from the upstream data source if the `RowKind` is set up per row;
+it can then apply UPDATE and DELETE at the row level. You can then sync a nearβrealβtime snapshot on Hudi for all kinds
+of RDBMS.
+
+### Batch Delete
+
+```sql
+-- delete all the records with age greater than 23
+-- NOTE: only works for batch sql queries
+SET 'execution.runtime-mode' = 'batch';
+DELETE FROM t1 WHERE age > 23;
+```
+
+:::note
+The `DELETE` statement is supported since Flink 1.17, so only Hudi Flink bundle compiled with Flink 1.17+ supplies this functionality.
+Only **batch** queries on Hudi table with record key work correctly.
+:::
+
+</TabItem>
+
+<TabItem value="dataStream">
+
+First, create a Flink Hudi table and insert data into the Hudi table using the DataStream API as below.
+When new rows with the same record key and Row Kind as DELETE arrive in the stream, they will be deleted.
+
+Refer Delete Example [here](https://github.com/ad1happy2go/hudi-examples/blob/main/flink/src/main/java/com/hudi/flink/quickstart/HudiDataStreamWriter.java)
+
+</TabItem
+>
+
+</Tabs
+>
+
+## Streaming Query
+
+Hudi Flink also provides the capability to obtain a stream of records that changed since a given commit timestamp.
+This can be achieved using Hudi's streaming querying and providing a start time from which changes need to be streamed.
+We do not need to specify endTime, if we want all changes after the given commit (as is the common case).
+
+```sql
+CREATE TABLE t1(
+ uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = '${path}',
+ 'table.type' = 'MERGE_ON_READ',
+ 'read.streaming.enabled' = 'true', -- this option enable the streaming read
+ 'read.start-commit' = '20210316134557', -- specifies the start commit instant time
+ 'read.streaming.check-interval' = '4' -- specifies the check interval for finding new source commits; default is 60s.
+);
+
+-- Then query the table in stream mode
+select * from t1;
+```
+
+## Change Data Capture Query
+
+Hudi Flink also provides the capability to obtain a stream of records with Change Data Capture.
+CDC queries are useful for applications that need to obtain all changes, along with before/after images of records.
+
+```sql
+set sql-client.execution.result-mode = tableau;
+
+CREATE TABLE hudi_table(
+ ts BIGINT,
+ uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
+ rider VARCHAR(20),
+ driver VARCHAR(20),
+ fare DOUBLE,
+ city VARCHAR(20)
+)
+PARTITIONED BY (`city`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'file:///tmp/hudi_table',
+ 'table.type' = 'COPY_ON_WRITE',
+ 'cdc.enabled' = 'true' -- this option enables CDC logging
+);
+-- insert data using values
+INSERT INTO hudi_table
+VALUES
+(1695159649087,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554788,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462179,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066204,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137016,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo'),
+(1695376420876,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo'),
+(1695173887231,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai'),
+(1695115999911,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+SET 'execution.runtime-mode' = 'batch';
+UPDATE hudi_table SET fare = 25.0 WHERE uuid = '334e26e9-8355-45cc-97c6-c31daf0df330';
+-- Query the table in stream mode in another shell to see change logs
+SET 'execution.runtime-mode' = 'streaming';
+select * from hudi_table/*+ OPTIONS('read.streaming.enabled'='true')*/;
+```
+
+This will give all changes that happened after the `read.start-commit` commit. The unique thing about this
+feature is that it lets you author streaming pipelines on streaming or batch data sources.
+
+## Where To Go From Here?
+
+- **Quick Start**: Read the Quick Start section above to get started quickly with the Flink SQL client to write to (and read from) Hudi.
+- **Configuration**: For [Global Configuration](flink_tuning.md#global-configurations), set up through `$FLINK_HOME/conf/flink-conf.yaml`. For per-job configuration, set up through [Table Option](flink_tuning.md#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink.md#cdc-ingestion), [Bulk Insert](ingestion_flink.md#bulk-insert), [Index Bootstrap](ingestion_flink.md#index-bootstrap), [Changelog Mode](ingestion_flink.md#changelog-mode) and [Append Mode](ingestion_flink.md#append-mode). For high-throughput append pipelines, choose an [append write buffer mode](ingestion_flink.md#append-write-buffer). For upsert workloads at scale, use [Record-Level Index (RLI) bucket indexing](ingestion_flink.md#record-level-index-rli-bucket-indexing-for-flink). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml.md#non-blocking-concurrency-control-experimental).
+- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries.md#streaming-query) and [Incremental Query](sql_queries.md#incremental-query). For improved push-down and resumable reads, see [Flink Source V2](ingestion_flink.md#flink-source-v2). For dimension-table joins, use [lookup join](ingestion_flink.md#lookup-join) with an optional off-heap RocksDB cache.
+- **Tuning**: For write/read tasks, this guide provides some tuning suggestions, such as [Memory Optimization](flink_tuning.md#memory-optimization), the [Managed-Memory Write Buffer](flink_tuning.md#managed-memory-write-buffer), and [Write Rate Limit](flink_tuning.md#write-rate-limit).
+- **Optimization**: Offline compaction is supported: [Offline Compaction](compaction.md#flink-offline-compaction).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore.md#flink-setup), [Presto Query](sql_queries.md#presto).
+- **Catalog**: A Hudiβspecific catalog is supported: [Hudi Catalog](sql_ddl/#create-catalog).
+
+If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
+
+- [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
Review Comment:
π€ The link `[Hudi Catalog](sql_ddl/#create-catalog)` is missing the `.md` extension that the surrounding links use (e.g., `sql_queries.md#streaming-query`). For consistency and to ensure the link resolves correctly in the versioned docs, this should be `sql_ddl.md#create-catalog`.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_ddl.md:
##########
@@ -0,0 +1,1218 @@
+---
+title: SQL DDL
+summary: "In this page, we discuss using SQL DDL commands with Hudi"
+toc: true
+last_modified_at: 2026-05-29T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This page describes support for creating and altering tables using SQL across various engines.
+
+## Spark SQL
+
+### Create table
+
+You can create tables using standard CREATE TABLE syntax, which supports partitioning and passing table properties.
+
+```sql
+CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ [(col_name data_type [COMMENT col_comment], ...)]
+ [COMMENT table_comment]
+ [PARTITIONED BY (col_name, ...)]
+ [ROW FORMAT row_format]
+ [STORED AS file_format]
+ [LOCATION path]
+ [TBLPROPERTIES (property_name=property_value, ...)]
+ [AS select_statement];
+```
+
+:::note NOTE:
+For users running this tutorial locally and have a Spark-Hive(HMS) integration in their environment: If you use
+`default` database or if you don't provide `[LOCATION path]` with the DDL statement, Spark will return
+`java.io.IOException: Mkdirs failed to create file:/user/hive/warehouse/hudi_table/.hoodie` error.
+To get around this, you can follow either of the two options mentioned below:
+1. Create a database i.e. `CREATE DATABASE hudidb;` and use it i.e. `USE hudidb;` before running the DDL statement.
+2. Or provide a path using `LOCATION` keyword to persist the data with the DDL statement.
+:::
+
+### Create non-partitioned table
+
+Creating a non-partitioned table is as simple as creating a regular table.
+
+```sql
+-- create a Hudi table
+CREATE TABLE IF NOT EXISTS hudi_table (
+ id INT,
+ name STRING,
+ price DOUBLE
+) USING hudi;
+```
+
+### Create partitioned table
+A partitioned table can be created by adding a `partitioned by` clause. Partitioning helps to organize the data into multiple folders
+based on the partition columns. It can also help speed up queries and index lookups by limiting the amount of metadata, index and data scanned.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_partitioned (
+ id BIGINT,
+ name STRING,
+ dt STRING,
+ hh STRING
+) USING hudi
+TBLPROPERTIES (
+ type = 'cow'
+)
+PARTITIONED BY (dt);
+```
+
+:::note
+You can also create a table partitioned by multiple fields by supplying comma-separated field names.
+When creating a table partitioned by multiple fields, ensure that you specify the columns in the `PARTITIONED BY` clause
+in the same order as they appear in the `CREATE TABLE` schema. For example, for the above table, the partition fields
+should be specified as `PARTITIONED BY (dt, hh)`.
+:::
+
+### Create table with record keys and ordering fields
+
+As discussed [here](quick-start-guide.md#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
+key for each new record in the examples so far. If you want to use an existing field as the key, you can set the `primaryKey` option.
+Typically, this is also accompanied by configuring ordering fields (via `orderingFields` option) to deal with out-of-order data and potential
+duplicate records with the same key in the incoming writes.
+
+:::note
+You can choose multiple fields as primary keys for a given table on a need basis. For eg, "primaryKey = 'id, name'", and
+this materializes a composite key of the two fields, which can be useful for exploring the table.
+:::
+
+Here is an example of creating a table using both options. Typically, a field that denotes the time of the event or
+fact, e.g., order creation time, event generation time etc., is used as the ordering field (via `orderingFields`). Hudi resolves multiple versions
+of the same record by ordering based on this field when queries are run on the table.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_keyed (
+ id INT,
+ name STRING,
+ price DOUBLE,
+ ts BIGINT
+) USING hudi
+TBLPROPERTIES (
+ type = 'cow',
+ primaryKey = 'id',
+ orderingFields = 'ts'
+);
+```
+
+### Create table with merge modes {#create-table-with-record-merge-mode}
+
+Hudi supports different [record merge modes](record_merger.md) to handle merge of incoming records with existing
+records. To create a table with specific record merge mode, you can set `recordMergeMode` option.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'EVENT_TIME_ORDERING'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode/';
+```
+
+With `EVENT_TIME_ORDERING`, the record with the larger event time (specified via `orderingFields`) overwrites the record with the
+smaller event time on the same key, regardless of transaction's commit time. Users can set `CUSTOM` mode to provide their own
+merge logic. With `CUSTOM` merge mode, you can provide a custom class that implements the merge logic. The interfaces
+to implement is explained in detail [here](record_merger.md#custom).
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'CUSTOM',
+ 'hoodie.record.merge.strategy.id' = '<unique-uuid>'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
+```
+
+### Create table with unstructured and semi-structured column types {#create-table-with-unstructured-and-semi-structured-column-types}
+
+Hudi supports three column types for unstructured and semi-structured data, plus a Lance base file
+format.
+
+#### `VECTOR(dim[, elementType])` {#vector}
+
+A fixed-dimension embedding column. `elementType` is one of `FLOAT` (default), `DOUBLE`, or `INT8`
+(alias `BYTE`):
+
+| Element type | Storage |
+|:-------------|:--------|
+| `FLOAT` (default) | `ArrayType(FloatType)` |
+| `DOUBLE` | `ArrayType(DoubleType)` |
+| `INT8` / `BYTE` | `ArrayType(ByteType)` |
+
+```sql
+CREATE TABLE products (
+ product_id STRING,
+ name STRING,
+ embedding VECTOR(768) -- defaults to FLOAT
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'product_id',
+ type = 'cow',
+ hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger'
+);
+
+-- Other element types
+-- embedding VECTOR(768, FLOAT)
+-- embedding VECTOR(768, DOUBLE)
+-- embedding VECTOR(256, INT8)
+```
+
+Hudi's SQL parser normalizes `VECTOR(128, FLOAT)` to `VECTOR(128)`.
+
+VECTOR columns must be **top-level fields**; nesting inside `STRUCT`, `ARRAY`, or `MAP` is not
+supported. Dimension and element type cannot be changed via schema evolution after table creation.
+
+**Internal storage.** In Parquet, a VECTOR column is stored as `FIXED_LEN_BYTE_ARRAY` with
+`hudi_type=VECTOR(dim[,elementType])` field metadata so the Spark reader decodes the bytes back into
+a typed array. On Lance-backed tables, VECTOR is stored natively as
+`FixedSizeList<Float32/Float64, dim>`; only `FLOAT` and `DOUBLE` element types are accepted on Lance.
+
+Query VECTOR columns with the [`hudi_vector_search` TVF](sql_queries.md#vector-similarity-search).
+
+**Engine constraints.** Flink cannot decode VECTOR columns (the underlying Parquet
+`FIXED_LEN_BYTE_ARRAY` is not converted back into a typed array); Flink can still read other columns
+in a table that contains a VECTOR column.
+
+#### `BLOB` {#blob}
+
+A binary column with two storage modes:
+
+| Mode | Storage | Read pattern |
+|:-----|:--------|:-------------|
+| `INLINE` | Raw bytes embedded in the table row | Direct read; no external fetch |
+| `OUT_OF_LINE` | Pointer to a byte range in an external file | On-demand via [`read_blob()`](sql_queries.md#reading-blob-columns) |
+
+```sql
+CREATE TABLE media_assets (
+ asset_id STRING,
+ file_name STRING,
+ mime_type STRING,
+ file_size BIGINT,
+ content BLOB
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'asset_id',
+ type = 'cow'
+);
+```
+
+**Internal struct.** A BLOB column is represented internally as:
+
+```
+STRUCT<
+ type: STRING, -- 'INLINE' or 'OUT_OF_LINE'
+ data: BINARY, -- raw bytes for INLINE; null for OUT_OF_LINE
+ reference: STRUCT<
+ external_path: STRING, -- file path for OUT_OF_LINE
+ offset: BIGINT, -- byte offset; null = start of file
+ length: BIGINT, -- byte length; null = read to end
+ managed: BOOLEAN -- advisory flag for the (future) cleaner
+ >
Review Comment:
π€ This statement says "Hudi 1.2.0 supports Spark 3.4 and 3.5", but the Spark Support Matrix in `quick-start-guide.md` lists Hudi 1.2.x as supporting Spark 3.3.x, 3.4.x, 3.5.x, 4.0.x, and 4.1.x. The intent here is likely "of the Spark 3.x versions Hudi 1.2.0 supports, none can construct a native VariantType" β could you reword to either enumerate the Spark 3.x versions (3.3 / 3.4 / 3.5) or phrase it as "On the Spark 3.x runtimes Hudi 1.2.0 supports, ..."? @yihua could you confirm which Spark 3.x versions this backward-compat read path is actually tested against?
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/concepts.md:
##########
@@ -0,0 +1,177 @@
+---
+version: 0.6.0
Review Comment:
π€ The front matter has `version: 0.6.0` at the top β this looks like a stale field that was carried over from very old docs (it's not a standard Docusaurus content-versioning field, and the page is otherwise the current 1.2.0 concepts page). Probably safe to remove during the snapshot.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_queries.md:
##########
@@ -0,0 +1,980 @@
+---
+title: SQL Queries
+summary: "In this page, we go over querying Hudi tables using SQL"
+toc: true
+last_modified_at: 2026-05-29T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
+This page will show how to issue different queries and discuss any specific instructions for each query engine.
+
+## Spark SQL
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+
+:::tip Setting Hudi read options at the session level
+Hudi 1.2.0 supports setting read options at the **Spark session level** using the `spark.hoodie.*` prefix.
+Any `spark.hoodie.X` config set via `spark.conf.set` or `--conf` is treated equivalently to `hoodie.X`.
+
+Config precedence (low β high):
+1. Global DFS properties
+2. `spark.hoodie.*` session-level configs (normalized to `hoodie.*`)
+3. Explicit `hoodie.*` data source options or per-table `SET` commands
+
+```sql
+-- Apply a Hudi read option for the entire session
+SET spark.hoodie.metadata.column.stats.enable = true;
+SELECT * FROM hudi_table WHERE price BETWEEN 10.0 AND 50.0;
+```
+
+If both `spark.hoodie.X` and `hoodie.X` are set, the explicit `hoodie.X` value takes precedence.
+:::
+
+### Snapshot Query
+Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+Using session properties, you can specify options around indexing to optimize query performance, as shown below.
+
+```sql
+-- You can turn on relevant options for indexing.
+
+-- Turn on use of column stat index, to perform range queries.
+SET hoodie.metadata.column.stats.enable=true;
+SELECT * FROM hudi_table
+WHERE price > 1.0 and price < 10.0
+
+-- Turn on use of record level index, to perform point queries.
+SET hoodie.metadata.record.index.enable=true;
+SELECT * FROM hudi_table
+WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa'
+```
+
+:::note Integration with Spark
+Users are encouraged to migrate to Hudi versions > 0.12.x, for the best spark experience and discouraged from using any older approaches
+using path filters. We expect that native integration with Spark's optimized table readers along with Hudi's automatic table
+management will yield great performance benefits in those versions.
+:::
+
+### Snapshot Query with Index Acceleration
+
+In this section we would go over the various indexes and how they help in data skipping in Hudi. We will first create
+a hudi table without any index.
+
+```sql
+-- Create a table with primary key
+CREATE TABLE hudi_indexed_table (
+ ts BIGINT,
+ uuid STRING,
+ rider STRING,
+ driver STRING,
+ fare DOUBLE,
+ city STRING
+) USING HUDI
+options(
+ primaryKey ='uuid',
+ hoodie.write.record.merge.mode = "COMMIT_TIME_ORDERING"
+)
+PARTITIONED BY (city);
+
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo' ),
+(1695376420,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo' ),
+(1695173887,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai' ),
+(1695115999,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+UPDATE hudi_indexed_table SET rider = 'rider-B', driver = 'driver-N', ts = '1697516137' WHERE rider = 'rider-A';
+```
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data. Let's create a secondary index on the rider column.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+```
+
+
+<p align = "left">Figure: Query pruning without secondary index</p>
+
+### Query using Secondary Index
+
+We will run the query again after creating secondary index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+:::note
+Please note in order to create secondary index:
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
+2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
+:::
+
+```sql
+-- We will first create a record index since secondary index is dependent upon it
+CREATE INDEX record_index ON hudi_indexed_table (uuid);
+-- We create a secondary index on rider column
+CREATE INDEX idx_rider ON hudi_indexed_table (rider);
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+DROP INDEX record_index on hudi_indexed_table;
+DROP INDEX secondary_index_idx_rider on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with secondary index</p>
+
+### Query using Bloom Filter Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet on the `driver` column.
+All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+```
+
+
+<p align = "left">Figure: Query pruning without bloom filter expression index</p>
+
+We will run the query again after creating bloom filter expression index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a bloom filter expression index on driver column
+CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+DROP INDEX expr_index_idx_bloom_driver on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with bloom filter expression index</p>
+
+### Query using Column Stats Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+```
+
+
+<p align = "left">Figure: Query pruning without column stat expression index</p>
+
+We will run the query again after creating column stat expression index on ts column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a column stat expression index on ts column
+CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(expr='from_unixtime', format = 'yyyy-MM-dd');
+-- We run the same query again
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+DROP INDEX expr_index_idx_column_ts on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with column stat expression index</p>
+
+### Query using Partition Stats Index
+
+With the query run below, we will see no data skipping or pruning since there is no partition stats index created yet in the table as can
+be seen in the image below. All the partitions are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+```
+
+
+<p align = "left">Figure: Query pruning without partition stats index</p>
+
+We will run the query again after creating partition stats index. The query would now show the partitions scanned as 1
+compared to 3 partitions scanned without index.
+
+```sql
+-- We will need to enable column stats as well since partition stats index leverages it
+SET hoodie.metadata.index.partition.stats.enable=true;
+SET hoodie.metadata.index.column.stats.enable=true;
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'854g46e0-8355-45cc-97c6-c31daf0df330','rider-H','driver-T',19.10,'chennai');
+-- Run the query again on the table with partition stats index
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+DROP INDEX column_stats on hudi_indexed_table;
+DROP INDEX partition_stats on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with partition stats index</p>
+
+### Snapshot Query with Event Time Ordering
+
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
+time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
+event time ordering merge mode.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'EVENT_TIME_ORDERING'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but lower ts
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 900, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=1000, price=10.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode;
Review Comment:
π€ This looks like a leftover author TODO/draft note (`# add note on checkpoint translation from 0.x to 1.x. same for incremental query below`) that wasn't cleaned up before publishing β it renders as an H1 in the generated page. Could you remove this line? The note block that follows already covers the checkpoint translation behavior.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_queries.md:
##########
@@ -0,0 +1,980 @@
+---
+title: SQL Queries
+summary: "In this page, we go over querying Hudi tables using SQL"
+toc: true
+last_modified_at: 2026-05-29T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
+This page will show how to issue different queries and discuss any specific instructions for each query engine.
+
+## Spark SQL
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+
+:::tip Setting Hudi read options at the session level
+Hudi 1.2.0 supports setting read options at the **Spark session level** using the `spark.hoodie.*` prefix.
+Any `spark.hoodie.X` config set via `spark.conf.set` or `--conf` is treated equivalently to `hoodie.X`.
+
+Config precedence (low β high):
+1. Global DFS properties
+2. `spark.hoodie.*` session-level configs (normalized to `hoodie.*`)
+3. Explicit `hoodie.*` data source options or per-table `SET` commands
+
+```sql
+-- Apply a Hudi read option for the entire session
+SET spark.hoodie.metadata.column.stats.enable = true;
+SELECT * FROM hudi_table WHERE price BETWEEN 10.0 AND 50.0;
+```
+
+If both `spark.hoodie.X` and `hoodie.X` are set, the explicit `hoodie.X` value takes precedence.
+:::
+
+### Snapshot Query
+Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+Using session properties, you can specify options around indexing to optimize query performance, as shown below.
+
+```sql
+-- You can turn on relevant options for indexing.
+
+-- Turn on use of column stat index, to perform range queries.
+SET hoodie.metadata.column.stats.enable=true;
+SELECT * FROM hudi_table
+WHERE price > 1.0 and price < 10.0
+
+-- Turn on use of record level index, to perform point queries.
+SET hoodie.metadata.record.index.enable=true;
+SELECT * FROM hudi_table
+WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa'
+```
+
+:::note Integration with Spark
+Users are encouraged to migrate to Hudi versions > 0.12.x, for the best spark experience and discouraged from using any older approaches
+using path filters. We expect that native integration with Spark's optimized table readers along with Hudi's automatic table
+management will yield great performance benefits in those versions.
+:::
+
+### Snapshot Query with Index Acceleration
+
+In this section we would go over the various indexes and how they help in data skipping in Hudi. We will first create
+a hudi table without any index.
+
+```sql
+-- Create a table with primary key
+CREATE TABLE hudi_indexed_table (
+ ts BIGINT,
+ uuid STRING,
+ rider STRING,
+ driver STRING,
+ fare DOUBLE,
+ city STRING
+) USING HUDI
+options(
+ primaryKey ='uuid',
+ hoodie.write.record.merge.mode = "COMMIT_TIME_ORDERING"
+)
+PARTITIONED BY (city);
+
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo' ),
+(1695376420,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo' ),
+(1695173887,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai' ),
+(1695115999,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+UPDATE hudi_indexed_table SET rider = 'rider-B', driver = 'driver-N', ts = '1697516137' WHERE rider = 'rider-A';
+```
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data. Let's create a secondary index on the rider column.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+```
+
+
+<p align = "left">Figure: Query pruning without secondary index</p>
+
+### Query using Secondary Index
+
+We will run the query again after creating secondary index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+:::note
+Please note in order to create secondary index:
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
+2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
+:::
+
+```sql
+-- We will first create a record index since secondary index is dependent upon it
+CREATE INDEX record_index ON hudi_indexed_table (uuid);
+-- We create a secondary index on rider column
+CREATE INDEX idx_rider ON hudi_indexed_table (rider);
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+DROP INDEX record_index on hudi_indexed_table;
+DROP INDEX secondary_index_idx_rider on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with secondary index</p>
+
+### Query using Bloom Filter Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet on the `driver` column.
+All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+```
+
+
+<p align = "left">Figure: Query pruning without bloom filter expression index</p>
+
+We will run the query again after creating bloom filter expression index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a bloom filter expression index on driver column
+CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+DROP INDEX expr_index_idx_bloom_driver on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with bloom filter expression index</p>
+
+### Query using Column Stats Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+```
+
+
+<p align = "left">Figure: Query pruning without column stat expression index</p>
+
+We will run the query again after creating column stat expression index on ts column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a column stat expression index on ts column
+CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(expr='from_unixtime', format = 'yyyy-MM-dd');
+-- We run the same query again
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+DROP INDEX expr_index_idx_column_ts on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with column stat expression index</p>
+
+### Query using Partition Stats Index
+
+With the query run below, we will see no data skipping or pruning since there is no partition stats index created yet in the table as can
+be seen in the image below. All the partitions are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+```
+
+
+<p align = "left">Figure: Query pruning without partition stats index</p>
+
+We will run the query again after creating partition stats index. The query would now show the partitions scanned as 1
+compared to 3 partitions scanned without index.
+
+```sql
+-- We will need to enable column stats as well since partition stats index leverages it
+SET hoodie.metadata.index.partition.stats.enable=true;
+SET hoodie.metadata.index.column.stats.enable=true;
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'854g46e0-8355-45cc-97c6-c31daf0df330','rider-H','driver-T',19.10,'chennai');
+-- Run the query again on the table with partition stats index
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+DROP INDEX column_stats on hudi_indexed_table;
+DROP INDEX partition_stats on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with partition stats index</p>
+
+### Snapshot Query with Event Time Ordering
+
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
+time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
+event time ordering merge mode.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'EVENT_TIME_ORDERING'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but lower ts
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 900, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=1000, price=10.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode;
+```
+
+With `EVENT_TIME_ORDERING`, the record with the larger event time (specified via `orderingFields`) overwrites the record with the
+smaller event time on the same key, regardless of transaction time.
+
+### Snapshot Query with Custom Merge Mode
+
+Users can set `CUSTOM` mode to provide their own merge logic. With `CUSTOM` merge mode, you also need to provide your
+payload class that implements the merge logic. For example, you can use `PartialUpdateAvroPayload` to merge the records
+as below.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'CUSTOM',
+ 'hoodie.datasource.write.payload.class' = 'org.apache.hudi.common.model.PartialUpdateAvroPayload'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but set higher ts and name as null to show partial update
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, null, 2000, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=2000, price=20.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode_custom;
+```
+
+As you can see, not only the record with higher ordering field overwrites the record with lower ordering value, but also
+the name field is partially updated.
+
+### Time Travel Query
+
+You can also query the table at a specific commit time using the `AS OF` syntax. This is useful for debugging and auditing purposes, as well as for
+machine learning pipelines where you want to train models on a specific point in time.
+
+```sql
+SELECT * FROM <table name>
+TIMESTAMP AS OF '<timestamp in yyyy-MM-dd HH:mm:ss.SSS or yyyy-MM-dd or yyyyMMddHHmmssSSS>'
+WHERE <filter conditions>
+```
+
+### Change Data Capture
+
+Change Data Capture (CDC) queries are useful when you want to obtain all changes to a Hudi table within a given time window, along with before/after images and change operation
+of the changed records. Similar to many relational database counterparts, Hudi provides flexible ways of controlling supplemental logging levels, to balance storage/logging costs
+by materializing more versus compute costs of computing the changes on the fly, using `hoodie.table.cdc.supplemental.logging.mode` configuration.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
+ <pathToTable | tableName>,
+ 'cdc',
+ <'earliest' | <time to capture from>>
+ [, <time to capture to>]
+)
Review Comment:
π€ Small typo: `compatiblity` β `compatibility`.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_queries.md:
##########
@@ -0,0 +1,980 @@
+---
+title: SQL Queries
+summary: "In this page, we go over querying Hudi tables using SQL"
+toc: true
+last_modified_at: 2026-05-29T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
+This page will show how to issue different queries and discuss any specific instructions for each query engine.
+
+## Spark SQL
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+
+:::tip Setting Hudi read options at the session level
+Hudi 1.2.0 supports setting read options at the **Spark session level** using the `spark.hoodie.*` prefix.
+Any `spark.hoodie.X` config set via `spark.conf.set` or `--conf` is treated equivalently to `hoodie.X`.
+
+Config precedence (low β high):
+1. Global DFS properties
+2. `spark.hoodie.*` session-level configs (normalized to `hoodie.*`)
+3. Explicit `hoodie.*` data source options or per-table `SET` commands
+
+```sql
+-- Apply a Hudi read option for the entire session
+SET spark.hoodie.metadata.column.stats.enable = true;
+SELECT * FROM hudi_table WHERE price BETWEEN 10.0 AND 50.0;
+```
+
+If both `spark.hoodie.X` and `hoodie.X` are set, the explicit `hoodie.X` value takes precedence.
+:::
+
+### Snapshot Query
+Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+Using session properties, you can specify options around indexing to optimize query performance, as shown below.
+
+```sql
+-- You can turn on relevant options for indexing.
+
+-- Turn on use of column stat index, to perform range queries.
+SET hoodie.metadata.column.stats.enable=true;
+SELECT * FROM hudi_table
+WHERE price > 1.0 and price < 10.0
+
+-- Turn on use of record level index, to perform point queries.
+SET hoodie.metadata.record.index.enable=true;
+SELECT * FROM hudi_table
+WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa'
+```
+
+:::note Integration with Spark
+Users are encouraged to migrate to Hudi versions > 0.12.x, for the best spark experience and discouraged from using any older approaches
+using path filters. We expect that native integration with Spark's optimized table readers along with Hudi's automatic table
+management will yield great performance benefits in those versions.
+:::
+
+### Snapshot Query with Index Acceleration
+
+In this section we would go over the various indexes and how they help in data skipping in Hudi. We will first create
+a hudi table without any index.
+
+```sql
+-- Create a table with primary key
+CREATE TABLE hudi_indexed_table (
+ ts BIGINT,
+ uuid STRING,
+ rider STRING,
+ driver STRING,
+ fare DOUBLE,
+ city STRING
+) USING HUDI
+options(
+ primaryKey ='uuid',
+ hoodie.write.record.merge.mode = "COMMIT_TIME_ORDERING"
+)
+PARTITIONED BY (city);
+
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo' ),
+(1695376420,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo' ),
+(1695173887,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai' ),
+(1695115999,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+UPDATE hudi_indexed_table SET rider = 'rider-B', driver = 'driver-N', ts = '1697516137' WHERE rider = 'rider-A';
+```
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data. Let's create a secondary index on the rider column.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+```
+
+
+<p align = "left">Figure: Query pruning without secondary index</p>
+
+### Query using Secondary Index
+
+We will run the query again after creating secondary index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+:::note
+Please note in order to create secondary index:
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
+2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
+:::
+
+```sql
+-- We will first create a record index since secondary index is dependent upon it
+CREATE INDEX record_index ON hudi_indexed_table (uuid);
+-- We create a secondary index on rider column
+CREATE INDEX idx_rider ON hudi_indexed_table (rider);
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+DROP INDEX record_index on hudi_indexed_table;
+DROP INDEX secondary_index_idx_rider on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with secondary index</p>
+
+### Query using Bloom Filter Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet on the `driver` column.
+All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+```
+
+
+<p align = "left">Figure: Query pruning without bloom filter expression index</p>
+
+We will run the query again after creating bloom filter expression index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a bloom filter expression index on driver column
+CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+DROP INDEX expr_index_idx_bloom_driver on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with bloom filter expression index</p>
+
+### Query using Column Stats Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+```
+
+
+<p align = "left">Figure: Query pruning without column stat expression index</p>
+
+We will run the query again after creating column stat expression index on ts column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a column stat expression index on ts column
+CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(expr='from_unixtime', format = 'yyyy-MM-dd');
+-- We run the same query again
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+DROP INDEX expr_index_idx_column_ts on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with column stat expression index</p>
+
+### Query using Partition Stats Index
+
+With the query run below, we will see no data skipping or pruning since there is no partition stats index created yet in the table as can
+be seen in the image below. All the partitions are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+```
+
+
+<p align = "left">Figure: Query pruning without partition stats index</p>
+
+We will run the query again after creating partition stats index. The query would now show the partitions scanned as 1
+compared to 3 partitions scanned without index.
+
+```sql
+-- We will need to enable column stats as well since partition stats index leverages it
+SET hoodie.metadata.index.partition.stats.enable=true;
+SET hoodie.metadata.index.column.stats.enable=true;
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'854g46e0-8355-45cc-97c6-c31daf0df330','rider-H','driver-T',19.10,'chennai');
+-- Run the query again on the table with partition stats index
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+DROP INDEX column_stats on hudi_indexed_table;
+DROP INDEX partition_stats on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with partition stats index</p>
+
+### Snapshot Query with Event Time Ordering
+
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
+time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
+event time ordering merge mode.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'EVENT_TIME_ORDERING'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but lower ts
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 900, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=1000, price=10.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode;
+```
+
+With `EVENT_TIME_ORDERING`, the record with the larger event time (specified via `orderingFields`) overwrites the record with the
+smaller event time on the same key, regardless of transaction time.
+
+### Snapshot Query with Custom Merge Mode
+
+Users can set `CUSTOM` mode to provide their own merge logic. With `CUSTOM` merge mode, you also need to provide your
+payload class that implements the merge logic. For example, you can use `PartialUpdateAvroPayload` to merge the records
+as below.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'CUSTOM',
+ 'hoodie.datasource.write.payload.class' = 'org.apache.hudi.common.model.PartialUpdateAvroPayload'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but set higher ts and name as null to show partial update
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, null, 2000, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=2000, price=20.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode_custom;
+```
+
+As you can see, not only the record with higher ordering field overwrites the record with lower ordering value, but also
+the name field is partially updated.
+
+### Time Travel Query
+
+You can also query the table at a specific commit time using the `AS OF` syntax. This is useful for debugging and auditing purposes, as well as for
+machine learning pipelines where you want to train models on a specific point in time.
+
+```sql
+SELECT * FROM <table name>
+TIMESTAMP AS OF '<timestamp in yyyy-MM-dd HH:mm:ss.SSS or yyyy-MM-dd or yyyyMMddHHmmssSSS>'
+WHERE <filter conditions>
+```
+
+### Change Data Capture
+
+Change Data Capture (CDC) queries are useful when you want to obtain all changes to a Hudi table within a given time window, along with before/after images and change operation
+of the changed records. Similar to many relational database counterparts, Hudi provides flexible ways of controlling supplemental logging levels, to balance storage/logging costs
+by materializing more versus compute costs of computing the changes on the fly, using `hoodie.table.cdc.supplemental.logging.mode` configuration.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
Review Comment:
π€ The two checkpoint admonitions use inconsistent version naming β the CDC note above says `Hudi 0.x and 1.x` while this incremental query note says `Hudi 0.x and 1.0.`. Could you make these consistent? The CDC version (`0.x and 1.x`) reads better since 1.2.0 is shipping with this doc.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_queries.md:
##########
@@ -0,0 +1,980 @@
+---
+title: SQL Queries
+summary: "In this page, we go over querying Hudi tables using SQL"
+toc: true
+last_modified_at: 2026-05-29T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
+This page will show how to issue different queries and discuss any specific instructions for each query engine.
+
+## Spark SQL
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+
+:::tip Setting Hudi read options at the session level
+Hudi 1.2.0 supports setting read options at the **Spark session level** using the `spark.hoodie.*` prefix.
+Any `spark.hoodie.X` config set via `spark.conf.set` or `--conf` is treated equivalently to `hoodie.X`.
+
+Config precedence (low β high):
+1. Global DFS properties
+2. `spark.hoodie.*` session-level configs (normalized to `hoodie.*`)
+3. Explicit `hoodie.*` data source options or per-table `SET` commands
+
+```sql
+-- Apply a Hudi read option for the entire session
+SET spark.hoodie.metadata.column.stats.enable = true;
+SELECT * FROM hudi_table WHERE price BETWEEN 10.0 AND 50.0;
+```
+
+If both `spark.hoodie.X` and `hoodie.X` are set, the explicit `hoodie.X` value takes precedence.
+:::
+
+### Snapshot Query
+Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+Using session properties, you can specify options around indexing to optimize query performance, as shown below.
+
+```sql
+-- You can turn on relevant options for indexing.
+
+-- Turn on use of column stat index, to perform range queries.
+SET hoodie.metadata.column.stats.enable=true;
+SELECT * FROM hudi_table
+WHERE price > 1.0 and price < 10.0
+
+-- Turn on use of record level index, to perform point queries.
+SET hoodie.metadata.record.index.enable=true;
+SELECT * FROM hudi_table
+WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa'
+```
+
+:::note Integration with Spark
+Users are encouraged to migrate to Hudi versions > 0.12.x, for the best spark experience and discouraged from using any older approaches
+using path filters. We expect that native integration with Spark's optimized table readers along with Hudi's automatic table
+management will yield great performance benefits in those versions.
+:::
+
+### Snapshot Query with Index Acceleration
+
+In this section we would go over the various indexes and how they help in data skipping in Hudi. We will first create
+a hudi table without any index.
+
+```sql
+-- Create a table with primary key
+CREATE TABLE hudi_indexed_table (
+ ts BIGINT,
+ uuid STRING,
+ rider STRING,
+ driver STRING,
+ fare DOUBLE,
+ city STRING
+) USING HUDI
+options(
+ primaryKey ='uuid',
+ hoodie.write.record.merge.mode = "COMMIT_TIME_ORDERING"
+)
+PARTITIONED BY (city);
+
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo' ),
+(1695376420,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo' ),
+(1695173887,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai' ),
+(1695115999,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+UPDATE hudi_indexed_table SET rider = 'rider-B', driver = 'driver-N', ts = '1697516137' WHERE rider = 'rider-A';
+```
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data. Let's create a secondary index on the rider column.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+```
+
+
+<p align = "left">Figure: Query pruning without secondary index</p>
+
+### Query using Secondary Index
+
+We will run the query again after creating secondary index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+:::note
+Please note in order to create secondary index:
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
+2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
+:::
+
+```sql
+-- We will first create a record index since secondary index is dependent upon it
+CREATE INDEX record_index ON hudi_indexed_table (uuid);
+-- We create a secondary index on rider column
+CREATE INDEX idx_rider ON hudi_indexed_table (rider);
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+DROP INDEX record_index on hudi_indexed_table;
+DROP INDEX secondary_index_idx_rider on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with secondary index</p>
+
+### Query using Bloom Filter Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet on the `driver` column.
+All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+```
+
+
+<p align = "left">Figure: Query pruning without bloom filter expression index</p>
+
+We will run the query again after creating bloom filter expression index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a bloom filter expression index on driver column
+CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+DROP INDEX expr_index_idx_bloom_driver on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with bloom filter expression index</p>
+
+### Query using Column Stats Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+```
+
+
+<p align = "left">Figure: Query pruning without column stat expression index</p>
+
+We will run the query again after creating column stat expression index on ts column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a column stat expression index on ts column
+CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(expr='from_unixtime', format = 'yyyy-MM-dd');
+-- We run the same query again
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+DROP INDEX expr_index_idx_column_ts on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with column stat expression index</p>
+
+### Query using Partition Stats Index
+
+With the query run below, we will see no data skipping or pruning since there is no partition stats index created yet in the table as can
+be seen in the image below. All the partitions are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+```
+
+
+<p align = "left">Figure: Query pruning without partition stats index</p>
+
+We will run the query again after creating partition stats index. The query would now show the partitions scanned as 1
+compared to 3 partitions scanned without index.
+
+```sql
+-- We will need to enable column stats as well since partition stats index leverages it
+SET hoodie.metadata.index.partition.stats.enable=true;
+SET hoodie.metadata.index.column.stats.enable=true;
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'854g46e0-8355-45cc-97c6-c31daf0df330','rider-H','driver-T',19.10,'chennai');
+-- Run the query again on the table with partition stats index
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+DROP INDEX column_stats on hudi_indexed_table;
+DROP INDEX partition_stats on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with partition stats index</p>
+
+### Snapshot Query with Event Time Ordering
+
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
+time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
+event time ordering merge mode.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'EVENT_TIME_ORDERING'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but lower ts
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 900, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=1000, price=10.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode;
+```
+
+With `EVENT_TIME_ORDERING`, the record with the larger event time (specified via `orderingFields`) overwrites the record with the
+smaller event time on the same key, regardless of transaction time.
+
+### Snapshot Query with Custom Merge Mode
+
+Users can set `CUSTOM` mode to provide their own merge logic. With `CUSTOM` merge mode, you also need to provide your
+payload class that implements the merge logic. For example, you can use `PartialUpdateAvroPayload` to merge the records
+as below.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'CUSTOM',
+ 'hoodie.datasource.write.payload.class' = 'org.apache.hudi.common.model.PartialUpdateAvroPayload'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but set higher ts and name as null to show partial update
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, null, 2000, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=2000, price=20.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode_custom;
+```
+
+As you can see, not only the record with higher ordering field overwrites the record with lower ordering value, but also
+the name field is partially updated.
+
+### Time Travel Query
+
+You can also query the table at a specific commit time using the `AS OF` syntax. This is useful for debugging and auditing purposes, as well as for
+machine learning pipelines where you want to train models on a specific point in time.
+
+```sql
+SELECT * FROM <table name>
+TIMESTAMP AS OF '<timestamp in yyyy-MM-dd HH:mm:ss.SSS or yyyy-MM-dd or yyyyMMddHHmmssSSS>'
+WHERE <filter conditions>
+```
+
+### Change Data Capture
+
+Change Data Capture (CDC) queries are useful when you want to obtain all changes to a Hudi table within a given time window, along with before/after images and change operation
+of the changed records. Similar to many relational database counterparts, Hudi provides flexible ways of controlling supplemental logging levels, to balance storage/logging costs
+by materializing more versus compute costs of computing the changes on the fly, using `hoodie.table.cdc.supplemental.logging.mode` configuration.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
+ <pathToTable | tableName>,
+ 'cdc',
+ <'earliest' | <time to capture from>>
+ [, <time to capture to>]
+)
+```
+
+# add note on checkpoint translation from 0.x to 1.x. same for incremental query below
+:::note CDC Query Checkpointing between Hudi 0.x and 1.x
+In Hudi 1.0, we switch the incremental and CDC queries to used completion time, instead of requested instant time, to determine the
+range of commits to incrementally pull from. The checkpoint stored for Hudi incremental source and related sources is
+also changed to use completion time. To seamless migration without downtime or data duplication, Hudi does an automatic checkpoint
+translation from requested instant time to completion time depending on the source table version.
+:::
+
+### Incremental Query
+
+Incremental queries are useful when you want to obtain the latest values for all records that have changed after a given commit time. They help author incremental data pipelines with
+orders of magnitude efficiency over batch counterparts by only processing the changed records. Hudi users have realized [large gains](https://www.uber.com/blog/ubers-lakehouse-architecture/) in
+query efficiency by using incremental queries in this fashion. Hudi supports incremental queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
+ <pathToTable | tableName>,
+ 'latest_state',
+ <'earliest' | <time to capture from>>
+ [, <time to capture to>]
+)
+```
+
+:::info Incremental vs CDC Queries
+Incremental queries offer even better query efficiency than even the CDC queries above, since they amortize the cost of compactions across your data lake.
+For e.g the table has received 10 million modifications across 1 million records over a time window, incremental queries can fetch the latest value for
+1 million records using Hudi's record level metadata. On the other hand, the CDC queries will process 10 million records and useful in cases, where you want to
+see all changes in a given time window and not just the latest values.
+:::
+
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
+
+:::note Incremental Query Checkpointing between Hudi 0.x and 1.0.
+In Hudi 1.0, we switch the incremental and CDC query to used completion time, instead of instant time, to determine the
+range of commits to incrementally pull from. The checkpoint stored for Hudi incremental source and related sources is
+also changed to use completion time. To support compatiblity, Hudi does a checkpoint translation from requested instant
+time to completion time depending on the source table version.
+:::
+
+### Vector Similarity Search
+
+The `hudi_vector_search` table-valued function (TVF) runs top-K similarity search over a
+[`VECTOR`](sql_ddl.md#vector) column. It returns the `top_k` rows whose VECTOR column is closest to
+a query vector under the chosen distance metric.
+
+```sql
+SELECT *
+FROM hudi_vector_search(
+ table_name, -- STRING: registered table name or path
+ vector_column, -- STRING: VECTOR column name
+ query_vector, -- ARRAY: query embedding
+ top_k, -- INT: number of nearest neighbors
+ [distance_metric], -- STRING: 'cosine' (default), 'l2', 'dot_product'
+ [algorithm] -- STRING: 'brute_force' (default)
+)
+```
+
+Parameters:
+
+| Parameter | Type | Default | Description |
+|:----------|:-----|:--------|:------------|
+| `table_name` | STRING | (required) | Registered table name or table path. |
+| `vector_column` | STRING | (required) | Name of the VECTOR column. |
+| `query_vector` | ARRAY<FLOAT> | (required) | Query embedding; must match the column's dimension and element type. |
+| `top_k` | INT | (required) | Number of nearest neighbors. |
+| `distance_metric` | STRING | `'cosine'` | One of `'cosine'`, `'l2'`, `'dot_product'`. |
+| `algorithm` | STRING | `'brute_force'` | Only `'brute_force'` is currently supported. |
+
+Return schema: all columns from the source table (excluding the embedding column) plus
+`_hudi_distance DOUBLE`. Results are ordered by `_hudi_distance` ascending, so the closest matches
+come first.
+
+Distance metrics:
+
+| Metric | Formula | Range | Notes |
+|:-------|:--------|:------|:------|
+| `cosine` | 1 β cos(a, b), clamped to [0, 2] | [0, 2] | Returns 1.0 for zero vectors. |
+| `l2` | sqrt(sum((a[i] β b[i])Β²)) | [0, +β) | |
+| `dot_product` | β(a Β· b) | (ββ, +β) | Negated so ascending sort surfaces the most similar rows first. |
+
+```sql
+-- Find the 10 nearest neighbors to a query embedding
+SELECT product_id, name, _hudi_distance
+FROM hudi_vector_search('products', 'embedding',
+ ARRAY(0.12, -0.03, 0.87, /* ... */),
+ 10, 'cosine')
+ORDER BY _hudi_distance;
+```
+
+RAG context retrieval to apply a distance threshold to drop weak matches:
+
+```sql
+-- Retrieve the 5 most relevant document chunks for an LLM prompt
+SELECT chunk_id, text_content, _hudi_distance
+FROM hudi_vector_search(
+ 'document_chunks', 'embedding',
+ ARRAY(/* embedding of the user's question */),
+ 5, 'cosine'
+)
+WHERE _hudi_distance < 0.3;
+```
+
+Cross-modal search to query a corpus of image embeddings with a text embedding (e.g., CLIP):
+
+```sql
+SELECT image_id, caption, _hudi_distance
+FROM hudi_vector_search(
+ 'image_catalog', 'clip_embedding',
+ ARRAY(/* text embedding from CLIP */),
+ 20, 'cosine'
+);
+```
+
+The TVF also works on Lance-backed tables. Set `hoodie.table.base.file.format=lance` at table
+creation (see [Storage Layouts β Lance](storage_layouts.md#lance-base-file-format)); the call site
+is unchanged.
+
+:::note
+`cosine` distance computes `1 β cos(a, b)`. If embeddings are not L2-normalized before write,
+results reflect both vector direction and magnitude.
+:::
+
+#### `hudi_vector_search_batch`
+
+The batch variant runs many query vectors at once:
+
+```sql
+SELECT *
+FROM hudi_vector_search_batch(
+ corpus_table, -- STRING
+ corpus_embedding_col, -- STRING
+ query_table, -- STRING: table holding query vectors
+ query_embedding_col, -- STRING: VECTOR column in query table
+ top_k, -- INT: neighbors per query
+ [distance_metric],
+ [algorithm]
+)
+```
+
+Return schema: corpus columns + query columns + `_hudi_distance DOUBLE` +
+`_hudi_query_index LONG` (identifies the query that produced the row). If corpus and query share
+column names, query columns are prefixed with `_hudi_query_`.
+
+### Reading BLOB Columns
+
+The `read_blob()` SQL function materializes raw bytes from a [`BLOB`](sql_ddl.md#blob) column. It
+handles both storage modes uniformly: for `INLINE`, it extracts the embedded bytes; for
+`OUT_OF_LINE`, it reads `reference.length` bytes starting at `reference.offset` from
+`reference.external_path`.
+
+```sql
+SELECT asset_id, read_blob(content) AS raw_bytes
+FROM media_assets
+WHERE asset_id = 'asset_001';
+```
+
+Standard queries on a BLOB column return the underlying struct (descriptor + bytes), not raw bytes:
+
+```sql
+SELECT asset_id, content.type, content.reference.external_path
+FROM media_assets;
+```
+
+#### Inline read mode
+
+| Property | Default | Description |
+|:---------|:--------|:------------|
+| `hoodie.read.blob.inline.mode` | `DESCRIPTOR` | Controls how `INLINE` BLOBs surface on read. `DESCRIPTOR` (default) returns an out-of-line-shaped reference pointing at the in-file coordinates of the bytes. No bytes are materialized. `CONTENT` materializes the raw inline bytes in the `data` field on every read. |
+| `hoodie.blob.batching.max.gap.bytes` | `4096` | Maximum gap between consecutive byte ranges before they are merged into a single read. Larger values reduce I/O calls at the cost of reading some unused bytes. |
+| `hoodie.blob.batching.lookahead.size` | `50` | Number of rows to buffer for batch-read detection. Larger values improve batching for sorted data but increase memory usage. |
+
+`CONTENT` mode is always used for internal operations (compaction, merge, log replay) regardless of
+this setting.
+
+:::caution Calling `read_blob()` on INLINE columns under DESCRIPTOR mode
+Under the default `DESCRIPTOR` mode, calling `read_blob()` on an `INLINE` BLOB column throws.
+The raw bytes are not materialized in the scan, so there is nothing for `read_blob()` to return.
+To read inline bytes with `read_blob()`, switch to `CONTENT` mode first:
+
+```sql
+SET hoodie.read.blob.inline.mode=CONTENT;
+SELECT asset_id, read_blob(content) AS raw_bytes
+FROM media_assets WHERE asset_id = 'asset_001';
+```
+
+This setting affects only `INLINE` columns. `OUT_OF_LINE` always fetches from the external path.
+:::
+
+`read_blob()` is a Spark SQL function; Hive, BigQuery, and other engines reading the underlying
+struct directly do not have it. Apply predicates before calling `read_blob()` to bound the bytes
+resolved per row.
+
+### Querying VARIANT Columns
+
+[`VARIANT`](sql_ddl.md#variant) columns hold semi-structured (JSON-like) data. Cast to `STRING` for
+JSON output:
+
+```sql
+SELECT event_id, cast(payload as STRING) AS payload_json FROM events;
+```
+
+VARIANT columns support `UPDATE`, `DELETE`, and `MERGE` on both COW and MOR tables.
+
+:::note Spark 3.x (3.4 / 3.5)
+`parse_json()`, `variant_get()`, and `cast(... as STRING)` require Spark 4.0+. On Spark 3.x,
+read raw binary buffers via the
+[backward-compat DDL](sql_ddl.md#reading-variant-from-spark-3x-backward-compatibility);
+predicate pushdown into VARIANT fields is not available.
+:::
+
+#### End-to-end example
+
+`hudi_vector_search` and `read_blob()` compose in a single query that returns both the matching
+rows and the materialized bytes for each:
+
+```sql
+SELECT image_id, category,
+ read_blob(image_bytes) AS resolved_bytes,
+ _hudi_distance
+FROM hudi_vector_search(
+ '/tmp/hudi_pets', 'embedding',
+ ARRAY(0.12, -0.03, /* ... */),
+ 5, 'cosine'
+)
+ORDER BY _hudi_distance;
+```
+
+See [Unstructured Data Quick Start Guide](unstructured-data-quick-start-guide.md) for a full end-to-end walk-through.
+
+### Query Indexes and Timeline
+
+Hudi also allows users to directly query the metadata partitions and check the metadata corresponding to the table
+and the various indexes. In this section we will check the various queries which can be used for this purpose.
+
+Let's first create a table with various indexes created.
+```sql
+-- Create a table with primary key
+CREATE TABLE hudi_indexed_table (
+ ts BIGINT,
+ uuid STRING,
+ rider STRING,
+ driver STRING,
+ fare DOUBLE,
+ city STRING
+) USING HUDI
+options(
+ primaryKey ='uuid',
+ hoodie.write.record.merge.mode = "COMMIT_TIME_ORDERING"
+)
+PARTITIONED BY (city);
+
+-- Create partition stat index
+SET hoodie.metadata.index.partition.stats.enable=true;
+SET hoodie.metadata.index.column.stats.enable=true;
+
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo' ),
+(1695376420,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo' ),
+(1695173887,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai' ),
+(1695115999,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+
+-- Create column stat expression index on ts column
+CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(expr='from_unixtime', format = 'yyyy-MM-dd');
+-- Create secondary index on rider column
+CREATE INDEX record_index ON hudi_indexed_table (uuid);
+CREATE INDEX idx_rider ON hudi_indexed_table (rider);
+SET hoodie.metadata.record.index.enable=true;
+```
+
+```sql
+-- Query the secondary keys stores in the secondary index partition
+SELECT key FROM hudi_metadata('hudi_indexed_table') WHERE type=7;
+
+-- Query the column stat records stored in the column stat indexes or column stat expression index
+select ColumnStatsMetadata.columnName, ColumnStatsMetadata.minValue, ColumnStatsMetadata.maxValue from hudi_metadata('hudi_indexed_table') where type=3;
+-- Query can be further refined to get nested fields and exact values for a particular partition.
+-- Below query fetches the column stats metadata for column stat expression index on ts column.
+select ColumnStatsMetadata.columnName, ColumnStatsMetadata.minValue.member6.value, ColumnStatsMetadata.maxValue.member6.value from hudi_metadata('hudi_indexed_table') where type=3 AND ColumnStatsMetadata.columnName='ts';
+
+-- Query the partition stat index records for rider column. Partition stat index records use the same schema as column stat index records
+select ColumnStatsMetadata.columnName, ColumnStatsMetadata.minValue.member6.value, ColumnStatsMetadata.maxValue.member6.value from hudi_metadata('hudi_indexed_table') where type=6 AND ColumnStatsMetadata.columnName='rider';
+```
+
+All the different index types can be queries by specifying the type column for that index. Here are the metadata partitions
+and the corresponding type column value.
+
+| MDT Partition | Type Column Value |
+|:------------------:|:-------------------:|
+| Files | 2 |
+| Column Stat | 3 |
+| Bloom Filters | 4 |
+| Record Index | 5 |
+| Secondary Index | 7 |
+| Partition Stats | 6 |
+
+
+## Flink SQL
+Once the Flink Hudi tables have been registered to the Flink catalog, they can be queried using the Flink SQL. It supports all query types across both Hudi table types,
+relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide.md).
+
+
+### Snapshot Query
+By default, Flink SQL will try to use its optimized native readers (for e.g. reading parquet files) instead of Hive SerDes.
+Additionally, partition pruning is applied by Flink if a partition predicate is specified in the filter. Filters push down may not be supported yet (please check Flink roadmap).
+
+```sql
+select * from hudi_table/*+ OPTIONS('metadata.enabled'='true', 'read.data.skipping.enabled'='false','hoodie.metadata.index.column.stats.enable'='true')*/;
+```
+
+#### Options
+| Option Name | Required | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `metadata.enabled` | `false` | false | Set to `true` to enable |
+| `read.data.skipping.enabled` | `false` | false | Whether to enable data skipping for batch snapshot read, by default disabled |
+| `hoodie.metadata.index.column.stats.enable` | `false` | false | Whether to enable column statistics (max/min) |
+| `hoodie.metadata.index.column.stats.column.list` | `false` | N/A | Columns(separated by comma) to collect the column statistics |
+
+### Streaming Query
+By default, the hoodie table is read as batch, that is to read the latest snapshot data set and returns. Turns on the streaming read
+mode by setting option `read.streaming.enabled` as `true`. Sets up option `read.start-commit` to specify the read start offset, specifies the
+value as `earliest` if you want to consume all the history data set.
+
+```sql
+select * from hudi_table/*+ OPTIONS('read.streaming.enabled'='true', 'read.start-commit'='earliest')*/;
+```
+
+#### Options
+| Option Name | Required | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `read.streaming.enabled` | false | `false` | Specify `true` to read as streaming |
+| `read.start-commit` | false | the latest commit | Start commit time in format 'yyyyMMddHHmmss', use `earliest` to consume from the start commit |
+| `read.streaming.skip_compaction` | false | `false` | Whether to skip compaction instants for streaming read, generally for two purpose: 1) Avoid consuming duplications from compaction instants created for created by Hudi versions < 0.11.0 or when `hoodie.compaction.preserve.commit.metadata` is disabled 2) When change log mode is enabled, to only consume change for right semantics. |
+| `clean.retain_commits` | false | `10` | The max number of commits to retain before cleaning, when change log mode is enabled, tweaks this option to adjust the change log live time. For example, the default strategy keeps 50 minutes of change logs if the checkpoint interval is set up as 5 minutes. |
+
+:::note
+Users are encouraged to use Hudi versions > 0.12.3, for the best experience and discouraged from using any older versions.
+Specifically, `read.streaming.skip_compaction` should only be enabled if the MOR table is compacted by Hudi with versions `< 0.11.0`.
+This is so as the `hoodie.compaction.preserve.commit.metadata` feature is only introduced in Hudi versions `>=0.11.0`.
+Older versions will overwrite the original commit time for each row with the compaction plan's instant time and cause
+row-level instant range checks to not work properly.
+:::
+
+### Incremental Query
+There are 3 use cases for incremental query:
+1. Streaming query: specify the start commit with option `read.start-commit`;
+2. Batch query: specify the start commit with option `read.start-commit` and end commit with option `read.end-commit`,
+ the interval is a closed one: both start commit and end commit are inclusive;
+3. Time Travel: consume as batch for an instant time, specify the `read.end-commit` is enough because the start commit is latest by default.
Review Comment:
π€ The remarks for `read.streaming.skip_compaction` contain a doubled phrase: `compaction instants created for created by Hudi versions < 0.11.0`. It looks like an editing artifact β consider trimming to `compaction instants created by Hudi versions < 0.11.0`.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/syncing_metastore.md:
##########
@@ -0,0 +1,361 @@
+---
+title: Apache Hive Metastore
+keywords: [hudi, hive, metastore]
+---
+
+[Hive Metastore](https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration) is an
+RDBMS-backed service from Apache Hive that acts as a catalog for your data warehouse or data lake. It can store all the
+metadata about the tables, such as partitions, columns, column types, etc. One can sync the Hudi table metadata to the
+Hive metastore as well. This unlocks the capability to query Hudi tables not only through Hive but also using
+interactive query engines such as Presto and Trino. In this document, we will go through different ways to sync the Hudi
+table to Hive metastore.
+
+## Spark Data Source example
+
+Prerequisites: setup hive metastore properly and configure the Spark installation to point to the hive metastore by placing `hive-site.xml` under `$SPARK_HOME/conf`
+
+Assume that
+ - hiveserver2 is running at port 10000
+ - metastore is running at port 9083
+
+Then start a spark-shell with Hudi spark bundle jar as a dependency (refer to Quickstart example)
+
+We can run the following script to create a sample hudi table and sync it to hive.
+
+```scala
+// spark-shell
+import org.apache.hudi.QuickstartUtils._
+import scala.collection.JavaConversions._
+import org.apache.spark.sql.SaveMode._
+import org.apache.hudi.DataSourceReadOptions._
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.config.HoodieWriteConfig._
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.Row
+
+
+val databaseName = "my_db"
+val tableName = "hudi_cow"
+val basePath = "/user/hive/warehouse/hudi_cow"
+
+val schema = StructType(Array(
+StructField("rowId", StringType,true),
+StructField("partitionId", StringType,true),
+StructField("orderingField", LongType,true),
+StructField("name", StringType,true),
+StructField("versionId", StringType,true),
+StructField("toBeDeletedStr", StringType,true),
+StructField("intToLong", IntegerType,true),
+StructField("longToInt", LongType,true)
+))
+
+val data0 = Seq(Row("row_1", "2021/01/01",0L,"bob","v_0","toBeDel0",0,1000000L),
+ Row("row_2", "2021/01/01",0L,"john","v_0","toBeDel0",0,1000000L),
+ Row("row_3", "2021/01/02",0L,"tom","v_0","toBeDel0",0,1000000L))
+
+var dfFromData0 = spark.createDataFrame(data0,schema)
+
+dfFromData0.write.format("hudi").
+ options(getQuickstartWriteConfigs).
+ option("hoodie.table.ordering.fields", "orderingField").
+ option("hoodie.datasource.write.recordkey.field", "rowId").
+ option("hoodie.datasource.write.partitionpath.field", "partitionId").
+ option("hoodie.database.name", databaseName).
+ option("hoodie.table.name", tableName).
+ option("hoodie.datasource.write.table.type", "COPY_ON_WRITE").
+ option("hoodie.datasource.write.operation", "upsert").
+ option("hoodie.datasource.write.hive_style_partitioning","true").
+ option("hoodie.datasource.meta.sync.enable", "true").
+ option("hoodie.datasource.hive_sync.mode", "hms").
+ option("hoodie.datasource.hive_sync.metastore.uris", "thrift://hive-metastore:9083").
+ mode(Overwrite).
+ save(basePath)
+```
+
+:::note
+If prefer to use JDBC instead of HMS sync mode, omit `hoodie.datasource.hive_sync.metastore.uris` and configure these instead
+
+```
+hoodie.datasource.hive_sync.mode=jdbc
+hoodie.datasource.hive_sync.jdbcurl=<e.g., jdbc:hive2://hiveserver:10000>
+hoodie.datasource.hive_sync.username=<username>
+hoodie.datasource.hive_sync.password=<password>
+```
+:::
+
+### Query using HiveQL
+
+```
+beeline -u jdbc:hive2://hiveserver:10000/my_db
+ --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat
+ --hiveconf hive.stats.autogather=false
+
+Beeline version 1.2.1.spark2 by Apache Hive
+0: jdbc:hive2://hiveserver:10000> show tables;
++-----------+--+
+| tab_name |
++-----------+--+
+| hudi_cow |
++-----------+--+
+1 row selected (0.531 seconds)
+0: jdbc:hive2://hiveserver:10000> select * from hudi_cow limit 1;
++-------------------------------+--------------------------------+------------------------------+----------------------------------+----------------------------------------------------------------------------+-----------------+-------------------+----------------+---------------------+--------------------------+---------------------+---------------------+-----------------------+--+
+| hudi_cow._hoodie_commit_time | hudi_cow._hoodie_commit_seqno | hudi_cow._hoodie_record_key | hudi_cow._hoodie_partition_path | hudi_cow._hoodie_file_name | hudi_cow.rowid | hudi_cow.orderingfield | hudi_cow.name | hudi_cow.versionid | hudi_cow.tobedeletedstr | hudi_cow.inttolong | hudi_cow.longtoint | hudi_cow.partitionid |
++-------------------------------+--------------------------------+------------------------------+----------------------------------+----------------------------------------------------------------------------+-----------------+-------------------+----------------+---------------------+--------------------------+---------------------+---------------------+-----------------------+--+
+| 20220120090023631 | 20220120090023631_1_2 | row_1 | partitionId=2021/01/01 | 0bf9b822-928f-4a57-950a-6a5450319c83-0_1-24-314_20220120090023631.parquet | row_1 | 0 | bob | v_0 | toBeDel0 | 0 | 1000000 | 2021/01/01 |
++-------------------------------+--------------------------------+------------------------------+----------------------------------+----------------------------------------------------------------------------+-----------------+-------------------+----------------+---------------------+--------------------------+---------------------+---------------------+-----------------------+--+
+1 row selected (5.475 seconds)
+0: jdbc:hive2://hiveserver:10000>
+```
+
+### Use partition extractor properly
+
+When sync to hive metastore, partition values are extracted using `hoodie.datasource.hive_sync.partition_value_extractor`. Before 0.12, this is by default set to
+`org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor` and users usually need to overwrite this manually. Since 0.12, the default value is changed to a more
+generic `org.apache.hudi.hive.MultiPartKeysValueExtractor` which extracts partition values using `/` as the separator.
+
+In case of using some key generator like `TimestampBasedKeyGenerator`, the partition values can be in form of `yyyy/MM/dd`. This is usually undesirable to have the
+partition values extracted as multiple parts like `[yyyy, MM, dd]`. Users can set `org.apache.hudi.hive.SinglePartPartitionValueExtractor` to extract the partition
+values as `yyyy-MM-dd`.
+
+When the table is not partitioned, `org.apache.hudi.hive.NonPartitionedExtractor` should be set. And this is automatically inferred from partition fields configs,
+so users may not need to set it manually. Similarly, if hive-style partitioning is used for the table, then `org.apache.hudi.hive.HiveStylePartitionValueExtractor`
+will be inferred and set automatically.
+
+## Hive Sync Tool
+
+Writing data with [DataSource](writing_data.md) writer or [Hudi Streamer](hoodie_streaming_ingestion.md) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
+In case, it's preferable to run this from commandline or in an independent jvm, Hudi provides a `HiveSyncTool`, which can be invoked as below,
+once you have built the hudi-hive module. Following is how we sync the above Datasource Writer written table to Hive metastore.
+
+```java
+cd hudi-hive
+./run_sync_tool.sh --jdbc-url jdbc:hive2://hiveserver:10000 --user hive --pass hive --partitioned-by partition --base-path <basePath> --database default --table <tableName>
+```
+
+Starting with Hudi 0.5.1 version read optimized version of merge-on-read tables are suffixed '_ro' by default. For backwards compatibility with older Hudi versions, an optional HiveSyncConfig - `--skip-ro-suffix`, has been provided to turn off '_ro' suffixing if desired. Explore other hive sync options using the following command:
+
+```java
+cd hudi-hive
+./run_sync_tool.sh
+ [hudi-hive]$ ./run_sync_tool.sh --help
+```
+
+### Hive Sync Configuration
+
+Please take a look at the arguments that can be passed to `run_sync_tool` in [HiveSyncConfig](https://github.com/apache/hudi/blob/master/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java).
+Among them, following are the required arguments:
+```java
+@Parameter(names = {"--database"}, description = "name of the target database in Hive", required = true);
+@Parameter(names = {"--table"}, description = "name of the target table in Hive", required = true);
+@Parameter(names = {"--base-path"}, description = "Basepath of Hudi table to sync", required = true);
+```
+Corresponding datasource options for the most commonly used hive sync configs are as follows:
+
+:::note
+In the table below **(N/A)** means there is no default value set.
+:::
+
+| HiveSyncConfig | DataSourceWriteOption | Default Value | Description |
+|-----------------------------|-------------------------------------------------------|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
+| --database | hoodie.datasource.hive_sync.database | default | Name of the target database in Hive metastore |
+| --table | hoodie.datasource.hive_sync.table | (N/A) | Name of the target table in Hive. Inferred from the table name in Hudi table config if not specified. |
+| --user | hoodie.datasource.hive_sync.username | hive | Username for hive metastore |
+| --pass | hoodie.datasource.hive_sync.password | hive | Password for hive metastore |
+| --jdbc-url | hoodie.datasource.hive_sync.jdbcurl | jdbc:hive2://localhost:10000 | Hive server url if using `jdbc` mode to sync |
+| --sync-mode | hoodie.datasource.hive_sync.mode | (N/A) | Mode to choose for Hive ops. Valid values are `hms`, `jdbc` and `hiveql`. More details in the following section. |
+| --partitioned-by | hoodie.datasource.hive_sync.partition_fields | (N/A) | Comma-separated column names in the table to use for determining hive partition. |
+| --partition-value-extractor | hoodie.datasource.hive_sync.partition_extractor_class | `org.apache.hudi.hive.MultiPartKeysValueExtractor` | Class which implements `PartitionValueExtractor` to extract the partition values. Inferred automatically depending on the partition fields specified. |
+
+
+### Sync modes
+
+`HiveSyncTool` supports three modes, namely `HMS`, `HIVEQL`, `JDBC`, to connect to Hive metastore server.
+These modes are just three different ways of executing DDL against Hive. Among these modes, JDBC or HMS is preferable over
+HIVEQL, which is mostly used for running DML rather than DDL.
+
+:::note
+All these modes assume that hive metastore has been configured and the corresponding properties set in
+`hive-site.xml` configuration file. Additionally, if you're using spark-shell/spark-sql to sync Hudi table to Hive then
+the `hive-site.xml` file also needs to be placed under `<SPARK_HOME>/conf` directory.
+:::
+
+#### HMS
+
+HMS mode uses the hive metastore client to sync Hudi table using thrift APIs directly.
+To use this mode, pass `--sync-mode=hms` to `run_sync_tool` and set `--use-jdbc=false`.
+Additionally, if you are using remote metastore, then `hive.metastore.uris` need to be set in hive-site.xml configuration file.
+Otherwise, the tool assumes that metastore is running locally on port 9083 by default.
+
+#### JDBC
+
+This mode uses the JDBC specification to connect to the hive metastore.
+
+```java
+@Parameter(names = {"--jdbc-url"}, description = "Hive jdbc connect url");
+```
+
+#### HIVEQL
+
+HQL is Hive's own SQL dialect.
+This mode simply uses the Hive QL's [driver](https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java) to execute DDL as HQL command.
+To use this mode, pass `--sync-mode=hiveql` to `run_sync_tool` and set `--use-jdbc=false`.
+
+## Flink Setup
+
+### Install
+
+Now you can git clone Hudi master branch to test Flink hive sync. The first step is to install Hudi to get `hudi-flink1.1x-bundle-0.x.x.jar`.
+ `hudi-flink-bundle` module pom.xml sets the scope related to hive as `provided` by default. If you want to use hive sync, you need to use the
+profile `flink-bundle-shade-hive` during packaging. Executing command below to install:
+
+```bash
+# Maven install command
+mvn install -DskipTests -Drat.skip=true -Pflink-bundle-shade-hive2
+
+# For hive1, you need to use profile -Pflink-bundle-shade-hive1
+# For hive3, you need to use profile -Pflink-bundle-shade-hive3
+```
+
+:::note
+Hive1.x can only synchronize metadata to hive, but cannot use hive query now. If you need to query, you can use spark to query hive table.
+:::
+
+:::note
+If using hive profile, you need to modify the hive version in the profile to your hive cluster version (Only need to modify the hive version in this profile).
+The location of this `pom.xml` is `packaging/hudi-flink-bundle/pom.xml`, and the corresponding profile is at the bottom of this file.
+:::
+
+### Hive Environment
+
+1. Import `hudi-hadoop-mr-bundle` into hive. Creating `auxlib/` folder under the root directory of hive, and moving `hudi-hadoop-mr-bundle-0.x.x-SNAPSHOT.jar` into `auxlib`.
+ `hudi-hadoop-mr-bundle-0.x.x-SNAPSHOT.jar` is at `packaging/hudi-hadoop-mr-bundle/target`.
+
+2. When Flink sql client connects hive metastore remotely, `hive metastore` and `hiveserver2` services need to be enabled, and the port number need to
+ be set correctly. Command to turn on the services:
+
+```bash
+# Enable hive metastore and hiveserver2
+nohup ./bin/hive --service metastore &
+nohup ./bin/hive --service hiveserver2 &
+
+# While modifying the jar package under auxlib, you need to restart the service.
+```
+
+### Sync Template
+
+Flink hive sync now supports two hive sync mode, `hms` and `jdbc`. `hms` mode only needs to configure metastore uris. For
+the `jdbc` mode, the JDBC attributes and metastore uris both need to be configured. The options template is as below:
+
+```sql
+-- hms mode template
+CREATE TABLE t1(
+ uuid VARCHAR(20),
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = '${db_path}/t1',
+ 'table.type' = 'COPY_ON_WRITE', -- If MERGE_ON_READ, hive query will not have output until the parquet file is generated
+ 'hive_sync.enable' = 'true', -- Required. To enable hive synchronization
+ 'hive_sync.mode' = 'hms', -- Required. Setting hive sync mode to hms, default hms. (Before 0.13, the default sync mode was jdbc.)
+ 'hive_sync.metastore.uris' = 'thrift://${ip}:9083' -- Required. The port need set on hive-site.xml
+);
+
+
+-- jdbc mode template
+CREATE TABLE t1(
+ uuid VARCHAR(20),
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = '${db_path}/t1',
+ 'table.type' = 'COPY_ON_WRITE', --If MERGE_ON_READ, hive query will not have output until the parquet file is generated
+ 'hive_sync.enable' = 'true', -- Required. To enable hive synchronization
+ 'hive_sync.mode' = 'jdbc', -- Required. Setting hive sync mode to jdbc, default hms. (Before 0.13, the default sync mode was jdbc.)
+ 'hive_sync.metastore.uris' = 'thrift://${ip}:9083', -- Required. The port need set on hive-site.xml
+ 'hive_sync.jdbc_url'='jdbc:hive2://${ip}:10000', -- required, hiveServer port
+ 'hive_sync.table'='${table_name}', -- required, hive table name
+ 'hive_sync.db'='${db_name}', -- required, hive database name
+ 'hive_sync.username'='${user_name}', -- required, JDBC username
+ 'hive_sync.password'='${password}' -- required, JDBC password
+);
+```
+
+### Query
+
+While using hive beeline query, you need to enter settings:
+```bash
+set hive.input.format = org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat;
+```
+
+## Spark Catalog Metastore Client
+
+When running Hudi inside a Spark environment that already has Hive support enabled (e.g., SparkSQL with `spark.sql.catalogImplementation=hive`), the standard `IMetaStoreClient` initialization can conflict with Spark's own Hive classloader. Setting
+
+```properties
+hoodie.datasource.hive_sync.use_spark_catalog=true
+```
+
+(default: `false`) makes Hudi use `SparkCatalogMetaStoreClient` β a Spark-native `IMetaStoreClient` implementation β instead of creating its own. This avoids classloader conflicts in Hive-on-Spark setups. Requires a `SparkSession` with Hive support active.
+
+## HMS 4.x Support via JDBC Fallback
+
+HMS 4.x changed several Thrift API method signatures (e.g., `get_table` β `get_table_req`), which makes the standard Thrift-based HMS client incompatible. Hudi 1.2.0 adds automatic fallback: when a Thrift metadata call surfaces a `TApplicationException` anywhere in its cause chain, Hudi flips an internal `thriftIncompatible` flag and reroutes the rest of that sync run through the JDBC path.
+
+**Requirement:** sync mode must be `jdbc` with a valid JDBC URL so the fallback client is available:
+
+```properties
+hoodie.datasource.hive_sync.mode=jdbc
+hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://hiveserver:10000
+hoodie.datasource.hive_sync.username=<username>
+hoodie.datasource.hive_sync.password=<password>
+```
Review Comment:
π€ The HMS 4.x JDBC fallback section says fallback only happens for callers that go through Thrift first, and the flag never resets within a `HoodieHiveSyncClient` instance. Should we explicitly note here whether long-running writers (e.g., Hudi Streamer continuous mode) will re-probe Thrift on each sync run, or whether the same sync-client instance is reused across runs and therefore stays in JDBC-fallback mode for the lifetime of the process? Users tuning continuous pipelines on HMS 4.x will want to know.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/cli.md:
##########
@@ -0,0 +1,791 @@
+---
+title: CLI
+keywords: [hudi, cli]
+last_modified_at: 2026-05-27T00:00:00-00:00
+---
+
+### Local set up
+Once hudi has been built, the shell can be fired by via `cd packaging/hudi-cli-bundle && hudi-cli-with-bundle.sh` or `packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh`.
+
+### Hudi CLI setup
+In release `0.13.0` we have now added new way of launching the `hudi cli`, which is using the `hudi-cli-bundle` script.
+
+#### Note: The traditional `hudi-cli.sh` script has been deprecated and replaced with `hudi-cli-with-bundle.sh` from `1.0.2` release onwards. Users should migrate to the new bundled CLI script `hudi-cli-with-bundle.sh` for better compatibility and ease of use.
+
+There are a couple of requirements such as having `spark` installed locally on your machine.
+It is required to use a spark distribution with hadoop dependencies packaged such as `spark-3.5.4-bin-hadoop3.tgz` from https://archive.apache.org/dist/spark/.
+We also recommend you set an env variable `$SPARK_HOME` to the path of where spark is installed on your machine.
+One important thing to note is that the `hudi-spark-bundle` should also be present when using the `hudi-cli-bundle`.
+To provide the locations of these bundle jars you can set them in your shell like so:
+`export CLI_BUNDLE_JAR=<path-to-cli-bundle-jar-to-use>` , `export SPARK_BUNDLE_JAR=<path-to-spark-bundle-jar-to-use>`.
+
+For steps see below if you are not compiling the project and downloading the jars:
+
+1. Create an empty folder as a new directory
+2. Copy the hudi-cli-bundle jars and hudi-spark*-bundle jars to this directory
+3. Copy the following script and folder to this directory
+```
+packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+packaging/hudi-cli-bundle/conf . the `conf` folder should be in this directory.
+```
+
+4. Start Hudi CLI shell with environment variables set
+```
+export SPARK_HOME=<spark-home-folder>
+export CLI_BUNDLE_JAR=<cli-bundle-jar-to-use>
+export SPARK_BUNDLE_JAR=<spark-bundle-jar-to-use>
+
+./hudi-cli-with-bundle.sh
+
+```
+
+### Base path
+A hudi table resides on DFS, in a location referred to as the `basePath` and
+we would need this location in order to connect to a Hudi table. Hudi library effectively manages this table internally, using `.hoodie` subfolder to track all metadata.
+
+
+
+
+### Using Hudi-cli in S3
+If you are using hudi that comes packaged with AWS EMR, you can find instructions to use hudi-cli [here](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-cli.html).
+If you are not using EMR, or would like to use latest hudi-cli from master, you can follow the below steps to access S3 dataset in your local environment (laptop).
+
+Build Hudi with corresponding Spark version, for eg, -Dspark3.5
+
+Set the following environment variables.
+```
+export AWS_REGION=us-east-2
+export AWS_ACCESS_KEY_ID=<key_id>
+export AWS_SECRET_ACCESS_KEY=<secret_key>
+
+export SPARK_HOME=<spark_home>
+export CLI_BUNDLE_JAR=<cli-bundle-jar-to-use>
+export SPARK_BUNDLE_JAR=<spark-bundle-jar-to-use>
+```
+Ensure you set the SPARK_HOME to your local spark home compatible to compiled hudi spark version above. One important thing to note is that the `hudi-spark-bundle` should also be present when using the `hudi-cli-bundle`.
+
+Apart from these, we might need to add aws jars to class path so that accessing S3 is feasible from local.
+We need two jars, namely, aws-java-sdk-bundle jar and hadoop-aws jar which you can find online.
+For eg:
+```
+wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -o /lib/spark-3.5.4-bin-hadoop3/jars/hadoop-aws-3.3.4.jar
+wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar -o /lib/spark-3.5.4-bin-hadoop3/jars/aws-java-sdk-bundle-1.12.262.jar
+```
+
+#### Note: These AWS jar versions below are specific to Spark 3.5.4 and Hadoop 3.3.4
+```
+export CLIENT_JAR=/lib/spark-3.5.4-bin-hadoop3/jars/aws-java-sdk-bundle-1.12.262.jar:/lib/spark-3.5.4-bin-hadoop3/jars/hadoop-aws-3.3.4.jar
+```
+Once these are set, you are good to launch hudi-cli and access S3 dataset.
+```
+./packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+```
+### Using hudi-cli on Google Dataproc
+[Dataproc](https://cloud.google.com/dataproc) is Google's managed service for running Apache Hadoop, Apache Spark,
+Apache Flink, Presto and many other frameworks, including Hudi. If you want to run the Hudi CLI on a Dataproc node
+which has not been launched with Hudi support enabled, you can use the steps below:
+
+These steps use Hudi version 1.2.0. If you want to use a different version you will have to edit the below commands
+appropriately:
+1. Once you've started the Dataproc cluster, you can ssh into it as follows:
+```
+$ gcloud compute ssh --zone "YOUR_ZONE" "HOSTNAME_OF_MASTER_NODE" --project "YOUR_PROJECT"
+```
+
+2. Download the Hudi CLI bundle
+```
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-cli-bundle_2.12/1.2.0/hudi-cli-bundle_2.12-1.2.0.jar
+```
+
+3. Download the Hudi Spark bundle
+```
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.5-bundle_2.12/1.2.0/hudi-spark3.5-bundle_2.12-1.2.0.jar
+```
+
+4. Download the shell script that launches Hudi CLI bundle
+```
+wget https://raw.githubusercontent.com/apache/hudi/release-1.2.0/packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+```
+
+5. Launch Hudi CLI bundle with appropriate environment variables as follows:
+```
+CLIENT_JAR=$DATAPROC_DIR/lib/gcs-connector.jar CLI_BUNDLE_JAR=hudi-cli-bundle_2.12-1.2.0.jar SPARK_BUNDLE_JAR=hudi-spark3.5-bundle_2.12-1.2.0.jar ./hudi-cli-with-bundle.sh
+```
+
+6. hudi->connect --path gs://path_to_some_table
+Metadata for table some_table loaded
+
+7. hudi:some_table->commits show --limit 5
+This command should show the recent commits, if the above steps work correctly.
+
+## Connect to a Kerberized cluster
+
+Before connecting to a Kerberized cluster, you can use **kerberos kinit** command. Following is the usage of this command.
+
+```shell
+hudi->help kerberos kinit
+NAME
+ kerberos kinit - Perform Kerberos authentication
+
+SYNOPSIS
+ kerberos kinit --krb5conf String [--principal String] [--keytab String]
+
+OPTIONS
+ --krb5conf String
+ Path to krb5.conf
+ [Optional, default = /etc/krb5.conf]
+
+ --principal String
+ Kerberos principal
+ [Mandatory]
+
+ --keytab String
+ Path to keytab
+ [Mandatory]
+```
+
+For example:
+
+```shell
+hudi->kerberos kinit --principal user/host@DOMAIN --keytab /etc/security/keytabs/user.keytab
+Perform Kerberos authentication
+Parameters:
+--krb5conf: /etc/krb5.conf
+--principal: user/host@DOMAIN
+--keytab: /etc/security/keytabs/user.keytab
+Kerberos current user: user/host@DOMAIN (auth:KERBEROS)
+Kerberos login user: user/host@DOMAIN (auth:KERBEROS)
+Kerberos authentication success
+```
+
+If you see "Kerberos authentication success" in the command output, it means Kerberos authentication has been successful.
+
+
+## Using hudi-cli
+
+To initialize a hudi table, use the following command.
+
+```java
+===================================================================
+* ___ ___ *
+* /\__\ ___ /\ \ ___ *
+* / / / /\__\ / \ \ /\ \ *
+* / /__/ / / / / /\ \ \ \ \ \ *
+* / \ \ ___ / / / / / \ \__\ / \__\ *
+* / /\ \ /\__\ / /__/ ___ / /__/ \ |__| / /\/__/ *
+* \/ \ \/ / / \ \ \ /\__\ \ \ \ / / / /\/ / / *
+* \ / / \ \ / / / \ \ / / / \ /__/ *
+* / / / \ \/ / / \ \/ / / \ \__\ *
+* / / / \ / / \ / / \/__/ *
+* \/__/ \/__/ \/__/ Apache Hudi CLI *
+* *
+===================================================================
+
+hudi->create --path /user/hive/warehouse/table1 --tableName hoodie_table_1 --tableType COPY_ON_WRITE
+.....
+```
+
+To see the description of hudi table, use the command:
+
+```java
+hudi:hoodie_table_1->desc
+18/09/06 15:57:19 INFO timeline.HoodieActiveTimeline: Loaded instants []
+ _________________________________________________________
+ | Property | Value |
+ |========================================================|
+ | basePath | ... |
+ | metaPath | ... |
+ | fileSystem | hdfs |
+ | hoodie.table.name | hoodie_table_1 |
+ | hoodie.table.type | COPY_ON_WRITE |
+ | hoodie.archivelog.folder| |
+```
+
+Following is a sample command to connect to a Hudi table contains uber trips.
+
+```java
+hudi:trips->connect --path /app/uber/trips
+
+16/10/05 23:20:37 INFO model.HoodieTableMetadata: All commits :HoodieCommits{commitList=[20161002045850, 20161002052915, 20161002055918, 20161002065317, 20161002075932, 20161002082904, 20161002085949, 20161002092936, 20161002105903, 20161002112938, 20161002123005, 20161002133002, 20161002155940, 20161002165924, 20161002172907, 20161002175905, 20161002190016, 20161002192954, 20161002195925, 20161002205935, 20161002215928, 20161002222938, 20161002225915, 20161002232906, 20161003003028, 20161003005958, 20161003012936, 20161003022924, 20161003025859, 20161003032854, 20161003042930, 20161003052911, 20161003055907, 20161003062946, 20161003065927, 20161003075924, 20161003082926, 20161003085925, 20161003092909, 20161003100010, 20161003102913, 20161003105850, 20161003112910, 20161003115851, 20161003122929, 20161003132931, 20161003142952, 20161003145856, 20161003152953, 20161003155912, 20161003162922, 20161003165852, 20161003172923, 20161003175923, 20161003195931, 20161003210118, 201610032129
19, 20161003215928, 20161003223000, 20161003225858, 20161004003042, 20161004011345, 20161004015235, 20161004022234, 20161004063001, 20161004072402, 20161004074436, 20161004080224, 20161004082928, 20161004085857, 20161004105922, 20161004122927, 20161004142929, 20161004163026, 20161004175925, 20161004194411, 20161004203202, 20161004211210, 20161004214115, 20161004220437, 20161004223020, 20161004225321, 20161004231431, 20161004233643, 20161005010227, 20161005015927, 20161005022911, 20161005032958, 20161005035939, 20161005052904, 20161005070028, 20161005074429, 20161005081318, 20161005083455, 20161005085921, 20161005092901, 20161005095936, 20161005120158, 20161005123418, 20161005125911, 20161005133107, 20161005155908, 20161005163517, 20161005165855, 20161005180127, 20161005184226, 20161005191051, 20161005193234, 20161005203112, 20161005205920, 20161005212949, 20161005223034, 20161005225920]}
+Metadata for table trips loaded
+```
+
+Once connected to the table, a lot of other commands become available. The shell has contextual autocomplete help (press TAB) and below is a list of all commands, few of which are reviewed in this section
+
+```shell
+hudi:trips->help
+* ! - Allows execution of operating system (OS) commands
+* // - Inline comment markers (start of line only)
+* ; - Inline comment markers (start of line only)
+* bootstrap index showmapping - Show bootstrap index mapping
+* bootstrap index showpartitions - Show bootstrap indexed partitions
+* bootstrap run - Run a bootstrap action for current Hudi table
+* clean showpartitions - Show partition level details of a clean
+* cleans refresh - Refresh table metadata
+* cleans run - run clean
+* cleans show - Show the cleans
+* clear - Clears the console
+* cls - Clears the console
+* clustering run - Run Clustering
+* clustering schedule - Schedule Clustering
+* clustering scheduleAndExecute - Run Clustering. Make a cluster plan first and execute that plan immediately
+* commit rollback - Rollback a commit
+* commits compare - Compare commits with another Hoodie table
+* commit show_write_stats - Show write stats of a commit
+* commit showfiles - Show file level details of a commit
+* commit showpartitions - Show partition level details of a commit
+* commits refresh - Refresh table metadata
+* commits show - Show the commits
+* commits showarchived - Show the archived commits
+* commits sync - Sync commits with another Hoodie table
+* compaction repair - Renames the files to make them consistent with the timeline as dictated by Hoodie metadata. Use when compaction unschedule fails partially.
+* compaction run - Run Compaction for given instant time
+* compaction schedule - Schedule Compaction
+* compaction scheduleAndExecute - Schedule compaction plan and execute this plan
+* compaction show - Shows compaction details for a specific compaction instant
+* compaction showarchived - Shows compaction details for a specific compaction instant
+* compactions show all - Shows all compactions that are in active timeline
+* compactions showarchived - Shows compaction details for specified time window
+* compaction unschedule - Unschedule Compaction
+* compaction unscheduleFileId - UnSchedule Compaction for a fileId
+* compaction validate - Validate Compaction
+* connect - Connect to a hoodie table
+* create - Create a hoodie table if not present
+* date - Displays the local date and time
+* desc - Describe Hoodie Table properties
+* downgrade table - Downgrades a table
+* exit - Exits the shell
+* export instants - Export Instants and their metadata from the Timeline
+* fetch table schema - Fetches latest table schema
+* hdfsparquetimport - Imports Parquet table to a hoodie table
+* help - List all commands usage
+* marker delete - Delete the marker
+* metadata create - Create the Metadata Table if it does not exist
+* metadata delete - Remove the Metadata Table
+* metadata init - Update the metadata table from commits since the creation
+* metadata list-files - Print a list of all files in a partition from the metadata
+* metadata list-partitions - List all partitions from metadata
+* metadata refresh - Refresh table metadata
+* metadata set - Set options for Metadata Table
+* metadata stats - Print stats about the metadata
+* metadata validate-files - Validate all files in all partitions from the metadata
+* quit - Exits the shell
+* refresh - Refresh table metadata
+* repair addpartitionmeta - Add partition metadata to a table, if not present
+* repair corrupted clean files - repair corrupted clean files
+* repair deduplicate - De-duplicate a partition path contains duplicates & produce repaired files to replace with
+* repair migrate-partition-meta - Migrate all partition meta file currently stored in text format to be stored in base file format. See HoodieTableConfig#PARTITION_METAFILE_USE_DATA_FORMAT.
+* repair overwrite-hoodie-props - Overwrite hoodie.properties with provided file. Risky operation. Proceed with caution!
+* savepoint create - Savepoint a commit
+* savepoint delete - Delete the savepoint
+* savepoint rollback - Savepoint a commit
+* savepoints refresh - Refresh table metadata
+* savepoints show - Show the savepoints
+* script - Parses the specified resource file and executes its commands
+* set - Set spark launcher env to cli
+* show archived commits - Read commits from archived files and show details
+* show archived commit stats - Read commits from archived files and show details
+* show env - Show spark launcher env by key
+* show envs all - Show spark launcher envs
+* show fsview all - Show entire file-system view
+* show fsview latest - Show latest file-system view
+* show logfile metadata - Read commit metadata from log files
+* show logfile records - Read records from log files
+* show rollback - Show details of a rollback instant
+* show rollbacks - List all rollback instants
+* stats filesizes - File Sizes. Display summary stats on sizes of files
+* stats wa - Write Amplification. Ratio of how many records were upserted to how many records were actually written
+* sync validate - Validate the sync by counting the number of records
+* system properties - Shows the shell's properties
+* table delete-configs - Delete the supplied table configs from the table.
+* table recover-configs - Recover table configs, from update/delete that failed midway.
+* table update-configs - Update the table configs with configs with provided file.
+* temp_delete - Delete view name
+* temp_query - query against created temp view
+* temp delete - Delete view name
+* temp query - query against created temp view
+* temps_show - Show all views name
+* temps show - Show all views name
+* upgrade table - Upgrades a table
+* utils loadClass - Load a class
+* version - Displays shell version
+
+hudi:trips->
+```
+
+
+### Inspecting Commits
+
+The task of upserting or inserting a batch of incoming records is known as a **commit** in Hudi. A commit provides basic atomicity guarantees such that only committed data is available for querying.
+Each commit has a monotonically increasing string/number called the **commit number**. Typically, this is the time at which we started the commit.
+
+To view some basic information about the last 10 commits,
+
+
+```java
+hudi:trips->commits show --sortBy "Total Bytes Written" --desc true --limit 10
+ ________________________________________________________________________________________________________________________________________________________________________
+ | CommitTime | Total Bytes Written| Total Files Added| Total Files Updated| Total Partitions Written| Total Records Written| Total Update Records Written| Total Errors|
+ |=======================================================================================================================================================================|
+ ....
+ ....
+ ....
+```
+
+At the start of each write, Hudi also writes a .inflight commit to the .hoodie folder. You can use the timestamp there to estimate how long the commit has been inflight
+
+
+```java
+$ hdfs dfs -ls /app/uber/trips/.hoodie/*.inflight
+-rw-r--r-- 3 vinoth supergroup 321984 2016-10-05 23:18 /app/uber/trips/.hoodie/20161005225920.inflight
+```
+
+To list all inflight and requested instants that have been running longer than a specified number of minutes, use `commits show_inflights`:
+
+```shell
+hudi:trips->commits show_inflights --lookbackInMins 30
+```
+
+This lists every inflight or requested instant whose requested timestamp is older than 30 minutes, showing the commit time, action type, and current state. This is useful for detecting hung or stuck writes. The `--lookbackInMins` option defaults to `0` (returns all inflight/requested instants).
+
+### Drilling Down to a specific Commit
+
+To understand how the writes spread across specific partiions,
+
+
+```java
+hudi:trips->commit showpartitions --commit 20161005165855 --sortBy "Total Bytes Written" --desc true --limit 10
+ __________________________________________________________________________________________________________________________________________
+ | Partition Path| Total Files Added| Total Files Updated| Total Records Inserted| Total Records Updated| Total Bytes Written| Total Errors|
+ |=========================================================================================================================================|
+ ....
+ ....
+```
+
+If you need file level granularity , we can do the following
+
+
+```java
+hudi:trips->commit showfiles --commit 20161005165855 --sortBy "Partition Path"
+ ________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File ID | Previous Commit| Total Records Updated| Total Records Written| Total Bytes Written| Total Errors|
+ |=======================================================================================================================================================|
+ ....
+ ....
+```
+
+
+### FileSystem View
+
+Hudi views each partition as a collection of file-groups with each file-group containing a list of file-slices in commit order (See concepts).
+The below commands allow users to view the file-slices for a data-set.
+
+```java
+hudi:stock_ticks_mor->show fsview all
+ ....
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta File Size| Delta Files |
+ |==============================================================================================================================================================================================================================================================================================================================================================================================================|
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]|
+
+
+
+hudi:stock_ticks_mor->show fsview latest --partitionPath "2018/08/31"
+ ......
+ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta Size| Delta Size - compaction scheduled| Delta Size - compaction unscheduled| Delta To Base Ratio - compaction scheduled| Delta To Base Ratio - compaction unscheduled| Delta Files - compaction scheduled | Delta Files - compaction unscheduled|
+ |=================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================|
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | 20.8 KB | 0.0 B | 0.0 B | 0.0 B | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]| [] |
+
+```
+
+
+### Statistics
+
+Since Hudi directly manages file sizes for DFS table, it might be good to get an overall picture
+
+
+```java
+hudi:trips->stats filesizes --partitionPath 2016/09/01 --sortBy "95th" --desc true --limit 10
+ ________________________________________________________________________________________________
+ | CommitTime | Min | 10th | 50th | avg | 95th | Max | NumFiles| StdDev |
+ |===============================================================================================|
+ | <COMMIT_ID> | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 2 | 2.3 KB |
+ ....
+ ....
+```
+
+In case of Hudi write taking much longer, it might be good to see the write amplification for any sudden increases
+
+
+```java
+hudi:trips->stats wa
+ __________________________________________________________________________
+ | CommitTime | Total Upserted| Total Written| Write Amplifiation Factor|
+ |=========================================================================|
+ ....
+ ....
+```
+
+
+### Archived Commits
+
+In order to limit the amount of growth of .commit files on DFS, Hudi archives older .commit files (with due respect to the cleaner policy) into a commits.archived file.
+This is a sequence file that contains a mapping from commitNumber => json with raw information about the commit (same that is nicely rolled up above).
+
+
+### Compactions
+
+To get an idea of the lag between compaction and writer applications, use the below command to list down all
+pending compactions.
+
+```java
+hudi:trips->compactions show all
+ ___________________________________________________________________
+ | Compaction Instant Time| State | Total FileIds to be Compacted|
+ |==================================================================|
+ | <INSTANT_1> | REQUESTED| 35 |
+ | <INSTANT_2> | INFLIGHT | 27 |
+```
+
+To inspect a specific compaction plan, use
+
+```java
+hudi:trips->compaction show --instant <INSTANT_1>
+ _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File Id | Base Instant | Data File Path | Total Delta Files| getMetrics |
+ |================================================================================================================================================================================================================================================
+ | 2018/07/17 | <UUID> | <INSTANT_1> | viewfs://ns-default/.../../UUID_<INSTANT>.parquet | 1 | {TOTAL_LOG_FILES=1.0, TOTAL_IO_READ_MB=1230.0, TOTAL_LOG_FILES_SIZE=2.51255751E8, TOTAL_IO_WRITE_MB=991.0, TOTAL_IO_MB=2221.0}|
+
+```
+
+To manually schedule or run a compaction, use the below command. This command uses spark launcher to perform compaction
+operations.
+
+**NOTE:** Make sure no other application is scheduling compaction for this table concurrently
+\{: .notice--info}
+
+```java
+hudi:trips->help compaction schedule
+Keyword: compaction schedule
+Description: Schedule Compaction
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: false
+ Default if specified: '__NULL__'
+ Default if unspecified: '1G'
+
+* compaction schedule - Schedule Compaction
+```
+
+```java
+hudi:trips->help compaction run
+Keyword: compaction run
+Description: Run Compaction for given instant time
+ Keyword: tableName
+ Help: Table name
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: parallelism
+ Help: Parallelism for hoodie compaction
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: schemaFilePath
+ Help: Path for Avro schema file
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: retry
+ Help: Number of retries
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: compactionInstant
+ Help: Base path for the target hoodie table
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+* compaction run - Run Compaction for given instant time
+```
+
+### Validate Compaction
+
+Validating a compaction plan : Check if all the files necessary for compactions are present and are valid
+
+```java
+hudi:stock_ticks_mor->compaction validate --instant 20181005222611
+...
+
+ COMPACTION PLAN VALID
+
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error|
+ |==========================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | true | |
+
+
+
+hudi:stock_ticks_mor->compaction validate --instant 20181005222601
+
+ COMPACTION PLAN INVALID
+
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error |
+ |=====================================================================================================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | false| All log files specified in compaction operation is not present. Missing .... |
+```
+
+**NOTE:** The following commands must be executed without any other writer/ingestion application running.
+\{: .notice--warning}
+
+Sometimes, it becomes necessary to remove a fileId from a compaction-plan inorder to speed-up or unblock compaction
+operation. Any new log-files that happened on this file after the compaction got scheduled will be safely renamed
+so that are preserved. Hudi provides the following CLI to support it
+
+
+### Unscheduling Compaction
+
+```java
+hudi:trips->compaction unscheduleFileId --fileId <FileUUID>
+....
+No File renames needed to unschedule file from pending compaction. Operation successful.
+```
+
+In other cases, an entire compaction plan needs to be reverted. This is supported by the following CLI
+
+```java
+hudi:trips->compaction unschedule --instant <compactionInstant>
+.....
+No File renames needed to unschedule pending compaction. Operation successful.
+```
+
+### Repair Compaction
+
+The above compaction unscheduling operations could sometimes fail partially (e:g -> DFS temporarily unavailable). With
+partial failures, the compaction operation could become inconsistent with the state of file-slices. When you run
+`compaction validate`, you can notice invalid compaction operations if there is one. In these cases, the repair
+command comes to the rescue, it will rearrange the file-slices so that there is no loss and the file-slices are
+consistent with the compaction plan
+
+```java
+hudi:stock_ticks_mor->compaction repair --instant 20181005222611
+......
+Compaction successfully repaired
+.....
+```
+
+### Savepoint and Restore
+As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery.md)
+
+To trigger savepoint for a hudi table
+```java
+connect --path /tmp/hudi_trips_cow/
+commits show
+set --conf SPARK_HOME=<SPARK_HOME>
+savepoint create --commit 20220128160245447 --sparkMaster local[2]
+```
+
+To restore the table to one of the savepointed commit:
+
+```java
+connect --path /tmp/hudi_trips_cow/
+commits show
Review Comment:
π€ This example still says `from version EIGHT(8) (current version) to SIX(6)`, which was accurate for the 1.0.x docs but is misleading inside the 1.2.0 snapshot β a fresh 1.2.0 table will be created at version NINE, not EIGHT. Consider updating the example to downgrade from NINE to a prior version so it reflects what a 1.2.0 user will actually see in `hoodie.properties`.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/cleaning.md:
##########
@@ -0,0 +1,206 @@
+---
+title: Cleaning
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+last_modified_at: 2026-05-27T00:00:00-00:00
+---
+## Background
+Cleaning is a table service employed by Hudi to reclaim space occupied by older versions of data and keep storage costs
+in check. Apache Hudi provides snapshot isolation between writers and readers by managing multiple versioned files with **MVCC**
+concurrency. These file versions provide history and enable time travel and rollbacks, but it is important to manage
+how much history you keep to balance your costs. Cleaning service plays a crucial role in manging the tradeoff between
+retaining long history of data and the associated storage costs.
+
+Hudi enables [Automatic Hudi cleaning](configurations.md#hoodiecleanautomatic) by default. Cleaning is invoked
+immediately after each commit, to delete older file slices. It's recommended to leave this enabled to ensure metadata
+and data storage growth is bounded. Cleaner can also be scheduled after every few commits instead of after every commit by
+configuring [hoodie.clean.max.commits](https://hudi.apache.org/docs/configurations#hoodiecleanmaxcommits).
+
+### Cleaning Retention Policies
+When cleaning old files, you should be careful not to remove files that are being actively used by long running queries.
+
+For spark based:
+
+| Config Name | Default | Description |
+|----------------------------------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
+| hoodie.clean.policy | KEEP_LATEST_COMMITS (Optional) | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. <br /><br />`Config Param: CLEANER_POLICY` |
+
+The corresponding config for Flink based engine is [`clean.policy`](https://hudi.apache.org/docs/configurations/#cleanpolicy).
+
+Hudi cleaner currently supports the below cleaning policies to keep a certain number of commits or file versions:
+
+- **KEEP_LATEST_COMMITS**: This is the default policy. This is a temporal cleaning policy that ensures the effect of
+ having lookback into all the changes that happened in the last X commits. Suppose a writer is ingesting data
+ into a Hudi dataset every 30 minutes and the longest running query can take 5 hours to finish, then the user should
+ retain atleast the last 10 commits. With such a configuration, we ensure that the oldest version of a file is kept on
Review Comment:
π€ The hyperlink for `hoodie.clean.commits.retained` points to `https://analytics.google.com/analytics/web/#/p300324801/reports/intelligenthome` β that looks like a Google Analytics dashboard URL pasted in by accident at some point. Since you're snapshotting docs for 1.2.0 it would be a good time to fix this; the link should point to `https://hudi.apache.org/docs/configurations/#hoodiecleancommitsretained`, matching the style used for the neighboring Flink config link two lines below.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/catalog_polaris.md:
##########
@@ -0,0 +1,94 @@
+---
+title: Apache Polaris (Incubating)
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+keywords: [hudi, polaris, catalog, integration]
+---
+
+:::warning Polaris Integration Status
Review Comment:
π€ The disclaimer says "Hudi 1.1.0 added support for Apache Polaris catalog integration β¦ However, a Polaris release that includes [this PR](https://github.com/apache/polaris/pull/1862) is pending before this integration to be available." Worth confirming this is still accurate as of 1.2.0 β if the upstream Polaris PR has since been released, this warning will silently mislead users into thinking the integration is unusable. If it's still pending, consider noting the expected Polaris release version so readers know what to wait for.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/deployment.md:
##########
@@ -0,0 +1,292 @@
+---
+title: Deployment
+keywords: [ hudi, administration, operation, devops, deployment]
+summary: This section offers an overview of tools available to operate an ecosystem of Hudi
+toc: true
+last_modified_at: 2026-05-27T00:00:00-00:00
+---
+
+This section provides all the help you need to deploy and operate Hudi tables at scale.
+Specifically, we will cover the following aspects.
+
+- [Deployment Model](#deploying) : How various Hudi components are deployed and managed.
+- [Upgrading Versions](#upgrading) : Picking up new releases of Hudi, guidelines and general best-practices.
+- [Downgrading Versions](#downgrading) : Reverting back to an older version of Hudi
+- [Migrating to Hudi](#migrating) : How to migrate your existing tables to Apache Hudi.
+
+## Deploying
+
+All in all, Hudi deploys with no long running servers or additional infrastructure cost to your data lake. In fact, Hudi pioneered this model of building a transactional distributed storage layer
+using existing infrastructure and its heartening to see other systems adopting similar approaches as well. Hudi writing is done via Spark jobs (Hudi Streamer or custom Spark datasource jobs), deployed per standard Apache Spark [recommendations](https://spark.apache.org/docs/latest/cluster-overview).
+Querying Hudi tables happens via libraries installed into Apache Hive, Apache Spark or PrestoDB and hence no additional infrastructure is necessary.
+
+A typical Hudi data ingestion can be achieved in 2 modes. In a single run mode, Hudi ingestion reads next batch of data, ingest them to Hudi table and exits. In continuous mode, Hudi ingestion runs as a long-running service executing ingestion in a loop.
+
+With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting delta files. Again, compaction can be performed in an asynchronous-mode by letting compaction run concurrently with ingestion or in a serial fashion with one after another.
+
+### Hudi Streamer
+
+[Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) is the standalone utility to incrementally pull upstream changes
+from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in two modes.
+
+To use Hudi Streamer in Spark, the `hudi-utilities-slim-bundle` and Hudi Spark bundle are required, by adding
+`--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.2.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.2.0` to the `spark-submit` command.
+
+Pick the Spark bundle that matches your Spark runtime β for example, `hudi-spark3.3-bundle_2.12`, `hudi-spark3.4-bundle_2.12`, `hudi-spark3.5-bundle_2.12` (Scala 2.12 or 2.13), `hudi-spark3.5-bundle_2.13`, `hudi-spark4.0-bundle_2.13`, or `hudi-spark4.1-bundle_2.13`. Spark 4.0 and 4.1 require Java 17 or later at runtime; Spark 3.x runs on Java 8 or later.
+
+- **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for every ingestion run and this can be changed by setting the property "hoodie.compact.inline.max.delta.commits". You can either manually run this spark application or use any cron trigger or workflow orchestrator (most common deployment strategy) such as Apache Airflow to spawn this application. See command line options in [this section](hoodie_streaming_ingestion.md#hudi-streamer) for running the spark application.
+
+Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
+
+```shell
+spark-submit \
+ --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.2.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.2.0 \
+ --master yarn \
+ --deploy-mode cluster \
+ --num-executors 10 \
+ --executor-memory 3g \
+ --driver-memory 6g \
+ --conf spark.driver.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_driver.hprof" \
+ --conf spark.executor.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_executor.hprof" \
+ --queue hadoop-platform-queue \
+ --conf spark.scheduler.mode=FAIR \
+ --conf spark.yarn.executor.memoryOverhead=1072 \
+ --conf spark.yarn.driver.memoryOverhead=2048 \
+ --conf spark.task.cpus=1 \
+ --conf spark.executor.cores=1 \
+ --conf spark.task.maxFailures=10 \
+ --conf spark.memory.fraction=0.4 \
+ --conf spark.rdd.compress=true \
+ --conf spark.kryoserializer.buffer.max=200m \
+ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+ --conf spark.memory.storageFraction=0.1 \
+ --conf spark.shuffle.service.enabled=true \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --conf spark.ui.port=5555 \
+ --conf spark.driver.maxResultSize=3g \
+ --conf spark.executor.heartbeatInterval=120s \
+ --conf spark.network.timeout=600s \
+ --conf spark.eventLog.overwrite=true \
+ --conf spark.eventLog.enabled=true \
+ --conf spark.eventLog.dir=hdfs:///user/spark/applicationHistory \
+ --conf spark.yarn.max.executor.failures=10 \
+ --conf spark.sql.catalogImplementation=hive \
+ --conf spark.sql.shuffle.partitions=100 \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
+```
+
+- **Continuous Mode** : Here, Hudi Streamer runs an infinite loop with each round performing one ingestion round as described in **Run Once Mode**. The frequency of data ingestion can be controlled by the configuration "--min-sync-interval-seconds". For Merge-On-Read tables, Compaction is run in asynchronous fashion concurrently with ingestion unless disabled by passing the flag "--disable-compaction". Every ingestion run triggers a compaction request asynchronously and this frequency can be changed by setting the property "hoodie.compact.inline.max.delta.commits". As both ingestion and compaction is running in the same spark context, you can use resource allocation configuration in Hudi Streamer CLI such as ("--delta-sync-scheduling-weight", "--compact-scheduling-weight", ""--delta-sync-scheduling-minshare", and "--compact-scheduling-minshare") to control executor allocation between ingestion and compaction.
+
+Here is an example invocation for reading from kafka topic in a continuous mode and writing to Merge On Read table type in a yarn cluster.
+
+```shell
+spark-submit \
+ --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.2.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.2.0 \
+ --master yarn \
+ --deploy-mode cluster \
+ --num-executors 10 \
+ --executor-memory 3g \
+ --driver-memory 6g \
+ --conf spark.driver.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_driver.hprof" \
+ --conf spark.executor.extraJavaOptions="-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/varadarb_ds_executor.hprof" \
+ --queue hadoop-platform-queue \
+ --conf spark.scheduler.mode=FAIR \
+ --conf spark.yarn.executor.memoryOverhead=1072 \
+ --conf spark.yarn.driver.memoryOverhead=2048 \
+ --conf spark.task.cpus=1 \
+ --conf spark.executor.cores=1 \
+ --conf spark.task.maxFailures=10 \
+ --conf spark.memory.fraction=0.4 \
+ --conf spark.rdd.compress=true \
+ --conf spark.kryoserializer.buffer.max=200m \
+ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+ --conf spark.memory.storageFraction=0.1 \
+ --conf spark.shuffle.service.enabled=true \
+ --conf spark.sql.hive.convertMetastoreParquet=false \
+ --conf spark.ui.port=5555 \
+ --conf spark.driver.maxResultSize=3g \
+ --conf spark.executor.heartbeatInterval=120s \
+ --conf spark.network.timeout=600s \
+ --conf spark.eventLog.overwrite=true \
+ --conf spark.eventLog.enabled=true \
+ --conf spark.eventLog.dir=hdfs:///user/spark/applicationHistory \
+ --conf spark.yarn.max.executor.failures=10 \
+ --conf spark.sql.catalogImplementation=hive \
+ --conf spark.sql.shuffle.partitions=100 \
+ --driver-class-path $HADOOP_CONF_DIR \
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer \
+ --table-type MERGE_ON_READ \
+ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
+ --source-ordering-field ts \
+ --target-base-path /user/hive/warehouse/stock_ticks_mor \
+ --target-table stock_ticks_mor \
+ --props /var/demo/config/kafka-source.properties \
+ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
+ --continuous
+```
+
+### Spark Datasource Writer Jobs
+
+As described in [Batch Writes](writing_data.md#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.delta.commits".
+
+Here is an example invocation using spark datasource
+
+```java
+inputDF.write()
+ .format("org.apache.hudi")
+ .options(clientOpts) // any of the Hudi client opts can be passed in as well
+ .option("hoodie.datasource.write.recordkey.field", "_row_key")
+ .option("hoodie.datasource.write.partitionpath.field", "partition")
+ .option("hoodie.table.ordering.fields", "timestamp")
+ .option("hoodie.table.name", tableName)
+ .mode(SaveMode.Append)
+ .save(basePath);
+```
+
+## Upgrading
+
+New Hudi releases are listed on the [releases page](/releases/download), with detailed notes which list all the changes, with highlights in each release.
+At the end of the day, Hudi is a storage system and with that comes a lot of responsibilities, which we take seriously.
+
+As general guidelines,
+
+ - We strive to keep all changes backwards compatible (i.e new code can read old data/timeline files) and when we cannot, we will provide upgrade/downgrade tools via the CLI
+ - We cannot always guarantee forward compatibility (i.e old code being able to read data/timeline files written by a greater version). This is generally the norm, since no new features can be built otherwise.
+ However any large such changes, will be turned off by default, for smooth transition to newer release. After a few releases and once enough users deem the feature stable in production, we will flip the defaults in a subsequent release.
+ - Always upgrade the query bundles (mr-bundle, presto-bundle, spark-bundle) first and then upgrade the writers (Hudi Streamer, spark jobs using datasource). This often provides the best experience and it's easy to fix
+ any issues by rolling forward/back the writer code (which typically you might have more control over)
+ - With large, feature rich releases we recommend migrating slowly, by first testing in staging environments and running your own tests. Upgrading Hudi is no different than upgrading any database system.
+
+Note that release notes can override this information with specific instructions, applicable on case-by-case basis.
+
+### Upgrading to 1.0.0
+
+1.0.0 is a major release with significant format changes. To ensure a smooth migration experience, we recommend the
+following steps:
+
+1. Stop any async table services in 0.x completely.
Review Comment:
π€ The 'Upgrading' section keeps the detailed 'Upgrading to 1.0.0' walkthrough but doesn't have anything about upgrading from 1.0.x or 1.1.x to 1.2.0. If 1.2.0 just continues the auto-upgrade path with no special steps, a one-line note here saying so would be reassuring; if there are any caveats specific to 1.2.0 (e.g., new MDT partitions, new configs that default-change), it would help to call them out. Happy to leave to a follow-up if release-notes-1.2.md will cover this.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/compaction.md:
##########
@@ -0,0 +1,290 @@
+---
+title: Compaction
+summary: "In this page, we describe async compaction in Hudi."
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+last_modified_at: 2026-05-27T00:00:00-00:00
+---
+## Background
+
+Compaction is a table service employed by Hudi specifically in Merge-on-Read (MOR) tables to merge updates from row-based log
+files to the corresponding columnar-based base file periodically to produce a new version of the base file. Compaction is
+not applicable to Copy-on-Write (COW) tables and only applies to MOR tables.
+
+### Why MOR tables need compaction?
+
+To understand the significance of compaction in MOR tables, it is helpful to understand the MOR table layout first. In Hudi,
+data is organized in terms of [file groups](https://hudi.apache.org/docs/file_layouts/). Each file group in a MOR table
+consists of a base file and one or more log files. Typically, during writes, inserts are stored in the base file, and updates
+are appended to log files.
+
+
+_Figure: MOR table file layout showing different file groups with base data file and log files_
+
+During the compaction process, updates from the log files are merged with the base file to form a new version of the
+base file as shown below. Since MOR is designed to be write-optimized, on new writes, after index tagging is complete,
+Hudi appends the records pertaining to each file groups as log blocks in log files. There is no synchronous merge
+happening during write, resulting in a lower write amplification and better write latency. In contrast, on new writes to a
+COW table, Hudi combines the new writes with the older base file to produce a new version of the base file resulting in
+a higher write amplification and higher write latencies.
+
+
+_Figure: Compaction on a given file group_
+
+While serving the read query(snapshot read), for each file group, records in base file and all its corresponding log
+files are merged together and served. And hence the read latency for MOR snapshot query might be higher compared to
+COW table since there is no merge involved in case of COW at read time. Compaction takes care of merging the updates from
+log files with the base file at regular intervals to bound the growth of log files and to ensure the read latencies do not
+spike up.
+
+## Compaction Architecture
+
+There are two steps to compaction.
+
+- **Compaction Scheduling**: In this step, Hudi scans the partitions and selects file slices to be compacted. A compaction plan is finally written to Hudi timeline.
+- **Compaction Execution**: In this step the compaction plan is read and file slices are compacted.
+
+### Strategies in Compaction Scheduling
+
+There are two strategies involved in scheduling the compaction:
+
+- Trigger Strategy: Determines how often to trigger scheduling of the compaction.
+- Compaction Strategy: Determines which file groups to compact.
+
+Hudi provides various options for both these strategies as discussed below.
+
+#### Incremental Scheduling
+
+Hudi supports incremental scheduling for compaction operations, which significantly improves performance on tables with a large number of partitions. Instead of scanning all partitions during each compaction scheduling run, incremental scheduling only processes partitions that have changed since the last completed compaction.
+
+This feature is enabled by default via `hoodie.table.services.incremental.enabled`. When enabled, compaction scheduling will:
+
+1. Identify partitions that have been modified since the last completed compaction by analyzing commit metadata within the time window between the last completed compaction and the current scheduling instant
+2. Include any partitions that were marked as missing from previous scheduling runs (e.g., due to IO limits)
+3. Only scan and process those incremental partitions
+4. Fall back to scanning all partitions if the last completed compaction instant cannot be found (e.g., due to archival) or if an exception occurs during incremental partition retrieval
+
+For tables with many partitions, this optimization can dramatically reduce scheduling overhead and improve overall job stability.
+
+#### Trigger Strategies
+
+| Config Name | Default | Description ||
+|----------------------------------------------------|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.compact.inline.trigger.strategy | NUM_COMMITS (Optional) | org.apache.hudi.table.action.compact.CompactionTriggerStrategy: Controls when compaction is scheduled.<br />`Config Param: INLINE_COMPACT_TRIGGER_STRATEGY` <br/>
+<ul><li>`NUM_COMMITS`: triggers compaction when there are at least N delta commits after last completed compaction.</li><li>`NUM_COMMITS_AFTER_LAST_REQUEST`: triggers compaction when there are at least N delta commits after last completed or requested compaction.</li><li>`TIME_ELAPSED`: triggers compaction after N seconds since last compaction.</li><li>`NUM_AND_TIME`: triggers compaction when both there are at least N delta commits and N seconds elapsed (both must be satisfied) after last completed compaction.</li><li>`NUM_OR_TIME`: triggers compaction when both there are at least N delta commits or N seconds elapsed (either condition is satisfied) after last completed compaction.</li></ul> |
+
+#### Compaction Strategies
+
+| Config Name | Default | Description |
+|----------------------------|---------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.compaction.strategy | org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy (Optional) | Compaction strategy decides which file groups are picked up for compaction during each compaction run. By default. Hudi picks the log file with most accumulated unmerged data. <br /><br />`Config Param: COMPACTION_STRATEGY` |
+
+Available Strategies (Provide the full package name when using the strategy): <ul><li>`LogFileNumBasedCompactionStrategy`:
+orders the compactions based on the total log files count, filters the file group with log files count greater than the
+threshold and limits the compactions within a configured IO bound.</li><li>`LogFileSizeBasedCompactionStrategy`: orders
+the compactions based on the total log files size, filters the file group which log files size is greater than the
+threshold and limits the compactions within a configured IO bound.</li><li>`BoundedIOCompactionStrategy`: CompactionStrategy
+which looks at total IO to be done for the compaction (read + write) and limits the list of compactions to be under a
+configured limit on the IO.</li><li>`BoundedPartitionAwareCompactionStrategy`:This strategy ensures that the last N partitions
+are picked up even if there are later partitions created for the table. lastNPartitions is defined as the N partitions before
+the currentDate. currentDay = 2018/01/01 The table has partitions for 2018/02/02 and 2018/03/03 beyond the currentDay This
+strategy will pick up the following partitions for compaction : (2018/01/01, allPartitionsInRange[(2018/01/01 - lastNPartitions)
+to 2018/01/01), 2018/02/02, 2018/03/03)</li><li>`DayBasedCompactionStrategy`:This strategy orders compactions in reverse
+order of creation of Hive Partitions. It helps to compact data in latest partitions first and then older capped at the
+Total_IO allowed.</li><li>`UnBoundedCompactionStrategy`: UnBoundedCompactionStrategy will not change ordering or filter
+any compaction. It is a pass-through and will compact all the base files which has a log file. This usually means
+no-intelligence on compaction.</li><li>`UnBoundedPartitionAwareCompactionStrategy`:UnBoundedPartitionAwareCompactionStrategy is a custom UnBounded Strategy. This will filter all the partitions that
+are eligible to be compacted by a \{@link BoundedPartitionAwareCompactionStrategy} and return the result. This is done
+so that a long running UnBoundedPartitionAwareCompactionStrategy does not step over partitions in a shorter running
+BoundedPartitionAwareCompactionStrategy. Essentially, this is an inverse of the partitions chosen in
+BoundedPartitionAwareCompactionStrategy</li></ul>
+
+:::note
+Please refer to [advanced configs](https://hudi.apache.org/docs/next/configurations#Compaction-Configs) for more details.
+:::
+
+#### Metadata Table Compaction Trigger Strategy
+
+Available since Hudi 1.2.0, the metadata table (MDT) supports the same set of compaction trigger strategies as the
+data table, plus a time-based option.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.metadata.compact.trigger.strategy` | `NUM_COMMITS` | Trigger strategy for MDT compaction. Accepts the same values as `hoodie.compact.inline.trigger.strategy`: `NUM_COMMITS`, `NUM_COMMITS_AFTER_LAST_REQUEST`, `TIME_ELAPSED`, `NUM_AND_TIME`, `NUM_OR_TIME`. |
+| `hoodie.metadata.compact.max.delta.commits` | `10` | Number of delta commits after the last MDT compaction before a new one is scheduled (for `NUM_COMMITS`-based strategies). |
+| `hoodie.metadata.compact.max.delta.seconds` | `7200` | Elapsed seconds after the last MDT compaction before scheduling a new one. Takes effect only for `TIME_ELAPSED`, `NUM_AND_TIME`, and `NUM_OR_TIME` strategies. |
+
+## Ways to trigger Compaction
+
+### Inline
+
+By default, compaction is run asynchronously.
+
+If latency of ingesting records is important for you, you are most likely using Merge-on-Read tables.
+Merge-on-Read tables store data using a combination of columnar (e.g parquet) + row based (e.g avro) file formats.
+Updates are logged to delta files & later compacted to produce new versions of columnar files.
+To improve ingestion latency, Async Compaction is the default configuration.
+
+If immediate read performance of a new commit is important for you, or you want simplicity of not managing separate compaction jobs,
+you may want synchronous inline compaction, which means that as a commit is written it is also compacted by the same job.
+
+For this deployment mode, please use `hoodie.compact.inline = true` for Spark Datasource and Spark SQL writers. For
+Hudi Streamer sync once mode inline compaction can be achieved by passing the flag `--disable-compaction` (Meaning to
+disable async compaction). Further in Hudi Streamer when both
+ingestion and compaction is running in the same spark context, you can use resource allocation configuration
+in Hudi Streamer CLI such as (`--delta-sync-scheduling-weight`,
+`--compact-scheduling-weight`, `--delta-sync-scheduling-minshare`, and `--compact-scheduling-minshare`)
+to control executor allocation between ingestion and compaction.
+
+### Async & Offline Compaction models
+
+There are a couple of ways here to trigger compaction .
+
+#### Async execution within the same process
+
+In streaming ingestion write models like Hudi Streamer
+continuous mode, Flink and Spark Streaming, async compaction is enabled by default and runs alongside without blocking
+regular ingestion.
+
+##### Spark Structured Streaming
+
+Compactions are scheduled and executed asynchronously inside the
+streaming job.Here is an example snippet in java
+
+```properties
+import org.apache.hudi.DataSourceWriteOptions;
+import org.apache.hudi.HoodieDataSourceHelpers;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import org.apache.spark.sql.streaming.OutputMode;
+import org.apache.spark.sql.streaming.ProcessingTime;
+ DataStreamWriter<Row> writer = streamingInput.writeStream().format("org.apache.hudi")
+ .option("hoodie.datasource.write.operation", operationType)
+ .option("hoodie.datasource.write.table.type", tableType)
+ .option("hoodie.datasource.write.recordkey.field", "_row_key")
+ .option("hoodie.datasource.write.partitionpath.field", "partition")
+ .option("hoodie.table.ordering.fields", "timestamp")
+ .option("hoodie.compact.inline.max.delta.commits", "10")
+ .option("hoodie.datasource.compaction.async.enable", "true")
+ .option("hoodie.table.name", tableName).option("checkpointLocation", checkpointLocation)
+ .outputMode(OutputMode.Append());
+ writer.trigger(new ProcessingTime(30000)).start(tablePath);
+```
+
+##### Hudi Streamer Continuous Mode
+
+Hudi Streamer provides continuous ingestion mode where a single long running spark application
+ingests data to Hudi table continuously from upstream sources. In this mode, Hudi supports managing asynchronous
+compactions. Here is an example snippet for running in continuous mode with async compactions
+
+```properties
+spark-submit --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.2.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.2.0 \
+--class org.apache.hudi.utilities.streamer.HoodieStreamer \
+--table-type MERGE_ON_READ \
+--target-base-path <hudi_base_path> \
+--target-table <hudi_table> \
+--source-class org.apache.hudi.utilities.sources.JsonDFSSource \
+--source-ordering-field ts \
+--props /path/to/source.properties \
+--continous
+```
+
+#### Scheduling and Execution by a separate process
+
+For some use cases with long running table services, instead of having the regular writes block, users have the option to run
+both steps of the compaction ([scheduling and execution](#compaction-architecture)) offline in a separate process altogether.
+This allows for regular writers to not bother about these compaction steps and allows users to provide more resources for
+the compaction job as needed.
+
+:::note
+This model needs a lock provider configured for all jobs - the regular writer as well as the offline compaction job.
+:::
+
+#### Scheduling inline and executing async
+
+In this model, it is possible for a Spark Datasource writer or a Flink job to just schedule the compaction inline ( that
+will serialize the compaction plan in the timeline but will not execute it). And then a separate utility like
+HudiCompactor or HoodieFlinkCompactor can take care of periodically executing the compaction plan.
+
+:::note
+This model may need a lock provider **if** metadata table is enabled.
+:::
+
+#### Hudi Compactor Utility
+
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli.md#compactions)
+The compactor utility allows to do scheduling and execution of compaction.
+
+Example:
+
+```properties
+spark-submit --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.2.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.2.0 \
+--class org.apache.hudi.utilities.HoodieCompactor \
+--base-path <base_path> \
+--table-name <table_name> \
+--schema-file <schema_file> \
+--instant-time <compaction_instant>
+```
+
+Note, the `instant-time` parameter is now optional for the Hudi Compactor Utility. If using the utility without `--instant time`,
+the spark-submit will execute the earliest scheduled compaction on the Hudi timeline.
+
+##### Available Options
+
+The HoodieCompactor utility supports the following retry and timeout options (effective in `scheduleAndExecute` mode):
+
+| Option Name | Short Flag | Default | Description |
+|--------------------------------|------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `--retry-last-failed-job` | `-rc` | `false` | When set to true, checks, rolls back, and executes the last failed compaction plan instead of planning a new compaction job directly. This is useful for recovering from previous failures. |
+| `--job-max-processing-time-ms` | `-jt` | `0` | Maximum processing time in milliseconds before considering a compaction job as failed. If this time is exceeded and the job is still unfinished, Hudi will consider the job as failed and relaunch it (when used with `--retry-last-failed-job`). A value of 0 or negative disables the timeout check. |
+
+:::note
+These retry options are only effective when using `--mode scheduleAndExecute`. The `--retry-last-failed-job` option requires `--job-max-processing-time-ms` to be set to a positive value to detect stale inflight instants.
+:::
+
+#### Hudi CLI
+
+Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli.md#compactions)
Review Comment:
π€ This Flink offline compaction example still references `hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar` / `hudi-flink-bundle_2.11-0.9.0.jar`, which is doubly stale: (1) the version is from 0.9.0 instead of 1.2.0, and (2) the artifact name no longer uses the `_2.11` Scala suffix β the current naming is `hudi-flink{flink.version}-bundle-{hudi.version}.jar` (e.g., `hudi-flink2.1-bundle-1.2.0.jar`). The PR description mentions bumping hardcoded examples to 1.2.0, so this one appears to have been missed. The clustering doc nearby just says `lib/hudi-flink-bundle.jar`, which sidesteps the issue if you want a simpler fix.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/reading_tables_streaming_reads.md:
##########
@@ -0,0 +1,143 @@
+---
+title: Streaming Reads
+keywords: [hudi, spark, flink, streaming, processing]
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Spark Streaming
+
+Structured Streaming reads are based on Hudi's Incremental Query feature, therefore streaming read can return data for which
+commits and base files were not yet removed by the cleaner. You can control commits retention time.
+
+<Tabs
+groupId="programming-language"
+defaultValue="python"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Python', value: 'python', },
+]}
+>
+
+<TabItem value="scala">
+
+```scala
+// spark-shell
+// reload data
+df.write.format("hudi").
+ options(getQuickstartWriteConfigs).
+ option("hoodie.table.ordering.fields", "ts").
+ option("hoodie.datasource.write.recordkey.field", "uuid").
+ option("hoodie.datasource.write.partitionpath.field", "partitionpath").
+ option("hoodie.table.name", tableName).
+ mode(Overwrite).
+ save(basePath)
+
+// read stream and output results to console
+spark.readStream.
+ format("hudi").
+ load(basePath).
+ writeStream.
+ format("console").
+ start()
+
+// read stream to streaming df
+val df = spark.readStream.
+ format("hudi").
+ load(basePath)
+
+```
+</TabItem>
+
+<TabItem value="python">
+
+```python
+# pyspark
+# reload data
+inserts = sc._jvm.org.apache.hudi.QuickstartUtils.convertToStringList(
+ dataGen.generateInserts(10))
+df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
+
+hudi_options = {
+ 'hoodie.table.name': tableName,
+ 'hoodie.datasource.write.recordkey.field': 'uuid',
+ 'hoodie.datasource.write.partitionpath.field': 'partitionpath',
+ 'hoodie.datasource.write.table.name': tableName,
+ 'hoodie.datasource.write.operation': 'upsert',
+ 'hoodie.table.ordering.fields': 'ts',
+ 'hoodie.upsert.shuffle.parallelism': 2,
+ 'hoodie.insert.shuffle.parallelism': 2
+}
+
+df.write.format("hudi"). \
+ options(**hudi_options). \
+ mode("overwrite"). \
+ save(basePath)
+
+# read stream to streaming df
+df = spark.readStream \
+ .format("hudi") \
+ .load(basePath)
+
+# ead stream and output results to console
+spark.readStream \
+ .format("hudi") \
+ .load(basePath) \
+ .writeStream \
+ .format("console") \
+ .start()
+
+```
+</TabItem>
Review Comment:
π€ Small typo in the comment: `# ead stream and output results to console` β `# read stream and output results to console`.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/indexes.md:
##########
@@ -0,0 +1,312 @@
+---
+title: Indexes
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+---
+
+In databases, indexes are auxiliary data structures maintained to quickly locate records needed, without reading unnecessary data
+from storage. Given that Hudiβs design has been heavily optimized for handling mutable change streams, with different
+write patterns, Hudi considers [indexing](#mapping-keys-to-file-groups) as an integral part of its design and has uniquely supported
+[indexing capabilities](https://hudi.apache.org/blog/2020/11/11/hudi-indexing-mechanisms/) from its inception, to speed
+up writes on the [data lakehouse](https://hudi.apache.org/blog/2024/07/11/what-is-a-data-lakehouse/), while still providing
+columnar query performance.
+
+## Mapping keys to file groups
+
+The most foundational index mechanism in Hudi tracks a mapping from a given key (record key + optionally partition path) consistently to a file id. Other types of indexes like secondary indexes,
+build on this foundation. This mapping between record key and file group/file id rarely changes once the first version of a record has been written to a file group.
+Only clustering or cross-partition updates that are implemented as deletes + inserts remap the record key to a different file group. Even then, a given record key is associated with exactly one
+file group at any completed instant on the timeline.
+
+## Need for indexing
+
+For [Copy-On-Write tables](table_types.md#copy-on-write-table), indexing enables fast upsert/delete operations, by avoiding the need to join against the entire dataset to determine which files to rewrite.
+For [Merge-On-Read tables](table_types.md#merge-on-read-table), indexing allows Hudi to bound the amount of change records any given base file needs to be merged against. Specifically, a given base file needs to be merged
+only against updates for records that are part of that base file.
+
+
+<p align = "center">Figure: Comparison of merge cost for updates (dark blue blocks) against base files (light blue blocks)</p>
+
+In contrast,
+
+- Designs without an indexing component (e.g: [Apache Hive/Apache Iceberg](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)) end up having to merge all the base files against all incoming updates/delete records
+ (10-100x more [read amplification](table_types.md#comparison)).
+- Designs that implement heavily write-optimized OLTP data structures like LSM trees do not require an indexing component. But they perform poorly scan heavy workloads
+ against cloud storage making them unsuitable for serving analytical queries.
+
+Hudi shines by achieving both great write performance and read performance, at the extra storage costs of an index, which can however unlock a lot more, as we explore below.
+
+## Multi-modal Indexing
+
+[Multi-modal indexing](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi),
+introduced in [0.11.0 Hudi release](https://hudi.apache.org/releases/release-0.11.0/#multi-modal-index),
+is a re-imagination of what a general purpose indexing subsystem should look like for the lake. Multi-modal indexing is
+implemented by enhancing the metadata table with the flexibility to extend to new index types as new partitions,
+along with an [asynchronous index](https://hudi.apache.org/docs/metadata_indexing/#setup-async-indexing) building
+
+Hudi supports a multi-modal index by augmenting the metadata table with the capability to incorporate new types of indexes, complemented by an
+asynchronous mechanism for [index construction](metadata_indexing.md). This enhancement supports a range of indexes within
+the [metadata table](metadata.md#metadata-table), significantly improving the efficiency of both writing to and reading from the table.
+
+
+<p align = "center">Figure: Indexes in Hudi</p>
+
+### Bloom Filters
+
+[Bloom filter](https://github.com/apache/hudi/blob/46f41d186c6c84a6af2c54a907ff2736b6013e15/rfc/rfc-37/rfc-37.md) indexes as *bloom_filter* partition in the metadata table.
+This index employs range-based pruning on the minimum and maximum values of the record keys and bloom-filter-based lookups to tag incoming records. For large tables, this
+involves reading the footers of all matching data files for bloom filters, which can be expensive in the case of random
+updates across the entire dataset. This index stores bloom filters of all data files centrally to avoid scanning the
+footers directly from all data files.
+
+Following are configurations that control enabling and configuring bloom filters.
+
+| Config Name | Default | Description |
+|-----------------------------------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.metadata.index.bloom.filter.enable | false | Enable indexing bloom filters of user data files under metadata table. When enabled, metadata table will have a partition to store the bloom filter index and will be used during the index lookups.<br />`Config Param: ENABLE_METADATA_INDEX_BLOOM_FILTER`<br />`Since Version: 0.11.0` |
+| hoodie.bloom.index.prune.by.ranges | true | Only applies if index type is BLOOM. When true, range information from files to leverage speed up index lookups. Particularly helpful, if the key has a monotonically increasing prefix, such as timestamp. If the record key is completely random, it is better to turn this off, since range pruning will only add an extra overhead to the index lookup.<br />`Config Param: BLOOM_INDEX_PRUNE_BY_RANGES` |
+| hoodie.bloom.index.update.partition.path | true | Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition<br />`Config Param: BLOOM_INDEX_UPDATE_PARTITION_PATH_ENABLE` |
+| hoodie.bloom.index.use.metadata | false | Only applies if index type is BLOOM.When true, the index lookup uses bloom filters and column stats from metadata table when available to speed up the process.<br />`Config Param: BLOOM_INDEX_USE_METADATA`<br />`Since Version: 0.11.0` |
+| hoodie.metadata.index.bloom.filter.column.list | (N/A) | Comma-separated list of columns for which bloom filter index will be built. If not set, only record key will be indexed.<br />`Config Param: BLOOM_FILTER_INDEX_FOR_COLUMNS`<br />`Since Version: 0.11.0` |
+| hoodie.metadata.index.bloom.filter.file.group.count | 4 | Metadata bloom filter index partition file group count. This controls the size of the base and log files and read parallelism in the bloom filter index partition. The recommendation is to size the file group count such that the base files are under 1GB.<br />`Config Param: METADATA_INDEX_BLOOM_FILTER_FILE_GROUP_COUNT`<br />`Since Version: 0.11.0` |
+
+### Record Index
+
+The record index is stored in the `record_index/` partition in the metadata table.
+Contains the mapping of the record key to location. This index aids in locating records faster than
+other existing indexes and can provide a speedup orders of magnitude faster in large deployments where index lookup dominates write latencies. To accommodate very high scales, it utilizes hash-based
+sharding of the key space. Additionally, when it comes to reading data, the index allows for point lookups significantly speeding up index mapping retrieval process.
+
+Hudi supports two variants of the Record Index:
+
+- **Global Record Index**: Enforces key uniqueness across all partitions in the table. This is the default variant introduced in 0.14.0.
+- **Partitioned Record Index**: Guarantees uniqueness for partition path and record key pairs. This variant speeds up lookups in very large partitioned datasets by limiting index lookups to the relevant partitions. Introduced in 1.1.0.
+
+Following are configurations that control enabling record index building and maintenance on the writer.
+
+| Config Name | Default | Description |
+|---------------------------------------------------------------|------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.metadata.record.index.enable | false | Create the HUDI Global Record Index within the Metadata Table (deprecated in favor of `hoodie.metadata.global.record.level.index.enable`)<br />`Config Param: RECORD_INDEX_ENABLE_PROP`<br />`Since Version: 0.14.0` |
+| hoodie.metadata.global.record.level.index.enable | false | Enable global Record Index within the metadata table. When enabled, enforces key uniqueness across all partitions in the table.<br />`Config Param: GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP`<br />`Since Version: 0.14.0` |
+| hoodie.metadata.record.level.index.enable | false | Enable partitioned Record Index within the metadata table. When enabled, guarantees uniqueness for partition path and record key pairs, speeding up lookups in very large partitioned datasets.<br />`Config Param: RECORD_LEVEL_INDEX_ENABLE_PROP`<br />`Since Version: 1.1.0` |
+| hoodie.metadata.record.index.growth.factor | 2.0 | The current number of records are multiplied by this number when estimating the number of file groups to create automatically. This helps account for growth in the number of records in the dataset.<br />`Config Param: RECORD_INDEX_GROWTH_FACTOR_PROP`<br />`Since Version: 0.14.0` |
+| hoodie.metadata.record.index.max.filegroup.count | 10000 | Maximum number of file groups to use for Record Index.<br />`Config Param: RECORD_INDEX_MAX_FILE_GROUP_COUNT_PROP`<br />`Since Version: 0.14.0` |
+| hoodie.metadata.record.index.max.filegroup.size | 1073741824 | Maximum size in bytes of a single file group. Large file group takes longer to compact.<br />`Config Param: RECORD_INDEX_MAX_FILE_GROUP_SIZE_BYTES_PROP`<br />`Since Version: 0.14.0` |
+| hoodie.metadata.record.index.min.filegroup.count | 10 | Minimum number of file groups to use for Global Record Index.<br />`Config Param: RECORD_INDEX_MIN_FILE_GROUP_COUNT_PROP`<br />`Since Version: 0.14.0`<br />**Note:** For global record index, use `hoodie.metadata.global.record.level.index.min.filegroup.count` instead. |
+| hoodie.metadata.global.record.level.index.min.filegroup.count | 10 | Minimum number of file groups to use for Global Record Index.<br />`Config Param: GLOBAL_RECORD_LEVEL_INDEX_MIN_FILE_GROUP_COUNT_PROP`<br />`Since Version: 0.14.0` |
+| hoodie.metadata.global.record.level.index.max.filegroup.count | 10000 | Maximum number of file groups to use for Global Record Index.<br />`Config Param: GLOBAL_RECORD_LEVEL_INDEX_MAX_FILE_GROUP_COUNT_PROP`<br />`Since Version: 0.14.0` |
+| hoodie.metadata.record.level.index.min.filegroup.count | 1 | Minimum number of file groups to use for Partitioned Record Index.<br />`Config Param: RECORD_LEVEL_INDEX_MIN_FILE_GROUP_COUNT_PROP`<br />`Since Version: 1.1.0` |
+| hoodie.metadata.record.level.index.max.filegroup.count | 10 | Maximum number of file groups to use for Partitioned Record Index.<br />`Config Param: RECORD_LEVEL_INDEX_MAX_FILE_GROUP_COUNT_PROP`<br />`Since Version: 1.1.0` |
+| hoodie.record.index.update.partition.path | false | Similar to Key: 'hoodie.bloom.index.update.partition.path' , default: true , isAdvanced: true , description: Only applies if index type is GLOBAL_BLOOM. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition since version: version is not defined deprecated after: version is not defined, but for record index.<br />`Config Param: RECORD_INDEX_UPDATE_PARTITION_PATH_ENABLE`<br />`Since Version: 0.14.0` |
+
+### Expression Index
+
+An [expression index](https://github.com/apache/hudi/blob/3789840be3d041cbcfc6b24786740210e4e6d6ac/rfc/rfc-63/rfc-63.md) is an index on a function of a column. If a query has a predicate on a function of a column, the expression index can
+be used to speed up the query. Expression index is stored in *expr_index_* prefixed partitions (one for each
+expression index) under metadata table. Expression index can be created using SQL syntax. Please check out SQL DDL
+docs [SQL DDL docs](sql_ddl.md#create-expression-index) for more details.
+
+Following are configurations that control enabling expression index building and maintenance on the writer.
+
+| Config Name | Default | Description |
+|---------------------------------------------------|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.metadata.index.expression.enable | false | Enable expression index within the metadata table. When this configuration property is enabled (`true`), the Hudi writer automatically keeps all expression indexes consistent with the data table. When disabled (`false`), all expression indexes are deleted. Note that individual expression index can only be created through a `CREATE INDEX` and deleted through a `DROP INDEX` statement in Spark SQL.<br />`Config Param: EXPRESSION_INDEX_ENABLE_PROP`<br />`Since Version: 1.0.0` |
+| hoodie.metadata.index.expression.file.group.count | 2 | Metadata expression index partition file group count.<br />`Config Param: EXPRESSION_INDEX_FILE_GROUP_COUNT`<br />`Since Version: 1.0.0` |
+| hoodie.expression.index.function | (N/A) | Function to be used for building the expression index.<br />`Config Param: INDEX_FUNCTION`<br />`Since Version: 1.0.0` |
+| hoodie.expression.index.name | (N/A) | Name of the expression index. This is also used for the partition name in the metadata table.<br />`Config Param: INDEX_NAME`<br />`Since Version: 1.0.0` |
+| hoodie.expression.index.type | COLUMN_STATS | Type of the expression index. Default is `column_stats` if there are no functions and expressions in the command. Valid options could be BITMAP, COLUMN_STATS, LUCENE, etc. If index_type is not provided, and there are functions or expressions in the command then a expression index using column stats will be created.<br />`Config Param: INDEX_TYPE`<br />`Since Version: 1.0.0` |
+
+### Secondary Index
+
+Secondary indexes allow users to create indexes on columns that are not part of record key columns in Hudi tables (for
+record key fields, Hudi supports [Record-level Index](/blog/2023/11/01/record-level-index). Secondary indexes
+can be used to speed up queries with predicate on columns other than record key columns.
+
+Following are configurations that control enabling secondary index building and maintenance on the writer.
+
+| Config Name | Default | Description |
+|---------------------------------------------|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.metadata.index.secondary.enable | true (Optional) | Enable secondary index within the metadata table. When this configuration property is enabled (`true`), the Hudi writer automatically keeps all secondary indexes consistent with the data table. When disabled (`false`), all secondary indexes are deleted. Note that individual secondary index can only be created through a `CREATE INDEX` and deleted through a `DROP INDEX` statement in Spark SQL. <br />`Config Param: SECONDARY_INDEX_ENABLE_PROP`<br />`Since Version: 1.0.0` |
+| hoodie.datasource.write.secondarykey.column | (N/A) | Columns that constitute the secondary key component. Actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`<br />`Config Param: SECONDARYKEY_COLUMN_NAME` |
+
+## Additional writer-side indexes
+
+All the indexes discussed above are available both readers/writers using integration with metadata table. There are also indexing mechanisms
+implemented by the storage engine, by efficiently reading/joining/processing incoming input records against information stored in base/log
+files themselves (e.g. bloom filters stored in parquet file footers) or intelligent data layout (e.g. bucket index).
+
+Currently, Hudi supports the following index types. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java
+engines. Writers can pick one of these options using `hoodie.index.type` config option.
+
+- **SIMPLE (default for Spark & Java engines)**: This is the standard index type for the Spark engine. It executes an efficient join of incoming records with keys retrieved from the table stored on disk. It requires keys to be partition-level unique so it can function correctly.
+
+- **RECORD_INDEX (deprecated)**: Use the global record index from section above as the writer side index. This was introduced in 0.14.0. **Deprecated** - use `GLOBAL_RECORD_LEVEL_INDEX` for global uniqueness or `RECORD_LEVEL_INDEX` for partition-level uniqueness instead.
+
+- **RECORD_LEVEL_INDEX** : Use the partitioned record index as the writer side index. This variant guarantees uniqueness for partition path and record key pairs and is optimized for large partitioned datasets. Available from 1.1.0.
+
+- **GLOBAL_RECORD_LEVEL_INDEX** : Use the global record index as the writer side index. This variant enforces key uniqueness across all partitions in the table. Available from 1.1.0.
+
+- **BLOOM**: Uses bloom filters generated from record keys, with the option to further narrow down candidate files based on the ranges of the record keys. It requires keys to be partition-level unique so it can function correctly.
+
+- **GLOBAL_BLOOM**: Utilizes bloom filters created from record keys, and may also refine the selection of candidate files by using the ranges of record keys. It requires keys to be table/global-level unique so it can function correctly.
+
+- **GLOBAL_SIMPLE**: Performs a lean join of the incoming records against keys extracted from the table on storage. It requires keys to be table/global-level unique so it can function correctly.
+
+- **HBASE**: Manages the index mapping through an external table in Apache HBase.
+
+- **INMEMORY**: Uses in-memory hashmap in Spark and Java engine and Flink in-memory state in Flink for indexing. Note that this is an alias for `FLINK_STATE` when used for Flink writers.
+
+- **FLINK_STATE (default for Flink)**: Uses the Flink state backend to store the index data: mappings of record keys to their residing file group's file IDs.
+
+- **BUCKET**: Utilizes bucket hashing to identify the file group that houses the records, which proves to be particularly advantageous on a large scale. The bucket index has three variants based on how buckets are configured and managed:
+
+ - **Simple Bucket Index (default)**: Employs a fixed number of buckets across all partitions. The bucket count is immutable once set and cannot increase or decrease. Applicable to both COW and MOR tables. Set via `hoodie.index.bucket.engine=SIMPLE` and `hoodie.bucket.index.num.buckets`. Due to the uniform bucket count across all partitions, this may not be ideal for tables with varying partition sizes or data skew.
+
+ - **Partition-Level Bucket Index**: Allows different fixed bucket counts for different partitions based on regex pattern matching. Existing simple bucket index tables can be upgraded to partition-level using the Spark `partition_bucket_index_manager` procedure, which rescales affected partitions. After upgrade, writers (Flink/Spark) automatically load partition-specific bucket configurations from table metadata. This addresses the limitation of uniform bucket counts while maintaining immutable buckets per partition. Applicable to both COW and MOR tables. Configure via `hoodie.index.bucket.engine=SIMPLE`, `hoodie.bucket.index.partition.expressions`, and `hoodie.bucket.index.partition.rule.type=regex`.
+
+ - **Consistent Hashing Bucket Index**: Accommodates a dynamic number of buckets with automatic resizing capability via clustering. Starts with an initial bucket count and can grow or shrink within configured min/max bounds based on file sizes. This addresses data skew in high-volume partitions by allowing dynamic resizing. Flink can schedule clustering plans, but execution currently requires Spark. Exclusively compatible with MOR tables. Configure via `hoodie.index.bucket.engine=CONSISTENT_HASHING`, `hoodie.bucket.index.num.buckets` (initial count), `hoodie.bucket.index.min.num.buckets`, and `hoodie.bucket.index.max.num.buckets`.
+
+- **Bring your own implementation:** You can extend this [public API](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndex.java)
+ and supply a subclass of `SparkHoodieIndex` (for Apache Spark writers) using `hoodie.index.class` to implement custom indexing.
+
+### Global and Non-Global Indexes
+
+Another key aspect worth understanding is the difference between global and non-global indexes. Both bloom and simple index have
+global options - `hoodie.index.type=GLOBAL_BLOOM` and `hoodie.index.type=GLOBAL_SIMPLE` - respectively. Record index supports both global and partitioned variants:
+
+- Global Record Index (`RECORD_INDEX` or `GLOBAL_RECORD_LEVEL_INDEX`): Enforces uniqueness across all partitions
+- Partitioned Record Index (`RECORD_LEVEL_INDEX`): Enforces uniqueness within each partition
+
+HBase index is by nature a global index.
+
+- **Global index:** Global indexes enforce uniqueness of keys across all partitions of a table i.e., guarantees that exactly
+ one record exists in the table for a given record key. Global indexes offer stronger guarantees, but the update/delete
+ cost can still grow with size of the table `O(size of table)`, since the record could belong to any partition in storage.
+ In case of non-global index, lookup involves file groups only for the matching partitions from the incoming records and
+ so it is not impacted by the total size of the table. These global indexes(GLOBAL_SIMPLE or GLOBAL_BLOOM), might be
+ acceptable for decent sized tables, but for large tables, a newly added index (0.14.0) called Record Level Index (RLI),
+ can offer pretty good index lookup performance compared to other global indexes(GLOBAL_SIMPLE or GLOBAL_BLOOM) or
+ Hbase and also avoids the operational overhead of maintaining external systems.
+- **Non Global index:** On the other hand, the default index implementations enforce this constraint only within a specific partition.
+ As one might imagine, non global indexes depend on the writer to provide the same consistent partition path for a given record key during update/delete,
+ but can deliver much better performance since the index lookup operation becomes `O(number of records updated/deleted)` and
+ scales well with write volume.
+
+### Configs
+
+#### Spark based configs
+
+For Spark DataSource, Spark SQL, Hudi Streamer and Structured Streaming following are the key configs that control
+indexing behavior. Please refer to [Advanced Configs](https://hudi.apache.org/docs/next/configurations#Common-Index-Configs-advanced-configs)
+for more details. All these, support the index types mentioned [above](#additional-writer-side-indexes).
+
+| Config Name | Default | Description |
+|-------------------------------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.index.type | N/A **(Required)** | org.apache.hudi.index.HoodieIndex$IndexType: Determines how input records are indexed, i.e., looked up based on the key for the location in the existing table. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java engines. Possible Values: <br /> <ul><li>BLOOM</li><li>GLOBAL_BLOOM</li><li>SIMPLE</li><li>GLOBAL_SIMPLE</li><li>HBASE</li><li>INMEMORY</li><li>FLINK_STATE</li><li>BUCKET</li><li>RECORD_INDEX</li><li>RECORD_LEVEL_INDEX</li><li>GLOBAL_RECORD_LEVEL_INDEX</li></ul><br />`Config Param: INDEX_TYPE` |
+| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> <br />`Config Param: BUCKET_INDEX_ENGINE_TYPE`<br />`Since Version: 0.11.0` |
+| hoodie.index.class | (Optional) | Full path of user-defined index class and must be a subclass of HoodieIndex class. It will take precedence over the hoodie.index.type configuration if specified<br /><br />`Config Param: INDEX_CLASS_NAME` |
+| hoodie.simple.index.update.partition.path | true (Optional) | Similar to Key: 'hoodie.bloom.index.update.partition.path' , Only applies if index type is GLOBAL_SIMPLE. When set to true, an update including the partition path of a record that already exists will result in inserting the incoming record into the new partition and deleting the original record in the old partition. When set to false, the original record will only be updated in the old partition, ignoring the new incoming partition if there is a mis-match between partition value for an incoming record with what's in storage. <br /><br />`Config Param: SIMPLE_INDEX_UPDATE_PARTITION_PATH_ENABLE` |
+| hoodie.hbase.index.update.partition.path | false (Optional) | Only applies if index type is HBASE. When an already existing record is upserted to a new partition compared to what's in storage, this config when set, will delete old record in old partition and will insert it as new record in new partition.<br /><br />`Config Param: UPDATE_PARTITION_PATH_ENABLE` |
+
+#### Flink based configs
+
+For Flink DataStream and Flink SQL, Bucket index, Flink state index, and record-level index are supported.
+Following are the basic configs that control the indexing behavior. Please refer [the Flink configurations](configurations.md#Flink-Options-advanced-configs) for advanced configs.
+
+| Config Name | Default | Description |
+|-------------------------------------------------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| index.type | FLINK_STATE (Optional) | Index type of Flink write job, default is using state backed index. Possible values:<br /> <ul><li>FLINK_STATE</li><li>BUCKET</li><li>GLOBAL_RECORD_LEVEL_INDEX</li><li>RECORD_LEVEL_INDEX</li></ul><br /> `Config Param: INDEX_TYPE` |
+| hoodie.index.bucket.engine | SIMPLE (Optional) | org.apache.hudi.index.HoodieIndex$BucketIndexEngineType: Determines the type of bucketing or hashing to use when `hoodie.index.type` is set to `BUCKET`. Possible Values: <br /> <ul><li>SIMPLE</li><li>CONSISTENT_HASHING</li></ul> |
+| metadata.enabled | true (Optional) | Enables the metadata table. Required for Flink record-level index lookups. |
+| index.global.enabled | true (Optional) | Whether to update the old partition path when the same record key arrives with a different partition path. This must be `true` for `GLOBAL_RECORD_LEVEL_INDEX` and is set to `false` for `RECORD_LEVEL_INDEX`. |
+| index.bootstrap.enabled | false (Optional) | When `index.type=GLOBAL_RECORD_LEVEL_INDEX`, controls whether Flink bootstraps the global index into a local RocksDB backend. If not explicitly set for global RLI, Flink enables bootstrap by default. Set to `false` to force native metadata-table RLI access. |
+| index.bootstrap.rocksdb.path | (Optional) | Local directory path for the RocksDB backend used when `index.bootstrap.enabled=true`. Each task manager creates a unique subdirectory under this path. |
+| index.rli.cache.size | 256 (Optional) | Maximum memory, in MB, allocated for the record-level index cache per bucket-assign task. Applies to native metadata-table RLI access and partitioned RLI caches. |
+| index.rli.cache.concurrent.partitions.num | 2 (Optional) | Expected number of partitions whose partitioned RLI caches are updated concurrently. Used to size each partition cache when historical cache usage is unavailable. |
+| index.rli.lookup.minibatch.size | 1000 (Optional) | Maximum number of input records buffered for mini-batch record-index lookup. Mini-batching reduces individual metadata-table lookup calls for native global RLI access. |
+| index.rli.write.buffer.size | 100 (Optional) | Maximum memory, in MB, for the index record writer buffer. When the threshold is reached, Flink flushes index records to avoid OOM. |
+| index.write.tasks | (N/A) | Parallelism for tasks that write record-level index records. Defaults to the execution environment parallelism when not set. |
+| metadata.compaction.schedule.enabled | true (Optional) | Schedules metadata table compaction plans. |
+| metadata.compaction.async.enabled | true (Optional) | Runs metadata table compaction in the Flink compaction pipeline when record-level index streaming writes are enabled. |
+| metadata.compaction.delta_commits | 10 (Optional) | Maximum metadata-table delta commits before metadata compaction is triggered. |
+| hoodie.metadata.record.level.index.defer.init | false (Optional) | Defers RLI initialization for fresh tables. Flink ingestion does not support deferred RLI initialization, so keep this set to `false` for Flink RLI writes. |
+| hoodie.metadata.global.record.level.index.min.filegroup.count | 10 (Optional) | Minimum number of file groups to use for Global Record Index. |
+| hoodie.metadata.global.record.level.index.max.filegroup.count | 10000 (Optional) | Maximum number of file groups to use for Global Record Index. |
+| hoodie.metadata.record.level.index.min.filegroup.count | 1 (Optional) | Minimum number of file groups to use for Partitioned Record Index. New data partitions use this value for their initial partitioned RLI file group count, which is also used by dynamic bucket assignment before the partition appears in the metadata table. |
+| hoodie.metadata.record.level.index.max.filegroup.count | 10 (Optional) | Maximum number of file groups to use for Partitioned Record Index. |
+
+Common Flink RLI configurations:
+
+| Use case | Required settings | Notes |
+|----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Flink global RLI with native MDT access | `index.type=GLOBAL_RECORD_LEVEL_INDEX`<br />`metadata.enabled=true`<br />`index.global.enabled=true`<br />`index.bootstrap.enabled=false`<br />`hoodie.metadata.record.level.index.defer.init=false` | Flink reads global record locations directly from the metadata table and uses the in-task RLI cache for recently accessed keys. Use this when a shared metadata-table index is preferred over task-local RocksDB state. |
+| Flink global RLI with local RocksDB cache | `index.type=GLOBAL_RECORD_LEVEL_INDEX`<br />`metadata.enabled=true`<br />`index.global.enabled=true`<br />`index.bootstrap.enabled=true`<br />`index.bootstrap.rocksdb.path=<local-path>`<br />`hoodie.metadata.record.level.index.defer.init=false` | Flink bootstraps the global RLI into a local RocksDB backend. This can reduce repeated metadata-table lookups at the cost of local disk usage and bootstrap time. |
+| Dynamic bucket scaling with partitioned RLI | `index.type=RECORD_LEVEL_INDEX`<br />`metadata.enabled=true`<br />`index.global.enabled=false`<br />`hoodie.metadata.record.level.index.min.filegroup.count=<initial-file-groups-per-partition>`<br />`hoodie.metadata.record.level.index.max.filegroup.count=<max-file-groups-per-partition>`<br />Optionally tune `index.rli.cache.size` and `index.rli.cache.concurrent.partitions.num` for the partition cache. | Flink uses partition-scoped RLI to route existing keys to their recorded file group and assigns new keys through dynamic bucket assignment. This supports streaming upsert and insert overwrite workloads. |
Review Comment:
π€ The default column for `index.bootstrap.enabled` says `false (Optional)`, but the description says "If not explicitly set for global RLI, Flink enables bootstrap by default." These appear contradictory. The companion table in `ingestion_flink.md` simply lists the default as `false`. Could you clarify whether the actual runtime default flips to `true` for `GLOBAL_RECORD_LEVEL_INDEX`, or whether the description is overstating it? Otherwise users wiring this up for the first time may set it explicitly only because they're unsure.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/cli.md:
##########
@@ -0,0 +1,791 @@
+---
+title: CLI
+keywords: [hudi, cli]
+last_modified_at: 2026-05-27T00:00:00-00:00
+---
+
+### Local set up
+Once hudi has been built, the shell can be fired by via `cd packaging/hudi-cli-bundle && hudi-cli-with-bundle.sh` or `packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh`.
+
+### Hudi CLI setup
+In release `0.13.0` we have now added new way of launching the `hudi cli`, which is using the `hudi-cli-bundle` script.
+
+#### Note: The traditional `hudi-cli.sh` script has been deprecated and replaced with `hudi-cli-with-bundle.sh` from `1.0.2` release onwards. Users should migrate to the new bundled CLI script `hudi-cli-with-bundle.sh` for better compatibility and ease of use.
+
+There are a couple of requirements such as having `spark` installed locally on your machine.
+It is required to use a spark distribution with hadoop dependencies packaged such as `spark-3.5.4-bin-hadoop3.tgz` from https://archive.apache.org/dist/spark/.
+We also recommend you set an env variable `$SPARK_HOME` to the path of where spark is installed on your machine.
+One important thing to note is that the `hudi-spark-bundle` should also be present when using the `hudi-cli-bundle`.
+To provide the locations of these bundle jars you can set them in your shell like so:
+`export CLI_BUNDLE_JAR=<path-to-cli-bundle-jar-to-use>` , `export SPARK_BUNDLE_JAR=<path-to-spark-bundle-jar-to-use>`.
+
+For steps see below if you are not compiling the project and downloading the jars:
+
+1. Create an empty folder as a new directory
+2. Copy the hudi-cli-bundle jars and hudi-spark*-bundle jars to this directory
+3. Copy the following script and folder to this directory
+```
+packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+packaging/hudi-cli-bundle/conf . the `conf` folder should be in this directory.
+```
+
+4. Start Hudi CLI shell with environment variables set
+```
+export SPARK_HOME=<spark-home-folder>
+export CLI_BUNDLE_JAR=<cli-bundle-jar-to-use>
+export SPARK_BUNDLE_JAR=<spark-bundle-jar-to-use>
+
+./hudi-cli-with-bundle.sh
+
+```
+
+### Base path
+A hudi table resides on DFS, in a location referred to as the `basePath` and
+we would need this location in order to connect to a Hudi table. Hudi library effectively manages this table internally, using `.hoodie` subfolder to track all metadata.
+
+
+
+
+### Using Hudi-cli in S3
+If you are using hudi that comes packaged with AWS EMR, you can find instructions to use hudi-cli [here](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-cli.html).
+If you are not using EMR, or would like to use latest hudi-cli from master, you can follow the below steps to access S3 dataset in your local environment (laptop).
+
+Build Hudi with corresponding Spark version, for eg, -Dspark3.5
+
+Set the following environment variables.
+```
+export AWS_REGION=us-east-2
+export AWS_ACCESS_KEY_ID=<key_id>
+export AWS_SECRET_ACCESS_KEY=<secret_key>
+
+export SPARK_HOME=<spark_home>
+export CLI_BUNDLE_JAR=<cli-bundle-jar-to-use>
+export SPARK_BUNDLE_JAR=<spark-bundle-jar-to-use>
+```
+Ensure you set the SPARK_HOME to your local spark home compatible to compiled hudi spark version above. One important thing to note is that the `hudi-spark-bundle` should also be present when using the `hudi-cli-bundle`.
+
+Apart from these, we might need to add aws jars to class path so that accessing S3 is feasible from local.
+We need two jars, namely, aws-java-sdk-bundle jar and hadoop-aws jar which you can find online.
+For eg:
+```
+wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -o /lib/spark-3.5.4-bin-hadoop3/jars/hadoop-aws-3.3.4.jar
+wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar -o /lib/spark-3.5.4-bin-hadoop3/jars/aws-java-sdk-bundle-1.12.262.jar
+```
+
+#### Note: These AWS jar versions below are specific to Spark 3.5.4 and Hadoop 3.3.4
+```
+export CLIENT_JAR=/lib/spark-3.5.4-bin-hadoop3/jars/aws-java-sdk-bundle-1.12.262.jar:/lib/spark-3.5.4-bin-hadoop3/jars/hadoop-aws-3.3.4.jar
+```
+Once these are set, you are good to launch hudi-cli and access S3 dataset.
+```
+./packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+```
+### Using hudi-cli on Google Dataproc
+[Dataproc](https://cloud.google.com/dataproc) is Google's managed service for running Apache Hadoop, Apache Spark,
+Apache Flink, Presto and many other frameworks, including Hudi. If you want to run the Hudi CLI on a Dataproc node
+which has not been launched with Hudi support enabled, you can use the steps below:
+
+These steps use Hudi version 1.2.0. If you want to use a different version you will have to edit the below commands
+appropriately:
+1. Once you've started the Dataproc cluster, you can ssh into it as follows:
+```
+$ gcloud compute ssh --zone "YOUR_ZONE" "HOSTNAME_OF_MASTER_NODE" --project "YOUR_PROJECT"
+```
+
+2. Download the Hudi CLI bundle
+```
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-cli-bundle_2.12/1.2.0/hudi-cli-bundle_2.12-1.2.0.jar
+```
+
+3. Download the Hudi Spark bundle
+```
+wget https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.5-bundle_2.12/1.2.0/hudi-spark3.5-bundle_2.12-1.2.0.jar
+```
+
+4. Download the shell script that launches Hudi CLI bundle
+```
+wget https://raw.githubusercontent.com/apache/hudi/release-1.2.0/packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
+```
+
+5. Launch Hudi CLI bundle with appropriate environment variables as follows:
+```
+CLIENT_JAR=$DATAPROC_DIR/lib/gcs-connector.jar CLI_BUNDLE_JAR=hudi-cli-bundle_2.12-1.2.0.jar SPARK_BUNDLE_JAR=hudi-spark3.5-bundle_2.12-1.2.0.jar ./hudi-cli-with-bundle.sh
+```
+
+6. hudi->connect --path gs://path_to_some_table
+Metadata for table some_table loaded
+
+7. hudi:some_table->commits show --limit 5
+This command should show the recent commits, if the above steps work correctly.
+
+## Connect to a Kerberized cluster
+
+Before connecting to a Kerberized cluster, you can use **kerberos kinit** command. Following is the usage of this command.
+
+```shell
+hudi->help kerberos kinit
+NAME
+ kerberos kinit - Perform Kerberos authentication
+
+SYNOPSIS
+ kerberos kinit --krb5conf String [--principal String] [--keytab String]
+
+OPTIONS
+ --krb5conf String
+ Path to krb5.conf
+ [Optional, default = /etc/krb5.conf]
+
+ --principal String
+ Kerberos principal
+ [Mandatory]
+
+ --keytab String
+ Path to keytab
+ [Mandatory]
+```
+
+For example:
+
+```shell
+hudi->kerberos kinit --principal user/host@DOMAIN --keytab /etc/security/keytabs/user.keytab
+Perform Kerberos authentication
+Parameters:
+--krb5conf: /etc/krb5.conf
+--principal: user/host@DOMAIN
+--keytab: /etc/security/keytabs/user.keytab
+Kerberos current user: user/host@DOMAIN (auth:KERBEROS)
+Kerberos login user: user/host@DOMAIN (auth:KERBEROS)
+Kerberos authentication success
+```
+
+If you see "Kerberos authentication success" in the command output, it means Kerberos authentication has been successful.
+
+
+## Using hudi-cli
+
+To initialize a hudi table, use the following command.
+
+```java
+===================================================================
+* ___ ___ *
+* /\__\ ___ /\ \ ___ *
+* / / / /\__\ / \ \ /\ \ *
+* / /__/ / / / / /\ \ \ \ \ \ *
+* / \ \ ___ / / / / / \ \__\ / \__\ *
+* / /\ \ /\__\ / /__/ ___ / /__/ \ |__| / /\/__/ *
+* \/ \ \/ / / \ \ \ /\__\ \ \ \ / / / /\/ / / *
+* \ / / \ \ / / / \ \ / / / \ /__/ *
+* / / / \ \/ / / \ \/ / / \ \__\ *
+* / / / \ / / \ / / \/__/ *
+* \/__/ \/__/ \/__/ Apache Hudi CLI *
+* *
+===================================================================
+
+hudi->create --path /user/hive/warehouse/table1 --tableName hoodie_table_1 --tableType COPY_ON_WRITE
+.....
+```
+
+To see the description of hudi table, use the command:
+
+```java
+hudi:hoodie_table_1->desc
+18/09/06 15:57:19 INFO timeline.HoodieActiveTimeline: Loaded instants []
+ _________________________________________________________
+ | Property | Value |
+ |========================================================|
+ | basePath | ... |
+ | metaPath | ... |
+ | fileSystem | hdfs |
+ | hoodie.table.name | hoodie_table_1 |
+ | hoodie.table.type | COPY_ON_WRITE |
+ | hoodie.archivelog.folder| |
+```
+
+Following is a sample command to connect to a Hudi table contains uber trips.
+
+```java
+hudi:trips->connect --path /app/uber/trips
+
+16/10/05 23:20:37 INFO model.HoodieTableMetadata: All commits :HoodieCommits{commitList=[20161002045850, 20161002052915, 20161002055918, 20161002065317, 20161002075932, 20161002082904, 20161002085949, 20161002092936, 20161002105903, 20161002112938, 20161002123005, 20161002133002, 20161002155940, 20161002165924, 20161002172907, 20161002175905, 20161002190016, 20161002192954, 20161002195925, 20161002205935, 20161002215928, 20161002222938, 20161002225915, 20161002232906, 20161003003028, 20161003005958, 20161003012936, 20161003022924, 20161003025859, 20161003032854, 20161003042930, 20161003052911, 20161003055907, 20161003062946, 20161003065927, 20161003075924, 20161003082926, 20161003085925, 20161003092909, 20161003100010, 20161003102913, 20161003105850, 20161003112910, 20161003115851, 20161003122929, 20161003132931, 20161003142952, 20161003145856, 20161003152953, 20161003155912, 20161003162922, 20161003165852, 20161003172923, 20161003175923, 20161003195931, 20161003210118, 201610032129
19, 20161003215928, 20161003223000, 20161003225858, 20161004003042, 20161004011345, 20161004015235, 20161004022234, 20161004063001, 20161004072402, 20161004074436, 20161004080224, 20161004082928, 20161004085857, 20161004105922, 20161004122927, 20161004142929, 20161004163026, 20161004175925, 20161004194411, 20161004203202, 20161004211210, 20161004214115, 20161004220437, 20161004223020, 20161004225321, 20161004231431, 20161004233643, 20161005010227, 20161005015927, 20161005022911, 20161005032958, 20161005035939, 20161005052904, 20161005070028, 20161005074429, 20161005081318, 20161005083455, 20161005085921, 20161005092901, 20161005095936, 20161005120158, 20161005123418, 20161005125911, 20161005133107, 20161005155908, 20161005163517, 20161005165855, 20161005180127, 20161005184226, 20161005191051, 20161005193234, 20161005203112, 20161005205920, 20161005212949, 20161005223034, 20161005225920]}
+Metadata for table trips loaded
+```
+
+Once connected to the table, a lot of other commands become available. The shell has contextual autocomplete help (press TAB) and below is a list of all commands, few of which are reviewed in this section
+
+```shell
+hudi:trips->help
+* ! - Allows execution of operating system (OS) commands
+* // - Inline comment markers (start of line only)
+* ; - Inline comment markers (start of line only)
+* bootstrap index showmapping - Show bootstrap index mapping
+* bootstrap index showpartitions - Show bootstrap indexed partitions
+* bootstrap run - Run a bootstrap action for current Hudi table
+* clean showpartitions - Show partition level details of a clean
+* cleans refresh - Refresh table metadata
+* cleans run - run clean
+* cleans show - Show the cleans
+* clear - Clears the console
+* cls - Clears the console
+* clustering run - Run Clustering
+* clustering schedule - Schedule Clustering
+* clustering scheduleAndExecute - Run Clustering. Make a cluster plan first and execute that plan immediately
+* commit rollback - Rollback a commit
+* commits compare - Compare commits with another Hoodie table
+* commit show_write_stats - Show write stats of a commit
+* commit showfiles - Show file level details of a commit
+* commit showpartitions - Show partition level details of a commit
+* commits refresh - Refresh table metadata
+* commits show - Show the commits
+* commits showarchived - Show the archived commits
+* commits sync - Sync commits with another Hoodie table
+* compaction repair - Renames the files to make them consistent with the timeline as dictated by Hoodie metadata. Use when compaction unschedule fails partially.
+* compaction run - Run Compaction for given instant time
+* compaction schedule - Schedule Compaction
+* compaction scheduleAndExecute - Schedule compaction plan and execute this plan
+* compaction show - Shows compaction details for a specific compaction instant
+* compaction showarchived - Shows compaction details for a specific compaction instant
+* compactions show all - Shows all compactions that are in active timeline
+* compactions showarchived - Shows compaction details for specified time window
+* compaction unschedule - Unschedule Compaction
+* compaction unscheduleFileId - UnSchedule Compaction for a fileId
+* compaction validate - Validate Compaction
+* connect - Connect to a hoodie table
+* create - Create a hoodie table if not present
+* date - Displays the local date and time
+* desc - Describe Hoodie Table properties
+* downgrade table - Downgrades a table
+* exit - Exits the shell
+* export instants - Export Instants and their metadata from the Timeline
+* fetch table schema - Fetches latest table schema
+* hdfsparquetimport - Imports Parquet table to a hoodie table
+* help - List all commands usage
+* marker delete - Delete the marker
+* metadata create - Create the Metadata Table if it does not exist
+* metadata delete - Remove the Metadata Table
+* metadata init - Update the metadata table from commits since the creation
+* metadata list-files - Print a list of all files in a partition from the metadata
+* metadata list-partitions - List all partitions from metadata
+* metadata refresh - Refresh table metadata
+* metadata set - Set options for Metadata Table
+* metadata stats - Print stats about the metadata
+* metadata validate-files - Validate all files in all partitions from the metadata
+* quit - Exits the shell
+* refresh - Refresh table metadata
+* repair addpartitionmeta - Add partition metadata to a table, if not present
+* repair corrupted clean files - repair corrupted clean files
+* repair deduplicate - De-duplicate a partition path contains duplicates & produce repaired files to replace with
+* repair migrate-partition-meta - Migrate all partition meta file currently stored in text format to be stored in base file format. See HoodieTableConfig#PARTITION_METAFILE_USE_DATA_FORMAT.
+* repair overwrite-hoodie-props - Overwrite hoodie.properties with provided file. Risky operation. Proceed with caution!
+* savepoint create - Savepoint a commit
+* savepoint delete - Delete the savepoint
+* savepoint rollback - Savepoint a commit
+* savepoints refresh - Refresh table metadata
+* savepoints show - Show the savepoints
+* script - Parses the specified resource file and executes its commands
+* set - Set spark launcher env to cli
+* show archived commits - Read commits from archived files and show details
+* show archived commit stats - Read commits from archived files and show details
+* show env - Show spark launcher env by key
+* show envs all - Show spark launcher envs
+* show fsview all - Show entire file-system view
+* show fsview latest - Show latest file-system view
+* show logfile metadata - Read commit metadata from log files
+* show logfile records - Read records from log files
+* show rollback - Show details of a rollback instant
+* show rollbacks - List all rollback instants
+* stats filesizes - File Sizes. Display summary stats on sizes of files
+* stats wa - Write Amplification. Ratio of how many records were upserted to how many records were actually written
+* sync validate - Validate the sync by counting the number of records
+* system properties - Shows the shell's properties
+* table delete-configs - Delete the supplied table configs from the table.
+* table recover-configs - Recover table configs, from update/delete that failed midway.
+* table update-configs - Update the table configs with configs with provided file.
+* temp_delete - Delete view name
+* temp_query - query against created temp view
+* temp delete - Delete view name
+* temp query - query against created temp view
+* temps_show - Show all views name
+* temps show - Show all views name
+* upgrade table - Upgrades a table
+* utils loadClass - Load a class
+* version - Displays shell version
+
+hudi:trips->
+```
+
+
+### Inspecting Commits
+
+The task of upserting or inserting a batch of incoming records is known as a **commit** in Hudi. A commit provides basic atomicity guarantees such that only committed data is available for querying.
+Each commit has a monotonically increasing string/number called the **commit number**. Typically, this is the time at which we started the commit.
+
+To view some basic information about the last 10 commits,
+
+
+```java
+hudi:trips->commits show --sortBy "Total Bytes Written" --desc true --limit 10
+ ________________________________________________________________________________________________________________________________________________________________________
+ | CommitTime | Total Bytes Written| Total Files Added| Total Files Updated| Total Partitions Written| Total Records Written| Total Update Records Written| Total Errors|
+ |=======================================================================================================================================================================|
+ ....
+ ....
+ ....
+```
+
+At the start of each write, Hudi also writes a .inflight commit to the .hoodie folder. You can use the timestamp there to estimate how long the commit has been inflight
+
+
+```java
+$ hdfs dfs -ls /app/uber/trips/.hoodie/*.inflight
+-rw-r--r-- 3 vinoth supergroup 321984 2016-10-05 23:18 /app/uber/trips/.hoodie/20161005225920.inflight
+```
+
+To list all inflight and requested instants that have been running longer than a specified number of minutes, use `commits show_inflights`:
+
+```shell
+hudi:trips->commits show_inflights --lookbackInMins 30
+```
+
+This lists every inflight or requested instant whose requested timestamp is older than 30 minutes, showing the commit time, action type, and current state. This is useful for detecting hung or stuck writes. The `--lookbackInMins` option defaults to `0` (returns all inflight/requested instants).
+
+### Drilling Down to a specific Commit
+
+To understand how the writes spread across specific partiions,
+
+
+```java
+hudi:trips->commit showpartitions --commit 20161005165855 --sortBy "Total Bytes Written" --desc true --limit 10
+ __________________________________________________________________________________________________________________________________________
+ | Partition Path| Total Files Added| Total Files Updated| Total Records Inserted| Total Records Updated| Total Bytes Written| Total Errors|
+ |=========================================================================================================================================|
+ ....
+ ....
+```
+
+If you need file level granularity , we can do the following
+
+
+```java
+hudi:trips->commit showfiles --commit 20161005165855 --sortBy "Partition Path"
+ ________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File ID | Previous Commit| Total Records Updated| Total Records Written| Total Bytes Written| Total Errors|
+ |=======================================================================================================================================================|
+ ....
+ ....
+```
+
+
+### FileSystem View
+
+Hudi views each partition as a collection of file-groups with each file-group containing a list of file-slices in commit order (See concepts).
+The below commands allow users to view the file-slices for a data-set.
+
+```java
+hudi:stock_ticks_mor->show fsview all
+ ....
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta File Size| Delta Files |
+ |==============================================================================================================================================================================================================================================================================================================================================================================================================|
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]|
+
+
+
+hudi:stock_ticks_mor->show fsview latest --partitionPath "2018/08/31"
+ ......
+ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta Size| Delta Size - compaction scheduled| Delta Size - compaction unscheduled| Delta To Base Ratio - compaction scheduled| Delta To Base Ratio - compaction unscheduled| Delta Files - compaction scheduled | Delta Files - compaction unscheduled|
+ |=================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================|
+ | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | 20.8 KB | 0.0 B | 0.0 B | 0.0 B | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]| [] |
+
+```
+
+
+### Statistics
+
+Since Hudi directly manages file sizes for DFS table, it might be good to get an overall picture
+
+
+```java
+hudi:trips->stats filesizes --partitionPath 2016/09/01 --sortBy "95th" --desc true --limit 10
+ ________________________________________________________________________________________________
+ | CommitTime | Min | 10th | 50th | avg | 95th | Max | NumFiles| StdDev |
+ |===============================================================================================|
+ | <COMMIT_ID> | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 93.9 MB | 2 | 2.3 KB |
+ ....
+ ....
+```
+
+In case of Hudi write taking much longer, it might be good to see the write amplification for any sudden increases
+
+
+```java
+hudi:trips->stats wa
+ __________________________________________________________________________
+ | CommitTime | Total Upserted| Total Written| Write Amplifiation Factor|
+ |=========================================================================|
+ ....
+ ....
+```
+
+
+### Archived Commits
+
+In order to limit the amount of growth of .commit files on DFS, Hudi archives older .commit files (with due respect to the cleaner policy) into a commits.archived file.
+This is a sequence file that contains a mapping from commitNumber => json with raw information about the commit (same that is nicely rolled up above).
+
+
+### Compactions
+
+To get an idea of the lag between compaction and writer applications, use the below command to list down all
+pending compactions.
+
+```java
+hudi:trips->compactions show all
+ ___________________________________________________________________
+ | Compaction Instant Time| State | Total FileIds to be Compacted|
+ |==================================================================|
+ | <INSTANT_1> | REQUESTED| 35 |
+ | <INSTANT_2> | INFLIGHT | 27 |
+```
+
+To inspect a specific compaction plan, use
+
+```java
+hudi:trips->compaction show --instant <INSTANT_1>
+ _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | Partition Path| File Id | Base Instant | Data File Path | Total Delta Files| getMetrics |
+ |================================================================================================================================================================================================================================================
+ | 2018/07/17 | <UUID> | <INSTANT_1> | viewfs://ns-default/.../../UUID_<INSTANT>.parquet | 1 | {TOTAL_LOG_FILES=1.0, TOTAL_IO_READ_MB=1230.0, TOTAL_LOG_FILES_SIZE=2.51255751E8, TOTAL_IO_WRITE_MB=991.0, TOTAL_IO_MB=2221.0}|
+
+```
+
+To manually schedule or run a compaction, use the below command. This command uses spark launcher to perform compaction
+operations.
+
+**NOTE:** Make sure no other application is scheduling compaction for this table concurrently
+\{: .notice--info}
+
+```java
+hudi:trips->help compaction schedule
+Keyword: compaction schedule
+Description: Schedule Compaction
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: false
+ Default if specified: '__NULL__'
+ Default if unspecified: '1G'
+
+* compaction schedule - Schedule Compaction
+```
+
+```java
+hudi:trips->help compaction run
+Keyword: compaction run
+Description: Run Compaction for given instant time
+ Keyword: tableName
+ Help: Table name
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: parallelism
+ Help: Parallelism for hoodie compaction
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: schemaFilePath
+ Help: Path for Avro schema file
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: sparkMemory
+ Help: Spark executor memory
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: retry
+ Help: Number of retries
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+ Keyword: compactionInstant
+ Help: Base path for the target hoodie table
+ Mandatory: true
+ Default if specified: '__NULL__'
+ Default if unspecified: '__NULL__'
+
+* compaction run - Run Compaction for given instant time
+```
+
+### Validate Compaction
+
+Validating a compaction plan : Check if all the files necessary for compactions are present and are valid
+
+```java
+hudi:stock_ticks_mor->compaction validate --instant 20181005222611
+...
+
+ COMPACTION PLAN VALID
+
+ ___________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error|
+ |==========================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | true | |
+
+
+
+hudi:stock_ticks_mor->compaction validate --instant 20181005222601
+
+ COMPACTION PLAN INVALID
+
+ _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
+ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error |
+ |=====================================================================================================================================================================================================================================================================================================|
+ | 05320e98-9a57-4c38-b809-a6beaaeb36bd| 20181005222445 | hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/05320e98-9a57-4c38-b809-a6beaaeb36bd_0_20181005222445.parquet| 1 | false| All log files specified in compaction operation is not present. Missing .... |
+```
+
+**NOTE:** The following commands must be executed without any other writer/ingestion application running.
+\{: .notice--warning}
+
+Sometimes, it becomes necessary to remove a fileId from a compaction-plan inorder to speed-up or unblock compaction
+operation. Any new log-files that happened on this file after the compaction got scheduled will be safely renamed
+so that are preserved. Hudi provides the following CLI to support it
+
+
+### Unscheduling Compaction
+
+```java
+hudi:trips->compaction unscheduleFileId --fileId <FileUUID>
+....
+No File renames needed to unschedule file from pending compaction. Operation successful.
+```
+
+In other cases, an entire compaction plan needs to be reverted. This is supported by the following CLI
+
+```java
+hudi:trips->compaction unschedule --instant <compactionInstant>
+.....
+No File renames needed to unschedule pending compaction. Operation successful.
+```
+
+### Repair Compaction
+
+The above compaction unscheduling operations could sometimes fail partially (e:g -> DFS temporarily unavailable). With
+partial failures, the compaction operation could become inconsistent with the state of file-slices. When you run
+`compaction validate`, you can notice invalid compaction operations if there is one. In these cases, the repair
+command comes to the rescue, it will rearrange the file-slices so that there is no loss and the file-slices are
+consistent with the compaction plan
+
+```java
+hudi:stock_ticks_mor->compaction repair --instant 20181005222611
+......
+Compaction successfully repaired
+.....
+```
+
+### Savepoint and Restore
+As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery.md)
+
+To trigger savepoint for a hudi table
+```java
Review Comment:
π€ The table version mapping table doesn't include a row for the 1.2.x release line, which is the version this whole document tree is for. Since 1.2.x continues to use table version 9 (`HoodieTableVersion.NINE`), I'd suggest updating the existing row to read `1.1.x - 1.2.x` (or splitting into two rows for clarity). As-is, a 1.2.0 user reading their own docs can't find an entry for their release. @yihua could you confirm whether 1.2.0 keeps table version 9 or introduces a new one?
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/reading_tables_streaming_reads.md:
##########
@@ -0,0 +1,143 @@
+---
+title: Streaming Reads
+keywords: [hudi, spark, flink, streaming, processing]
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Spark Streaming
+
+Structured Streaming reads are based on Hudi's Incremental Query feature, therefore streaming read can return data for which
+commits and base files were not yet removed by the cleaner. You can control commits retention time.
+
+<Tabs
+groupId="programming-language"
+defaultValue="python"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Python', value: 'python', },
+]}
+>
+
+<TabItem value="scala">
+
+```scala
+// spark-shell
+// reload data
+df.write.format("hudi").
+ options(getQuickstartWriteConfigs).
+ option("hoodie.table.ordering.fields", "ts").
+ option("hoodie.datasource.write.recordkey.field", "uuid").
+ option("hoodie.datasource.write.partitionpath.field", "partitionpath").
+ option("hoodie.table.name", tableName).
+ mode(Overwrite).
+ save(basePath)
+
+// read stream and output results to console
+spark.readStream.
+ format("hudi").
+ load(basePath).
+ writeStream.
+ format("console").
+ start()
+
+// read stream to streaming df
+val df = spark.readStream.
+ format("hudi").
+ load(basePath)
+
+```
+</TabItem>
+
+<TabItem value="python">
+
+```python
+# pyspark
+# reload data
+inserts = sc._jvm.org.apache.hudi.QuickstartUtils.convertToStringList(
+ dataGen.generateInserts(10))
+df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
+
+hudi_options = {
+ 'hoodie.table.name': tableName,
+ 'hoodie.datasource.write.recordkey.field': 'uuid',
+ 'hoodie.datasource.write.partitionpath.field': 'partitionpath',
+ 'hoodie.datasource.write.table.name': tableName,
+ 'hoodie.datasource.write.operation': 'upsert',
+ 'hoodie.table.ordering.fields': 'ts',
+ 'hoodie.upsert.shuffle.parallelism': 2,
+ 'hoodie.insert.shuffle.parallelism': 2
+}
+
+df.write.format("hudi"). \
+ options(**hudi_options). \
+ mode("overwrite"). \
+ save(basePath)
+
+# read stream to streaming df
+df = spark.readStream \
+ .format("hudi") \
+ .load(basePath)
+
+# ead stream and output results to console
+spark.readStream \
+ .format("hudi") \
+ .load(basePath) \
+ .writeStream \
+ .format("console") \
+ .start()
+
+```
+</TabItem>
+
+</Tabs
+>
+
+:::info
+Spark SQL can be used within ForeachBatch sink to do INSERT, UPDATE, DELETE and MERGE INTO.
+Target table must exist before write.
+:::
+
+## Flink Streaming Read
+
+Flink can continuously consume new commits from a Hudi table as a streaming source. Enable this by setting `read.streaming.enabled=true` and optionally a `read.start-commit`.
+
+```sql
+CREATE TABLE hudi_table (
+ uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = '${path}',
+ 'table.type' = 'MERGE_ON_READ',
+ 'read.streaming.enabled' = 'true', -- enable streaming read
+ 'read.start-commit' = '20210316134557', -- start from this instant (omit for latest)
+ 'read.streaming.check-interval' = '60' -- poll interval in seconds
+);
+
+SELECT * FROM hudi_table;
+```
+
+### Source V2 for Streaming
+
+As of Hudi 1.2.0, the [FLIP-27-based Source V2](ingestion_flink.md#flink-source-v2) is available as an opt-in for streaming reads. Source V2 participates in Flink's checkpoint protocol for finer-grained recovery and supports partition pruning:
+
+```sql
+WITH (
+ 'connector' = 'hudi',
Review Comment:
π€ This section introduces Source V2 with `read.source-v2.enabled` as a 1.2.0 opt-in and warns about savepoint incompatibility. Could you confirm the option name (`read.source-v2.enabled`) matches the actual config key shipped in 1.2.0, and that the savepoint-incompatibility warning is the recommended migration guidance? @danny0405 could you double-check the Source V2 docs match the implementation for the 1.2.0 release?
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_ddl.md:
##########
@@ -0,0 +1,1218 @@
+---
+title: SQL DDL
+summary: "In this page, we discuss using SQL DDL commands with Hudi"
+toc: true
+last_modified_at: 2026-05-29T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This page describes support for creating and altering tables using SQL across various engines.
+
+## Spark SQL
+
+### Create table
+
+You can create tables using standard CREATE TABLE syntax, which supports partitioning and passing table properties.
+
+```sql
+CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
+ [(col_name data_type [COMMENT col_comment], ...)]
+ [COMMENT table_comment]
+ [PARTITIONED BY (col_name, ...)]
+ [ROW FORMAT row_format]
+ [STORED AS file_format]
+ [LOCATION path]
+ [TBLPROPERTIES (property_name=property_value, ...)]
+ [AS select_statement];
+```
+
+:::note NOTE:
+For users running this tutorial locally and have a Spark-Hive(HMS) integration in their environment: If you use
+`default` database or if you don't provide `[LOCATION path]` with the DDL statement, Spark will return
+`java.io.IOException: Mkdirs failed to create file:/user/hive/warehouse/hudi_table/.hoodie` error.
+To get around this, you can follow either of the two options mentioned below:
+1. Create a database i.e. `CREATE DATABASE hudidb;` and use it i.e. `USE hudidb;` before running the DDL statement.
+2. Or provide a path using `LOCATION` keyword to persist the data with the DDL statement.
+:::
+
+### Create non-partitioned table
+
+Creating a non-partitioned table is as simple as creating a regular table.
+
+```sql
+-- create a Hudi table
+CREATE TABLE IF NOT EXISTS hudi_table (
+ id INT,
+ name STRING,
+ price DOUBLE
+) USING hudi;
+```
+
+### Create partitioned table
+A partitioned table can be created by adding a `partitioned by` clause. Partitioning helps to organize the data into multiple folders
+based on the partition columns. It can also help speed up queries and index lookups by limiting the amount of metadata, index and data scanned.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_partitioned (
+ id BIGINT,
+ name STRING,
+ dt STRING,
+ hh STRING
+) USING hudi
+TBLPROPERTIES (
+ type = 'cow'
+)
+PARTITIONED BY (dt);
+```
+
+:::note
+You can also create a table partitioned by multiple fields by supplying comma-separated field names.
+When creating a table partitioned by multiple fields, ensure that you specify the columns in the `PARTITIONED BY` clause
+in the same order as they appear in the `CREATE TABLE` schema. For example, for the above table, the partition fields
+should be specified as `PARTITIONED BY (dt, hh)`.
+:::
+
+### Create table with record keys and ordering fields
+
+As discussed [here](quick-start-guide.md#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
+key for each new record in the examples so far. If you want to use an existing field as the key, you can set the `primaryKey` option.
+Typically, this is also accompanied by configuring ordering fields (via `orderingFields` option) to deal with out-of-order data and potential
+duplicate records with the same key in the incoming writes.
+
+:::note
+You can choose multiple fields as primary keys for a given table on a need basis. For eg, "primaryKey = 'id, name'", and
+this materializes a composite key of the two fields, which can be useful for exploring the table.
+:::
+
+Here is an example of creating a table using both options. Typically, a field that denotes the time of the event or
+fact, e.g., order creation time, event generation time etc., is used as the ordering field (via `orderingFields`). Hudi resolves multiple versions
+of the same record by ordering based on this field when queries are run on the table.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_keyed (
+ id INT,
+ name STRING,
+ price DOUBLE,
+ ts BIGINT
+) USING hudi
+TBLPROPERTIES (
+ type = 'cow',
+ primaryKey = 'id',
+ orderingFields = 'ts'
+);
+```
+
+### Create table with merge modes {#create-table-with-record-merge-mode}
+
+Hudi supports different [record merge modes](record_merger.md) to handle merge of incoming records with existing
+records. To create a table with specific record merge mode, you can set `recordMergeMode` option.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'EVENT_TIME_ORDERING'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode/';
+```
+
+With `EVENT_TIME_ORDERING`, the record with the larger event time (specified via `orderingFields`) overwrites the record with the
+smaller event time on the same key, regardless of transaction's commit time. Users can set `CUSTOM` mode to provide their own
+merge logic. With `CUSTOM` merge mode, you can provide a custom class that implements the merge logic. The interfaces
+to implement is explained in detail [here](record_merger.md#custom).
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'CUSTOM',
+ 'hoodie.record.merge.strategy.id' = '<unique-uuid>'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
+```
+
+### Create table with unstructured and semi-structured column types {#create-table-with-unstructured-and-semi-structured-column-types}
+
+Hudi supports three column types for unstructured and semi-structured data, plus a Lance base file
+format.
+
+#### `VECTOR(dim[, elementType])` {#vector}
+
+A fixed-dimension embedding column. `elementType` is one of `FLOAT` (default), `DOUBLE`, or `INT8`
+(alias `BYTE`):
+
+| Element type | Storage |
+|:-------------|:--------|
+| `FLOAT` (default) | `ArrayType(FloatType)` |
+| `DOUBLE` | `ArrayType(DoubleType)` |
+| `INT8` / `BYTE` | `ArrayType(ByteType)` |
+
+```sql
+CREATE TABLE products (
+ product_id STRING,
+ name STRING,
+ embedding VECTOR(768) -- defaults to FLOAT
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'product_id',
+ type = 'cow',
+ hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger'
+);
Review Comment:
π€ The 1.2.0 snapshot introduces a number of significant new column types (VECTOR, BLOB, VARIANT) and a Lance base file format. Could a committer confirm these are intended to be GA / user-facing in 1.2.0 (vs. experimental) and that the storage-format / engine-support tables here are accurate? @yihua / @bvaradar β these touch storage format and public DDL surface, so PMC eyes would be helpful before this becomes the default version users land on.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_dml.md:
##########
@@ -0,0 +1,631 @@
+---
+title: SQL DML
+summary: "In this page, we go will cover details on how to use DML statements on Hudi tables."
+toc: true
+last_modified_at:
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Spark SQL
+
+SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
+from your Hudi tables. Let's explore them one by one.
+
+Please refer to [SQL DDL](sql_ddl.md) for creating Hudi tables using SQL.
+
+### Insert Into
+
+You can use the `INSERT INTO` statement to add data to a Hudi table using Spark SQL. Here are some examples:
+
+```sql
+INSERT INTO <table>
+SELECT <columns> FROM <source>;
+```
+
+:::info
+`INSERT INTO` statement does not support evolving table schema. Please use DDL (e.g., `ALTER TABLE`) or Datasource write (`df.write.format("hudi")....save(basePath)`) to evolve table schema.
+:::
+
+:::note Deprecations
+From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations.md#hoodiedatasourceinsertduppolicy).
+:::
+
+Examples:
+
+```sql
+-- Insert into a copy-on-write (COW) Hudi table
+INSERT INTO hudi_cow_nonpcf_tbl SELECT 1, 'a1', 20;
+
+-- Insert into a merge-on-read (MOR) Hudi table
+INSERT INTO hudi_mor_tbl SELECT 1, 'a1', 20, 1000;
+
+-- Insert into a COW Hudi table with static partition
+INSERT INTO hudi_cow_pt_tbl PARTITION(dt = '2021-12-09', hh='11') SELECT 2, 'a2', 1000;
+
+-- Insert into a COW Hudi table with dynamic partition
+INSERT INTO hudi_cow_pt_tbl PARTITION(dt, hh) SELECT 1 AS id, 'a1' AS name, 1000 AS ts, '2021-12-09' AS dt, '10' AS hh;
+```
+
+#### Inserting VECTOR columns
+
+A [`VECTOR(dim[, elementType])`](sql_ddl.md#vector) column accepts an `ARRAY` whose length and
+element type match the declared dimension and element type:
+
+```sql
+INSERT INTO products VALUES (
+ 'prod_001', 'Running Shoes', 'Lightweight trail runner',
+ ARRAY(0.123, -0.456, 0.789, /* ... 768 floats ... */)
+);
+```
+
+The query vector's element type must exactly match the corpus embedding's element type (no implicit
+casting).
+
+#### Inserting BLOB columns
+
+A [`BLOB`](sql_ddl.md#blob) column is internally a struct with `type` (`INLINE`/`OUT_OF_LINE`),
+`data`, and a `reference` pointer struct. Construct it via `named_struct`.
+
+**Inline** β bytes embedded directly in the row:
+
+```sql
+INSERT INTO media_assets VALUES (
+ 'asset_001', 'logo.png', 'image/png', 45230,
+ named_struct(
+ 'type', 'INLINE',
+ 'data', /* binary literal or column reference */,
+ 'reference', CAST(NULL AS STRUCT<external_path: STRING, offset: BIGINT, length: BIGINT, managed: BOOLEAN>)
+ )
+);
+```
+
+**Out-of-line** β pointer to a byte range in an external file:
+
+```sql
+INSERT INTO media_assets VALUES (
+ 'asset_002', 'video.mp4', 'video/mp4', 1073741824,
+ named_struct(
+ 'type', 'OUT_OF_LINE',
+ 'data', CAST(NULL AS BINARY),
+ 'reference', named_struct(
+ 'external_path', 's3://my-bucket/media/container_001.bin',
+ 'offset', 8388608,
+ 'length', 1073741824,
+ 'managed', false
+ )
+ )
+);
+```
+
+A common pattern for `OUT_OF_LINE` storage is to pack multiple objects into a single binary
+container file and have each BLOB row store the `(external_path, offset, length)` triple into that
+container:
+
+```
+container_001.bin
+βββ [offset=0, len=45230] β logo.png
+βββ [offset=45230, len=89012] β photo.jpg
+βββ [offset=134242, len=1073741824] β video.mp4
+βββ ...
+```
+
+#### Inserting VARIANT columns
+
+A [`VARIANT`](sql_ddl.md#variant) column accepts any JSON-compatible value. On Spark 4.0+, build
+VARIANT values with `parse_json`:
+
+```sql
+INSERT INTO events VALUES
+ ('evt_001', parse_json('{"action": "click", "x": 120, "y": 450}'), 1000),
+ ('evt_002', parse_json('{"action": "purchase", "items": ["sku_a", "sku_b"], "total": 59.99}'), 1001),
+ ('evt_003', parse_json('"simple string value"'), 1002);
+```
+
+:::note Mapping to write operations
+Hudi offers flexibility in choosing the underlying [write operation](write_operations.md) of a `INSERT INTO` statement using
+the `hoodie.spark.sql.insert.into.operation` configuration. Possible options include *"bulk_insert"* (large inserts), *"insert"* (with small file management),
+and *"upsert"* (with deduplication/merging). If ordering fields are not set, *"insert"* is chosen as the default. For a table with ordering fields set (via `orderingFields`),
+*"upsert"* is chosen as the default operation.
+:::
+
+
+### Insert Overwrite
+
+The `INSERT OVERWRITE` statement is used to replace existing data in a Hudi table.
+
+```sql
+INSERT OVERWRITE <table>
+SELECT <columns> FROM <source>;
+```
+
+All existing partitions that are affected by the `INSERT OVERWRITE` statement will replaced with the source data.
+Here are some examples:
+
+```sql
+-- Overwrite non-partitioned table
+INSERT OVERWRITE hudi_mor_tbl SELECT 99, 'a99', 20.0, 900;
+INSERT OVERWRITE hudi_cow_nonpcf_tbl SELECT 99, 'a99', 20.0;
+
+-- Overwrite partitioned table with dynamic partition
+INSERT OVERWRITE TABLE hudi_cow_pt_tbl SELECT 10, 'a10', 1100, '2021-12-09', '10';
+
+-- Overwrite partitioned table with static partition
+INSERT OVERWRITE hudi_cow_pt_tbl PARTITION(dt = '2021-12-09', hh='12') SELECT 13, 'a13', 1100;
+```
+
+### Update
+You can use the `UPDATE` statement to modify existing data in a Hudi table directly.
+
+```sql
+UPDATE tableIdentifier SET column = EXPRESSION(,column = EXPRESSION) [ WHERE boolExpression]
+```
+
+Here's an example:
+
+```sql
+-- Update data in a Hudi table
+UPDATE hudi_mor_tbl SET price = price * 2, ts = 1111 WHERE id = 1;
+
+-- Update data in a partitioned Hudi table
+UPDATE hudi_cow_pt_tbl SET name = 'a1_1', ts = 1001 WHERE id = 1;
+
+-- update using non-PK field
+update hudi_cow_pt_tbl set ts = 1001 where name = 'a1';
+```
+
+:::info
+The `UPDATE` operation requires the specification of ordering fields (via `orderingFields`).
+:::
+
+### Merge Into
+
+The `MERGE INTO` statement allows you to perform more complex updates and merges against source data. The `MERGE INTO` statement
+is similar to the `UPDATE` statement, but it allows you to specify different actions for matched and unmatched records.
+
+```sql
+MERGE INTO tableIdentifier AS target_alias
+USING (sub_query | tableIdentifier) AS source_alias
+ON <merge_condition>
+[ WHEN MATCHED [ AND <condition> ] THEN <matched_action> ]
+[ WHEN NOT MATCHED [ AND <condition> ] THEN <not_matched_action> ]
+
+<merge_condition> =A equal bool condition
+<matched_action> =
+ DELETE |
+ UPDATE SET * |
+ UPDATE SET column1 = expression1 [, column2 = expression2 ...]
+<not_matched_action> =
+ INSERT * |
+ INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 ...])
+```
+
+:::info
+`MERGE INTO` statement does not support evolving table schema. Please use DDL (e.g., `ALTER TABLE`) or Datasource write (`df.write.format("hudi")....save(basePath)`) to evolve table schema.
+
+`WHEN NOT MATCHED` clauses specify the action to perform if the values do not match.
+There are two kinds of `INSERT` clauses:
+1. `INSERT *` clauses require that the source table has the same columns as those in the target table.
+2. `INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 ...])` clauses do not require to specify all the columns of the target table. For unspecified target columns, insert the `NULL` value.
+
+For a Hudi table with user configured primary keys, the join condition and the `UPDATE`/`INSERT INTO` clause in `MERGE INTO` is expected to contain the primary keys of the table.
+
+For a table where Hudi auto generates primary keys, the join condition in `MERGE INTO` can be on any arbitrary data columns.
+
+if the `hoodie.record.merge.mode` is set to `EVENT_TIME_ORDERING`, ordering fields (via `orderingFields`) are required to be set with value in the `UPDATE`/`INSERT` clause.
+
+It is enforced that if the target table has primary key and partition key column, the source table counterparts must enforce the same data type accordingly. Plus, if the target table is configured with `hoodie.record.merge.mode` = `EVENT_TIME_ORDERING` where target table is expected to have valid ordering fields configuration, the source table counterpart must also have the same data type.
+:::
+
+Examples below
+
+```sql
+-- source table using hudi for testing merging into non-partitioned table
+create table merge_source (id int, name string, price double, ts bigint) using hudi
+tblproperties (primaryKey = 'id', orderingFields = 'ts');
+insert into merge_source values (1, "old_a1", 22.22, 900), (2, "new_a2", 33.33, 2000), (3, "new_a3", 44.44, 2000);
+
+merge into hudi_mor_tbl as target
+using merge_source as source
+on target.id = source.id
+when matched then update set *
+when not matched then insert *
+;
+
+-- source table using parquet for testing merging into partitioned table
+create table merge_source2 (id int, name string, flag string, dt string, hh string) using parquet;
+insert into merge_source2 values (1, "new_a1", 'update', '2021-12-09', '10'), (2, "new_a2", 'delete', '2021-12-09', '11'), (3, "new_a3", 'insert', '2021-12-09', '12');
+
+MERGE into hudi_cow_pt_tbl as target
+using (
+ select id, name, '1000' as ts, flag, dt, hh from merge_source2
+) source
+on target.id = source.id
+when matched and flag != 'delete' then
+ update set id = source.id, name = source.name, ts = source.ts, dt = source.dt, hh = source.hh
+when matched and flag = 'delete' then delete
+when not matched then
+ insert (id, name, ts, dt, hh) values(source.id, source.name, source.ts, source.dt, source.hh)
+;
+
+```
+
+:::note Key requirements
+For a Hudi table with user configured primary keys, the join condition in `Merge Into` is expected to contain the primary keys of the table.
+For a Table where Hudi auto generates primary keys, the join condition in MIT can be on any arbitrary data columns.
+:::
+
+### Merge Into with Partial Updates {#merge-into-partial-update}
+
+Partial updates only write updated columns instead of full change records. This is useful when you have wide tables (typical for ML feature stores)
+with hundreds of columns and only a few columns are updated. It reduces the write amplification as well as helps in lowering the query
+latency. `MERGE INTO` statement above can be modified to use partial updates as shown below.
+
+```sql
+-- Create a Merge-on-Read table
+CREATE TABLE tableName (
+ id INT,
+ name STRING,
+ price DOUBLE,
+ _ts LONG,
+ description STRING
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = '_ts'
+)
+LOCATION '/location/to/basePath';
+
+-- Insert values into the table
+INSERT INTO tableName VALUES
+ (1, 'a1', 10, 1000, 'a1: desc1'),
+ (2, 'a2', 20, 1200, 'a2: desc2'),
+ (3, 'a3', 30, 1250, 'a3: desc3');
+
+-- Perform partial updates using a MERGE INTO statement
+MERGE INTO tableName t0
+ USING (
+ SELECT 1 AS id, 'a1' AS name, 12 AS price, 1001 AS ts
+ UNION ALL
+ SELECT 3 AS id, 'a3' AS name, 25 AS price, 1260 AS ts
+ ) s0
+ ON t0.id = s0.id
+ WHEN MATCHED THEN UPDATE SET
+ price = s0.price,
+ _ts = s0.ts;
+
+SELECT id, name, price, _ts, description FROM tableName;
+```
+
+Notice, instead of `UPDATE SET *`, we are updating only the `price` and `_ts` columns.
+
+:::note
+Partial update is not yet supported in the following cases:
+1. When the target table is a bootstrapped table.
+2. When virtual keys is enabled.
+3. When schema on read is enabled.
+4. When there is an enum field in the source data.
+5. When a global index is enabled, e.g., GLOBAL_SIMPLE.
+
+For a Hudi table with user configured primary keys, the join condition and the `UPDATE`/`INSERT INTO` clause in `MERGE INTO` is expected to contain the primary keys of the table.
+:::
+
+### Delete From
+
+You can remove data from a Hudi table using the `DELETE FROM` statement.
+
+```sql
+DELETE FROM tableIdentifier [ WHERE boolExpression ]
+```
+
+Examples below
+
+```sql
+-- Delete data from a Hudi table
+DELETE FROM hudi_cow_nonpcf_tbl WHERE uuid = 1;
+
+-- Delete data from a MOR Hudi table based on a condition
+DELETE FROM hudi_mor_tbl WHERE id % 2 = 0;
+
+-- Delete data using a non-primary key field
+DELETE FROM hudi_cow_pt_tbl WHERE name = 'a1';
+```
+
+### Data Skipping and Indexing
+
+DML operations can be sped up using column statistics for data skipping and using indexes to reduce the amount of data scanned.
+For e.g. the following helps speed up the `DELETE` operation on a Hudi table, by using the record level index.
+
+```sql
+SET hoodie.metadata.record.index.enable=true;
+
+DELETE from hudi_table where uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa';
+```
+
+These DML operations give you powerful tools for managing your tables using Spark SQL.
+You can control the behavior of these operations using various configuration options, as explained in the documentation.
+
+## Flink SQL
+
+Flink SQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update and delete data from your Hudi tables. Let's explore them one by one.
+
+### Insert Into
+
+You can utilize the INSERT INTO statement to incorporate data into a Hudi table using Flink SQL. Here are a few illustrative examples:
+
+```sql
+INSERT INTO <table>
+SELECT <columns> FROM <source>;
+```
+
+Examples:
+
+```sql
+-- Insert into a Hudi table
+INSERT INTO hudi_table SELECT 1, 'a1', 20;
+```
+
+If the `write.operation` is 'upsert,' the INSERT INTO statement will not only insert new records but also update existing rows with the same record key.
+
+```sql
+-- Insert into a Hudi table in upsert mode
+INSERT INTO hudi_table/*+ OPTIONS('write.operation'='upsert')*/ SELECT 1, 'a1', 20;
+```
+
+### Update
+With Flink SQL, you can use update command to update the hudi table. Here are a few illustrative examples:
+
+```sql
+UPDATE tableIdentifier SET column = EXPRESSION(,column = EXPRESSION) [ WHERE boolExpression]
+```
+
+```sql
+UPDATE hudi_table SET price = price * 2, ts = 1111 WHERE id = 1;
+```
+
+:::note Key requirements
+Update query only work with batch excution mode.
+:::
+
+### Delete From
+With Flink SQL, you can use delete command to delete the rows from hudi table. Here are a few illustrative examples:
+
+```sql
+DELETE FROM tableIdentifier [ WHERE boolExpression ]
+```
+
+```sql
+DELETE FROM hudi_table WHERE price < 100;
+```
+
+
+```sql
+DELETE FROM hudi_table WHERE price < 100;
+```
+
+:::note Key requirements
+Delete query only work with batch excution mode.
+:::
+
+### Lookup Joins
Review Comment:
π€ Small typo: "excution" β "execution". (Same typo appears again a few lines below in the Delete note.)
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_dml.md:
##########
@@ -0,0 +1,631 @@
+---
+title: SQL DML
+summary: "In this page, we go will cover details on how to use DML statements on Hudi tables."
+toc: true
+last_modified_at:
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Spark SQL
+
+SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
+from your Hudi tables. Let's explore them one by one.
+
+Please refer to [SQL DDL](sql_ddl.md) for creating Hudi tables using SQL.
+
+### Insert Into
+
+You can use the `INSERT INTO` statement to add data to a Hudi table using Spark SQL. Here are some examples:
+
+```sql
+INSERT INTO <table>
+SELECT <columns> FROM <source>;
+```
+
+:::info
+`INSERT INTO` statement does not support evolving table schema. Please use DDL (e.g., `ALTER TABLE`) or Datasource write (`df.write.format("hudi")....save(basePath)`) to evolve table schema.
+:::
+
+:::note Deprecations
+From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations.md#hoodiedatasourceinsertduppolicy).
+:::
+
+Examples:
+
+```sql
+-- Insert into a copy-on-write (COW) Hudi table
+INSERT INTO hudi_cow_nonpcf_tbl SELECT 1, 'a1', 20;
+
+-- Insert into a merge-on-read (MOR) Hudi table
+INSERT INTO hudi_mor_tbl SELECT 1, 'a1', 20, 1000;
+
+-- Insert into a COW Hudi table with static partition
+INSERT INTO hudi_cow_pt_tbl PARTITION(dt = '2021-12-09', hh='11') SELECT 2, 'a2', 1000;
+
+-- Insert into a COW Hudi table with dynamic partition
+INSERT INTO hudi_cow_pt_tbl PARTITION(dt, hh) SELECT 1 AS id, 'a1' AS name, 1000 AS ts, '2021-12-09' AS dt, '10' AS hh;
+```
+
+#### Inserting VECTOR columns
+
+A [`VECTOR(dim[, elementType])`](sql_ddl.md#vector) column accepts an `ARRAY` whose length and
+element type match the declared dimension and element type:
+
+```sql
+INSERT INTO products VALUES (
+ 'prod_001', 'Running Shoes', 'Lightweight trail runner',
+ ARRAY(0.123, -0.456, 0.789, /* ... 768 floats ... */)
+);
+```
+
+The query vector's element type must exactly match the corpus embedding's element type (no implicit
+casting).
+
+#### Inserting BLOB columns
+
+A [`BLOB`](sql_ddl.md#blob) column is internally a struct with `type` (`INLINE`/`OUT_OF_LINE`),
+`data`, and a `reference` pointer struct. Construct it via `named_struct`.
+
+**Inline** β bytes embedded directly in the row:
+
+```sql
+INSERT INTO media_assets VALUES (
+ 'asset_001', 'logo.png', 'image/png', 45230,
+ named_struct(
+ 'type', 'INLINE',
+ 'data', /* binary literal or column reference */,
+ 'reference', CAST(NULL AS STRUCT<external_path: STRING, offset: BIGINT, length: BIGINT, managed: BOOLEAN>)
+ )
+);
+```
+
+**Out-of-line** β pointer to a byte range in an external file:
+
+```sql
+INSERT INTO media_assets VALUES (
+ 'asset_002', 'video.mp4', 'video/mp4', 1073741824,
+ named_struct(
+ 'type', 'OUT_OF_LINE',
+ 'data', CAST(NULL AS BINARY),
+ 'reference', named_struct(
+ 'external_path', 's3://my-bucket/media/container_001.bin',
+ 'offset', 8388608,
+ 'length', 1073741824,
+ 'managed', false
+ )
+ )
+);
+```
+
+A common pattern for `OUT_OF_LINE` storage is to pack multiple objects into a single binary
+container file and have each BLOB row store the `(external_path, offset, length)` triple into that
+container:
+
+```
+container_001.bin
+βββ [offset=0, len=45230] β logo.png
+βββ [offset=45230, len=89012] β photo.jpg
+βββ [offset=134242, len=1073741824] β video.mp4
+βββ ...
+```
+
+#### Inserting VARIANT columns
+
+A [`VARIANT`](sql_ddl.md#variant) column accepts any JSON-compatible value. On Spark 4.0+, build
+VARIANT values with `parse_json`:
+
+```sql
+INSERT INTO events VALUES
+ ('evt_001', parse_json('{"action": "click", "x": 120, "y": 450}'), 1000),
+ ('evt_002', parse_json('{"action": "purchase", "items": ["sku_a", "sku_b"], "total": 59.99}'), 1001),
+ ('evt_003', parse_json('"simple string value"'), 1002);
+```
+
+:::note Mapping to write operations
+Hudi offers flexibility in choosing the underlying [write operation](write_operations.md) of a `INSERT INTO` statement using
+the `hoodie.spark.sql.insert.into.operation` configuration. Possible options include *"bulk_insert"* (large inserts), *"insert"* (with small file management),
+and *"upsert"* (with deduplication/merging). If ordering fields are not set, *"insert"* is chosen as the default. For a table with ordering fields set (via `orderingFields`),
+*"upsert"* is chosen as the default operation.
+:::
+
+
+### Insert Overwrite
+
+The `INSERT OVERWRITE` statement is used to replace existing data in a Hudi table.
+
+```sql
+INSERT OVERWRITE <table>
+SELECT <columns> FROM <source>;
+```
+
+All existing partitions that are affected by the `INSERT OVERWRITE` statement will replaced with the source data.
+Here are some examples:
+
+```sql
+-- Overwrite non-partitioned table
+INSERT OVERWRITE hudi_mor_tbl SELECT 99, 'a99', 20.0, 900;
+INSERT OVERWRITE hudi_cow_nonpcf_tbl SELECT 99, 'a99', 20.0;
+
+-- Overwrite partitioned table with dynamic partition
+INSERT OVERWRITE TABLE hudi_cow_pt_tbl SELECT 10, 'a10', 1100, '2021-12-09', '10';
+
+-- Overwrite partitioned table with static partition
+INSERT OVERWRITE hudi_cow_pt_tbl PARTITION(dt = '2021-12-09', hh='12') SELECT 13, 'a13', 1100;
+```
+
+### Update
+You can use the `UPDATE` statement to modify existing data in a Hudi table directly.
+
+```sql
+UPDATE tableIdentifier SET column = EXPRESSION(,column = EXPRESSION) [ WHERE boolExpression]
+```
+
+Here's an example:
+
+```sql
+-- Update data in a Hudi table
+UPDATE hudi_mor_tbl SET price = price * 2, ts = 1111 WHERE id = 1;
+
+-- Update data in a partitioned Hudi table
+UPDATE hudi_cow_pt_tbl SET name = 'a1_1', ts = 1001 WHERE id = 1;
+
+-- update using non-PK field
+update hudi_cow_pt_tbl set ts = 1001 where name = 'a1';
+```
+
+:::info
+The `UPDATE` operation requires the specification of ordering fields (via `orderingFields`).
+:::
+
+### Merge Into
+
+The `MERGE INTO` statement allows you to perform more complex updates and merges against source data. The `MERGE INTO` statement
+is similar to the `UPDATE` statement, but it allows you to specify different actions for matched and unmatched records.
+
+```sql
+MERGE INTO tableIdentifier AS target_alias
+USING (sub_query | tableIdentifier) AS source_alias
+ON <merge_condition>
+[ WHEN MATCHED [ AND <condition> ] THEN <matched_action> ]
+[ WHEN NOT MATCHED [ AND <condition> ] THEN <not_matched_action> ]
+
+<merge_condition> =A equal bool condition
+<matched_action> =
+ DELETE |
+ UPDATE SET * |
+ UPDATE SET column1 = expression1 [, column2 = expression2 ...]
+<not_matched_action> =
+ INSERT * |
+ INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 ...])
+```
+
+:::info
+`MERGE INTO` statement does not support evolving table schema. Please use DDL (e.g., `ALTER TABLE`) or Datasource write (`df.write.format("hudi")....save(basePath)`) to evolve table schema.
+
+`WHEN NOT MATCHED` clauses specify the action to perform if the values do not match.
+There are two kinds of `INSERT` clauses:
+1. `INSERT *` clauses require that the source table has the same columns as those in the target table.
+2. `INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 ...])` clauses do not require to specify all the columns of the target table. For unspecified target columns, insert the `NULL` value.
+
+For a Hudi table with user configured primary keys, the join condition and the `UPDATE`/`INSERT INTO` clause in `MERGE INTO` is expected to contain the primary keys of the table.
+
+For a table where Hudi auto generates primary keys, the join condition in `MERGE INTO` can be on any arbitrary data columns.
+
+if the `hoodie.record.merge.mode` is set to `EVENT_TIME_ORDERING`, ordering fields (via `orderingFields`) are required to be set with value in the `UPDATE`/`INSERT` clause.
+
+It is enforced that if the target table has primary key and partition key column, the source table counterparts must enforce the same data type accordingly. Plus, if the target table is configured with `hoodie.record.merge.mode` = `EVENT_TIME_ORDERING` where target table is expected to have valid ordering fields configuration, the source table counterpart must also have the same data type.
+:::
+
+Examples below
+
+```sql
+-- source table using hudi for testing merging into non-partitioned table
+create table merge_source (id int, name string, price double, ts bigint) using hudi
+tblproperties (primaryKey = 'id', orderingFields = 'ts');
+insert into merge_source values (1, "old_a1", 22.22, 900), (2, "new_a2", 33.33, 2000), (3, "new_a3", 44.44, 2000);
+
+merge into hudi_mor_tbl as target
+using merge_source as source
+on target.id = source.id
+when matched then update set *
+when not matched then insert *
+;
+
+-- source table using parquet for testing merging into partitioned table
+create table merge_source2 (id int, name string, flag string, dt string, hh string) using parquet;
+insert into merge_source2 values (1, "new_a1", 'update', '2021-12-09', '10'), (2, "new_a2", 'delete', '2021-12-09', '11'), (3, "new_a3", 'insert', '2021-12-09', '12');
+
+MERGE into hudi_cow_pt_tbl as target
+using (
+ select id, name, '1000' as ts, flag, dt, hh from merge_source2
+) source
+on target.id = source.id
+when matched and flag != 'delete' then
+ update set id = source.id, name = source.name, ts = source.ts, dt = source.dt, hh = source.hh
+when matched and flag = 'delete' then delete
+when not matched then
+ insert (id, name, ts, dt, hh) values(source.id, source.name, source.ts, source.dt, source.hh)
+;
+
+```
+
+:::note Key requirements
+For a Hudi table with user configured primary keys, the join condition in `Merge Into` is expected to contain the primary keys of the table.
+For a Table where Hudi auto generates primary keys, the join condition in MIT can be on any arbitrary data columns.
+:::
+
+### Merge Into with Partial Updates {#merge-into-partial-update}
+
+Partial updates only write updated columns instead of full change records. This is useful when you have wide tables (typical for ML feature stores)
+with hundreds of columns and only a few columns are updated. It reduces the write amplification as well as helps in lowering the query
+latency. `MERGE INTO` statement above can be modified to use partial updates as shown below.
+
+```sql
+-- Create a Merge-on-Read table
+CREATE TABLE tableName (
+ id INT,
+ name STRING,
+ price DOUBLE,
+ _ts LONG,
+ description STRING
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = '_ts'
+)
+LOCATION '/location/to/basePath';
+
+-- Insert values into the table
+INSERT INTO tableName VALUES
+ (1, 'a1', 10, 1000, 'a1: desc1'),
+ (2, 'a2', 20, 1200, 'a2: desc2'),
+ (3, 'a3', 30, 1250, 'a3: desc3');
+
+-- Perform partial updates using a MERGE INTO statement
+MERGE INTO tableName t0
+ USING (
+ SELECT 1 AS id, 'a1' AS name, 12 AS price, 1001 AS ts
+ UNION ALL
+ SELECT 3 AS id, 'a3' AS name, 25 AS price, 1260 AS ts
+ ) s0
+ ON t0.id = s0.id
+ WHEN MATCHED THEN UPDATE SET
+ price = s0.price,
+ _ts = s0.ts;
+
+SELECT id, name, price, _ts, description FROM tableName;
+```
+
+Notice, instead of `UPDATE SET *`, we are updating only the `price` and `_ts` columns.
+
+:::note
+Partial update is not yet supported in the following cases:
+1. When the target table is a bootstrapped table.
+2. When virtual keys is enabled.
+3. When schema on read is enabled.
+4. When there is an enum field in the source data.
+5. When a global index is enabled, e.g., GLOBAL_SIMPLE.
+
+For a Hudi table with user configured primary keys, the join condition and the `UPDATE`/`INSERT INTO` clause in `MERGE INTO` is expected to contain the primary keys of the table.
+:::
+
+### Delete From
+
+You can remove data from a Hudi table using the `DELETE FROM` statement.
+
+```sql
+DELETE FROM tableIdentifier [ WHERE boolExpression ]
+```
+
+Examples below
+
+```sql
+-- Delete data from a Hudi table
+DELETE FROM hudi_cow_nonpcf_tbl WHERE uuid = 1;
+
+-- Delete data from a MOR Hudi table based on a condition
+DELETE FROM hudi_mor_tbl WHERE id % 2 = 0;
+
+-- Delete data using a non-primary key field
+DELETE FROM hudi_cow_pt_tbl WHERE name = 'a1';
+```
+
+### Data Skipping and Indexing
+
+DML operations can be sped up using column statistics for data skipping and using indexes to reduce the amount of data scanned.
+For e.g. the following helps speed up the `DELETE` operation on a Hudi table, by using the record level index.
+
+```sql
+SET hoodie.metadata.record.index.enable=true;
+
+DELETE from hudi_table where uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa';
+```
+
+These DML operations give you powerful tools for managing your tables using Spark SQL.
+You can control the behavior of these operations using various configuration options, as explained in the documentation.
+
+## Flink SQL
+
+Flink SQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update and delete data from your Hudi tables. Let's explore them one by one.
+
+### Insert Into
+
+You can utilize the INSERT INTO statement to incorporate data into a Hudi table using Flink SQL. Here are a few illustrative examples:
+
+```sql
+INSERT INTO <table>
+SELECT <columns> FROM <source>;
+```
+
+Examples:
+
+```sql
+-- Insert into a Hudi table
+INSERT INTO hudi_table SELECT 1, 'a1', 20;
+```
+
+If the `write.operation` is 'upsert,' the INSERT INTO statement will not only insert new records but also update existing rows with the same record key.
+
+```sql
+-- Insert into a Hudi table in upsert mode
+INSERT INTO hudi_table/*+ OPTIONS('write.operation'='upsert')*/ SELECT 1, 'a1', 20;
+```
+
+### Update
+With Flink SQL, you can use update command to update the hudi table. Here are a few illustrative examples:
+
+```sql
+UPDATE tableIdentifier SET column = EXPRESSION(,column = EXPRESSION) [ WHERE boolExpression]
+```
+
+```sql
+UPDATE hudi_table SET price = price * 2, ts = 1111 WHERE id = 1;
+```
+
+:::note Key requirements
+Update query only work with batch excution mode.
+:::
+
+### Delete From
+With Flink SQL, you can use delete command to delete the rows from hudi table. Here are a few illustrative examples:
+
+```sql
+DELETE FROM tableIdentifier [ WHERE boolExpression ]
+```
+
+```sql
+DELETE FROM hudi_table WHERE price < 100;
+```
+
+
+```sql
+DELETE FROM hudi_table WHERE price < 100;
+```
+
+:::note Key requirements
+Delete query only work with batch excution mode.
+:::
+
+### Lookup Joins
+
+A lookup join is typically used to enrich a table with data that is queried from an external system. The join requires
+one table to have a processing time attribute and the other table to be backed by a lookup source connector.
+
+```sql
+CREATE TABLE datagen_source(
+ id int,
+ name STRING,
+ proctime as PROCTIME()
+) WITH (
+'connector' = 'datagen',
+'rows-per-second'='1',
+'number-of-rows' = '2',
+'fields.id.kind'='sequence',
+'fields.id.start'='1',
+'fields.id.end'='2'
+);
Review Comment:
π€ Same `excution` β `execution` typo here as in the Update note above. Also note: a `DELETE FROM hudi_table WHERE price < 100;` block immediately precedes this note and another identical block appears right after β looks like the example was accidentally duplicated. Consider removing one of the two identical SQL blocks.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/timeline.md:
##########
@@ -0,0 +1,158 @@
+---
+title: Timeline
+toc: true
+toc_min_heading_level: 2
+toc_max_heading_level: 4
+---
+
+Changes to table state (writes, table services, schema changes, etc) are recorded as **_actions_** in the Hudi **_timeline_**. The Hudi timeline is a log of all actions performed
+on the table at different **_instants_** (points in time). It is a key component of Hudi's architecture, acting as a source of truth for the state of the table. All instant times
+used on the timeline follow [TrueTime](https://research.google/pubs/spanner-truetime-and-the-cap-theorem/) semantics, and are monotonically increasing globally across various
+processes involved. See TrueTime section below for more details.
+
+Each action has the following attributes associated with it.
+
+* **requested instant** : Instant time representing when the action was requested on the timeline and acts as the transaction id. An immutable plan for the action should be generated before the action is requested.
+* **completed instant** : Instant time representing when the action was completed on the timeline. All relevant changes to table data/metadata should be made before the action is completed.
+* **state** : state of the action. valid states are `REQUESTED`, `INFLIGHT` and `COMPLETED` during an action's lifecycle.
+* **type** : the kind of action performed. See below for full list of actions.
+
+
+<p align = "center">Figure: Actions in the timeline</p>
+
+### Action Types
+
+Following are the valid action types.
+
+* **COMMIT** - Write operation denoting an atomic write of a batch of records into a base files in the table.
+* **DELTA_COMMIT** - Write operation denoting an atomic write of a batch of records into merge-on-read type table, where some/all of the data could be just written to delta logs.
+* **REPLACE_COMMIT** - Write operation that atomically replaces a set of file groups in the table with another. Used for implementing batch write operations like _insert_overwrite_, _delete_partition_ etc, as well as table services
+ like clustering.
+* **CLEANS** - Table service that removes older file slices that are no longer needed from the table, by deleting those files.
+* **COMPACTION** - Table service to reconcile differential data between base and delta files, by merging delta files into base files.
+* **LOGCOMPACTION** - Table service to merge multiple small log files into a bigger log file in the same file slice.
+* **CLUSTERING** - Table service to rewrite existing file groups with optimized sort order or storage layouts, as new file groups in the table.
+* **INDEXING** - Table service to build an index of a requested type on a column of the table, consistent with the state of the table at the completed instant in face of ongoing writes.
+* **ROLLBACK** - Indicates that an unsuccessful write operation was rolled back, removing any partial/uncommitted files produced during such a write from storage.
+* **SAVEPOINT** - Marks certain file slices as "saved", such that cleaner will not delete them. It helps restore the table to a point on the timeline, in case of disaster/data recovery scenarios or perform time-travel queries as of those instants.
+* **RESTORE** - Restores a table to a given savepoint on the timeline, in case of disaster/data recovery scenarios.
+
+In some cases, action types in the completed state may be different from requested/inflight states, but still tracked by the same requested instant. For e.g. _CLUSTERING_ as in requested/inflight state,
+becomes _REPLACE_COMMIT_ in completed state. Compactions complete as _COMMIT_ action on the timeline producing new base files. In general, multiple write operations from the storage engine
+may map to the same action on the timeline.
+
+### State Transitions
+Actions go through state transitions on the timeline, with each transition recorded by a file of the pattern `<requsted instant>.<action>.<state>`(for other states) or
+`<requsted instant>_<completed instant>.<action>` (for COMPLETED state). Hudi guarantees that the state transitions are atomic and timeline consistent based on the instant time.
+Atomicity is achieved by relying on the atomic operations on the underlying storage (e.g. PUT calls to S3/Cloud Storage).
Review Comment:
π€ Two typos in the same line: `<requsted instant>` should be `<requested instant>` (both occurrences). This is the canonical file-pattern description, so it's worth getting the naming right.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_queries.md:
##########
@@ -0,0 +1,980 @@
+---
+title: SQL Queries
+summary: "In this page, we go over querying Hudi tables using SQL"
+toc: true
+last_modified_at: 2026-05-29T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
+This page will show how to issue different queries and discuss any specific instructions for each query engine.
+
+## Spark SQL
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+
+:::tip Setting Hudi read options at the session level
+Hudi 1.2.0 supports setting read options at the **Spark session level** using the `spark.hoodie.*` prefix.
+Any `spark.hoodie.X` config set via `spark.conf.set` or `--conf` is treated equivalently to `hoodie.X`.
+
+Config precedence (low β high):
+1. Global DFS properties
+2. `spark.hoodie.*` session-level configs (normalized to `hoodie.*`)
+3. Explicit `hoodie.*` data source options or per-table `SET` commands
+
+```sql
+-- Apply a Hudi read option for the entire session
+SET spark.hoodie.metadata.column.stats.enable = true;
+SELECT * FROM hudi_table WHERE price BETWEEN 10.0 AND 50.0;
+```
+
+If both `spark.hoodie.X` and `hoodie.X` are set, the explicit `hoodie.X` value takes precedence.
+:::
+
+### Snapshot Query
+Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+Using session properties, you can specify options around indexing to optimize query performance, as shown below.
+
+```sql
+-- You can turn on relevant options for indexing.
+
+-- Turn on use of column stat index, to perform range queries.
+SET hoodie.metadata.column.stats.enable=true;
+SELECT * FROM hudi_table
+WHERE price > 1.0 and price < 10.0
+
+-- Turn on use of record level index, to perform point queries.
+SET hoodie.metadata.record.index.enable=true;
+SELECT * FROM hudi_table
+WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa'
+```
+
+:::note Integration with Spark
+Users are encouraged to migrate to Hudi versions > 0.12.x, for the best spark experience and discouraged from using any older approaches
+using path filters. We expect that native integration with Spark's optimized table readers along with Hudi's automatic table
+management will yield great performance benefits in those versions.
+:::
+
+### Snapshot Query with Index Acceleration
+
+In this section we would go over the various indexes and how they help in data skipping in Hudi. We will first create
+a hudi table without any index.
+
+```sql
+-- Create a table with primary key
+CREATE TABLE hudi_indexed_table (
+ ts BIGINT,
+ uuid STRING,
+ rider STRING,
+ driver STRING,
+ fare DOUBLE,
+ city STRING
+) USING HUDI
+options(
+ primaryKey ='uuid',
+ hoodie.write.record.merge.mode = "COMMIT_TIME_ORDERING"
+)
+PARTITIONED BY (city);
+
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo' ),
+(1695376420,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo' ),
+(1695173887,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai' ),
+(1695115999,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+UPDATE hudi_indexed_table SET rider = 'rider-B', driver = 'driver-N', ts = '1697516137' WHERE rider = 'rider-A';
+```
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data. Let's create a secondary index on the rider column.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+```
+
+
+<p align = "left">Figure: Query pruning without secondary index</p>
+
+### Query using Secondary Index
+
+We will run the query again after creating secondary index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+:::note
+Please note in order to create secondary index:
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
+2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
+:::
+
+```sql
+-- We will first create a record index since secondary index is dependent upon it
+CREATE INDEX record_index ON hudi_indexed_table (uuid);
+-- We create a secondary index on rider column
+CREATE INDEX idx_rider ON hudi_indexed_table (rider);
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+DROP INDEX record_index on hudi_indexed_table;
+DROP INDEX secondary_index_idx_rider on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with secondary index</p>
+
+### Query using Bloom Filter Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet on the `driver` column.
+All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+```
+
+
+<p align = "left">Figure: Query pruning without bloom filter expression index</p>
+
+We will run the query again after creating bloom filter expression index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a bloom filter expression index on driver column
+CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+DROP INDEX expr_index_idx_bloom_driver on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with bloom filter expression index</p>
+
+### Query using Column Stats Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+```
+
+
+<p align = "left">Figure: Query pruning without column stat expression index</p>
+
+We will run the query again after creating column stat expression index on ts column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a column stat expression index on ts column
+CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(expr='from_unixtime', format = 'yyyy-MM-dd');
+-- We run the same query again
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+DROP INDEX expr_index_idx_column_ts on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with column stat expression index</p>
+
+### Query using Partition Stats Index
+
+With the query run below, we will see no data skipping or pruning since there is no partition stats index created yet in the table as can
+be seen in the image below. All the partitions are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+```
+
+
+<p align = "left">Figure: Query pruning without partition stats index</p>
+
+We will run the query again after creating partition stats index. The query would now show the partitions scanned as 1
+compared to 3 partitions scanned without index.
+
+```sql
+-- We will need to enable column stats as well since partition stats index leverages it
+SET hoodie.metadata.index.partition.stats.enable=true;
+SET hoodie.metadata.index.column.stats.enable=true;
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'854g46e0-8355-45cc-97c6-c31daf0df330','rider-H','driver-T',19.10,'chennai');
+-- Run the query again on the table with partition stats index
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+DROP INDEX column_stats on hudi_indexed_table;
+DROP INDEX partition_stats on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with partition stats index</p>
+
+### Snapshot Query with Event Time Ordering
+
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
+time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
+event time ordering merge mode.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'EVENT_TIME_ORDERING'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but lower ts
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 900, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=1000, price=10.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode;
+```
+
+With `EVENT_TIME_ORDERING`, the record with the larger event time (specified via `orderingFields`) overwrites the record with the
+smaller event time on the same key, regardless of transaction time.
+
+### Snapshot Query with Custom Merge Mode
+
+Users can set `CUSTOM` mode to provide their own merge logic. With `CUSTOM` merge mode, you also need to provide your
+payload class that implements the merge logic. For example, you can use `PartialUpdateAvroPayload` to merge the records
+as below.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'CUSTOM',
+ 'hoodie.datasource.write.payload.class' = 'org.apache.hudi.common.model.PartialUpdateAvroPayload'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but set higher ts and name as null to show partial update
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, null, 2000, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=2000, price=20.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode_custom;
+```
+
+As you can see, not only the record with higher ordering field overwrites the record with lower ordering value, but also
+the name field is partially updated.
+
+### Time Travel Query
+
+You can also query the table at a specific commit time using the `AS OF` syntax. This is useful for debugging and auditing purposes, as well as for
+machine learning pipelines where you want to train models on a specific point in time.
+
+```sql
+SELECT * FROM <table name>
+TIMESTAMP AS OF '<timestamp in yyyy-MM-dd HH:mm:ss.SSS or yyyy-MM-dd or yyyyMMddHHmmssSSS>'
+WHERE <filter conditions>
+```
+
+### Change Data Capture
+
+Change Data Capture (CDC) queries are useful when you want to obtain all changes to a Hudi table within a given time window, along with before/after images and change operation
+of the changed records. Similar to many relational database counterparts, Hudi provides flexible ways of controlling supplemental logging levels, to balance storage/logging costs
+by materializing more versus compute costs of computing the changes on the fly, using `hoodie.table.cdc.supplemental.logging.mode` configuration.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
+ <pathToTable | tableName>,
+ 'cdc',
+ <'earliest' | <time to capture from>>
+ [, <time to capture to>]
+)
+```
+
+# add note on checkpoint translation from 0.x to 1.x. same for incremental query below
+:::note CDC Query Checkpointing between Hudi 0.x and 1.x
+In Hudi 1.0, we switch the incremental and CDC queries to used completion time, instead of requested instant time, to determine the
+range of commits to incrementally pull from. The checkpoint stored for Hudi incremental source and related sources is
+also changed to use completion time. To seamless migration without downtime or data duplication, Hudi does an automatic checkpoint
+translation from requested instant time to completion time depending on the source table version.
+:::
+
+### Incremental Query
+
+Incremental queries are useful when you want to obtain the latest values for all records that have changed after a given commit time. They help author incremental data pipelines with
+orders of magnitude efficiency over batch counterparts by only processing the changed records. Hudi users have realized [large gains](https://www.uber.com/blog/ubers-lakehouse-architecture/) in
+query efficiency by using incremental queries in this fashion. Hudi supports incremental queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
+ <pathToTable | tableName>,
+ 'latest_state',
+ <'earliest' | <time to capture from>>
+ [, <time to capture to>]
+)
+```
+
+:::info Incremental vs CDC Queries
+Incremental queries offer even better query efficiency than even the CDC queries above, since they amortize the cost of compactions across your data lake.
+For e.g the table has received 10 million modifications across 1 million records over a time window, incremental queries can fetch the latest value for
+1 million records using Hudi's record level metadata. On the other hand, the CDC queries will process 10 million records and useful in cases, where you want to
+see all changes in a given time window and not just the latest values.
+:::
+
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
+
+:::note Incremental Query Checkpointing between Hudi 0.x and 1.0.
+In Hudi 1.0, we switch the incremental and CDC query to used completion time, instead of instant time, to determine the
+range of commits to incrementally pull from. The checkpoint stored for Hudi incremental source and related sources is
+also changed to use completion time. To support compatiblity, Hudi does a checkpoint translation from requested instant
+time to completion time depending on the source table version.
+:::
+
+### Vector Similarity Search
+
+The `hudi_vector_search` table-valued function (TVF) runs top-K similarity search over a
+[`VECTOR`](sql_ddl.md#vector) column. It returns the `top_k` rows whose VECTOR column is closest to
+a query vector under the chosen distance metric.
+
+```sql
+SELECT *
+FROM hudi_vector_search(
+ table_name, -- STRING: registered table name or path
+ vector_column, -- STRING: VECTOR column name
+ query_vector, -- ARRAY: query embedding
+ top_k, -- INT: number of nearest neighbors
+ [distance_metric], -- STRING: 'cosine' (default), 'l2', 'dot_product'
+ [algorithm] -- STRING: 'brute_force' (default)
+)
+```
+
+Parameters:
+
+| Parameter | Type | Default | Description |
+|:----------|:-----|:--------|:------------|
+| `table_name` | STRING | (required) | Registered table name or table path. |
+| `vector_column` | STRING | (required) | Name of the VECTOR column. |
+| `query_vector` | ARRAY<FLOAT> | (required) | Query embedding; must match the column's dimension and element type. |
+| `top_k` | INT | (required) | Number of nearest neighbors. |
+| `distance_metric` | STRING | `'cosine'` | One of `'cosine'`, `'l2'`, `'dot_product'`. |
+| `algorithm` | STRING | `'brute_force'` | Only `'brute_force'` is currently supported. |
+
+Return schema: all columns from the source table (excluding the embedding column) plus
+`_hudi_distance DOUBLE`. Results are ordered by `_hudi_distance` ascending, so the closest matches
+come first.
Review Comment:
π€ The rows in this MDT partition type table are out of numerical order (`Record Index = 5`, then `Secondary Index = 7`, then `Partition Stats = 6`). Could you reorder so Partition Stats (6) comes before Secondary Index (7)? Readers using the table to look up a type value will find it easier when type values are monotonically increasing.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_queries.md:
##########
@@ -0,0 +1,980 @@
+---
+title: SQL Queries
+summary: "In this page, we go over querying Hudi tables using SQL"
+toc: true
+last_modified_at: 2026-05-29T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
+This page will show how to issue different queries and discuss any specific instructions for each query engine.
+
+## Spark SQL
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+
+:::tip Setting Hudi read options at the session level
+Hudi 1.2.0 supports setting read options at the **Spark session level** using the `spark.hoodie.*` prefix.
+Any `spark.hoodie.X` config set via `spark.conf.set` or `--conf` is treated equivalently to `hoodie.X`.
+
+Config precedence (low β high):
+1. Global DFS properties
+2. `spark.hoodie.*` session-level configs (normalized to `hoodie.*`)
+3. Explicit `hoodie.*` data source options or per-table `SET` commands
+
+```sql
+-- Apply a Hudi read option for the entire session
+SET spark.hoodie.metadata.column.stats.enable = true;
+SELECT * FROM hudi_table WHERE price BETWEEN 10.0 AND 50.0;
+```
+
+If both `spark.hoodie.X` and `hoodie.X` are set, the explicit `hoodie.X` value takes precedence.
+:::
+
+### Snapshot Query
+Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+Using session properties, you can specify options around indexing to optimize query performance, as shown below.
+
+```sql
+-- You can turn on relevant options for indexing.
+
+-- Turn on use of column stat index, to perform range queries.
+SET hoodie.metadata.column.stats.enable=true;
+SELECT * FROM hudi_table
+WHERE price > 1.0 and price < 10.0
+
+-- Turn on use of record level index, to perform point queries.
+SET hoodie.metadata.record.index.enable=true;
+SELECT * FROM hudi_table
+WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa'
+```
+
+:::note Integration with Spark
+Users are encouraged to migrate to Hudi versions > 0.12.x, for the best spark experience and discouraged from using any older approaches
+using path filters. We expect that native integration with Spark's optimized table readers along with Hudi's automatic table
+management will yield great performance benefits in those versions.
+:::
+
+### Snapshot Query with Index Acceleration
+
+In this section we would go over the various indexes and how they help in data skipping in Hudi. We will first create
+a hudi table without any index.
+
+```sql
+-- Create a table with primary key
+CREATE TABLE hudi_indexed_table (
+ ts BIGINT,
+ uuid STRING,
+ rider STRING,
+ driver STRING,
+ fare DOUBLE,
+ city STRING
+) USING HUDI
+options(
+ primaryKey ='uuid',
+ hoodie.write.record.merge.mode = "COMMIT_TIME_ORDERING"
+)
+PARTITIONED BY (city);
+
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo' ),
+(1695376420,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo' ),
+(1695173887,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai' ),
+(1695115999,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+UPDATE hudi_indexed_table SET rider = 'rider-B', driver = 'driver-N', ts = '1697516137' WHERE rider = 'rider-A';
+```
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data. Let's create a secondary index on the rider column.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+```
+
+
+<p align = "left">Figure: Query pruning without secondary index</p>
+
+### Query using Secondary Index
+
+We will run the query again after creating secondary index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+:::note
+Please note in order to create secondary index:
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
+2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
+:::
+
+```sql
+-- We will first create a record index since secondary index is dependent upon it
+CREATE INDEX record_index ON hudi_indexed_table (uuid);
+-- We create a secondary index on rider column
+CREATE INDEX idx_rider ON hudi_indexed_table (rider);
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+DROP INDEX record_index on hudi_indexed_table;
+DROP INDEX secondary_index_idx_rider on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with secondary index</p>
+
+### Query using Bloom Filter Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet on the `driver` column.
+All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+```
+
+
+<p align = "left">Figure: Query pruning without bloom filter expression index</p>
+
+We will run the query again after creating bloom filter expression index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a bloom filter expression index on driver column
+CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+DROP INDEX expr_index_idx_bloom_driver on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with bloom filter expression index</p>
+
+### Query using Column Stats Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+```
+
+
+<p align = "left">Figure: Query pruning without column stat expression index</p>
+
+We will run the query again after creating column stat expression index on ts column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a column stat expression index on ts column
+CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(expr='from_unixtime', format = 'yyyy-MM-dd');
+-- We run the same query again
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+DROP INDEX expr_index_idx_column_ts on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with column stat expression index</p>
+
+### Query using Partition Stats Index
+
+With the query run below, we will see no data skipping or pruning since there is no partition stats index created yet in the table as can
+be seen in the image below. All the partitions are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+```
+
+
+<p align = "left">Figure: Query pruning without partition stats index</p>
+
+We will run the query again after creating partition stats index. The query would now show the partitions scanned as 1
+compared to 3 partitions scanned without index.
+
+```sql
+-- We will need to enable column stats as well since partition stats index leverages it
+SET hoodie.metadata.index.partition.stats.enable=true;
+SET hoodie.metadata.index.column.stats.enable=true;
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'854g46e0-8355-45cc-97c6-c31daf0df330','rider-H','driver-T',19.10,'chennai');
+-- Run the query again on the table with partition stats index
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+DROP INDEX column_stats on hudi_indexed_table;
+DROP INDEX partition_stats on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with partition stats index</p>
+
+### Snapshot Query with Event Time Ordering
+
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
+time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
+event time ordering merge mode.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'EVENT_TIME_ORDERING'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but lower ts
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 900, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=1000, price=10.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode;
+```
+
+With `EVENT_TIME_ORDERING`, the record with the larger event time (specified via `orderingFields`) overwrites the record with the
+smaller event time on the same key, regardless of transaction time.
+
+### Snapshot Query with Custom Merge Mode
+
+Users can set `CUSTOM` mode to provide their own merge logic. With `CUSTOM` merge mode, you also need to provide your
+payload class that implements the merge logic. For example, you can use `PartialUpdateAvroPayload` to merge the records
+as below.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'CUSTOM',
+ 'hoodie.datasource.write.payload.class' = 'org.apache.hudi.common.model.PartialUpdateAvroPayload'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but set higher ts and name as null to show partial update
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, null, 2000, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=2000, price=20.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode_custom;
+```
+
+As you can see, not only the record with higher ordering field overwrites the record with lower ordering value, but also
+the name field is partially updated.
+
+### Time Travel Query
+
+You can also query the table at a specific commit time using the `AS OF` syntax. This is useful for debugging and auditing purposes, as well as for
+machine learning pipelines where you want to train models on a specific point in time.
+
+```sql
+SELECT * FROM <table name>
+TIMESTAMP AS OF '<timestamp in yyyy-MM-dd HH:mm:ss.SSS or yyyy-MM-dd or yyyyMMddHHmmssSSS>'
+WHERE <filter conditions>
+```
+
+### Change Data Capture
+
+Change Data Capture (CDC) queries are useful when you want to obtain all changes to a Hudi table within a given time window, along with before/after images and change operation
+of the changed records. Similar to many relational database counterparts, Hudi provides flexible ways of controlling supplemental logging levels, to balance storage/logging costs
+by materializing more versus compute costs of computing the changes on the fly, using `hoodie.table.cdc.supplemental.logging.mode` configuration.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
+ <pathToTable | tableName>,
+ 'cdc',
+ <'earliest' | <time to capture from>>
+ [, <time to capture to>]
+)
+```
+
+# add note on checkpoint translation from 0.x to 1.x. same for incremental query below
+:::note CDC Query Checkpointing between Hudi 0.x and 1.x
+In Hudi 1.0, we switch the incremental and CDC queries to used completion time, instead of requested instant time, to determine the
+range of commits to incrementally pull from. The checkpoint stored for Hudi incremental source and related sources is
+also changed to use completion time. To seamless migration without downtime or data duplication, Hudi does an automatic checkpoint
+translation from requested instant time to completion time depending on the source table version.
+:::
+
+### Incremental Query
+
+Incremental queries are useful when you want to obtain the latest values for all records that have changed after a given commit time. They help author incremental data pipelines with
+orders of magnitude efficiency over batch counterparts by only processing the changed records. Hudi users have realized [large gains](https://www.uber.com/blog/ubers-lakehouse-architecture/) in
+query efficiency by using incremental queries in this fashion. Hudi supports incremental queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
+ <pathToTable | tableName>,
+ 'latest_state',
+ <'earliest' | <time to capture from>>
+ [, <time to capture to>]
+)
+```
+
+:::info Incremental vs CDC Queries
+Incremental queries offer even better query efficiency than even the CDC queries above, since they amortize the cost of compactions across your data lake.
+For e.g the table has received 10 million modifications across 1 million records over a time window, incremental queries can fetch the latest value for
+1 million records using Hudi's record level metadata. On the other hand, the CDC queries will process 10 million records and useful in cases, where you want to
+see all changes in a given time window and not just the latest values.
+:::
+
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
+
+:::note Incremental Query Checkpointing between Hudi 0.x and 1.0.
+In Hudi 1.0, we switch the incremental and CDC query to used completion time, instead of instant time, to determine the
+range of commits to incrementally pull from. The checkpoint stored for Hudi incremental source and related sources is
+also changed to use completion time. To support compatiblity, Hudi does a checkpoint translation from requested instant
+time to completion time depending on the source table version.
+:::
+
+### Vector Similarity Search
+
+The `hudi_vector_search` table-valued function (TVF) runs top-K similarity search over a
+[`VECTOR`](sql_ddl.md#vector) column. It returns the `top_k` rows whose VECTOR column is closest to
+a query vector under the chosen distance metric.
+
+```sql
+SELECT *
Review Comment:
π€ Minor wording cleanup: `secondary keys stores in the secondary index partition` β `secondary keys stored in the secondary index partition`. Also two lines later, `All the different index types can be queries by specifying...` β `queried`.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/storage_layouts.md:
##########
@@ -0,0 +1,161 @@
+---
+title: Storage Layouts
+toc: false
+---
+
+The following describes the general organization of files in storage for a Hudi table.
+
+* Hudi organizes data tables into a directory structure under a **_base path_** on a storage.
+* Tables are optionally broken up into **_partitions_**, based on partition columns defined in the table schema.
+* Within each partition, files are organized into **_file groups_**, uniquely identified by a file ID (uuid)
+* Each file group contains several **_file slices_**.
+* Each slice contains a **_base file_** (parquet/orc/hfile) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) )
+ written by a commit that completed at a certain instant, along with set of **_log files_** (*.log.*) written by commits that completed
+ before the next base file's requested instant.
+* Hudi employs Multiversion Concurrency Control (MVCC), where [compaction](compaction.md) action merges logs and base files to produce new
+ file slices and [cleaning](cleaning.md) action gets rid of unused/older file slices to reclaim space on the file system.
+* All metadata including timeline, metadata table are stored in a special `.hoodie` directory under the base path.
+
+
+
+Please refer the **[tech spec](https://hudi.apache.org/tech-specs#file-layout-hierarchy)** for a more detailed description of the file layouts.
+
+### Base Files
+
+Base files store full records, against which change records are stored in delta log files below. Hudi currently supports following
+base file formats.
+
+* columnar formats for vectorized reads, columnar compression and efficient column based access for analytics/data science.
+* row-oriented avro files for fast scans for reading whole records.
+* random access optimized HFiles for efficient searching for indexed records (based on [SSTable](https://github.com/facebook/rocksdb/wiki/A-Tutorial-of-RocksDB-SST-formats) format)
+* [Lance](#lance-base-file-format) files for native `VECTOR` storage and AI/ML workloads (Spark-only).
+
+
+
+#### Lance base file format {#lance-base-file-format}
+
+[Lance](https://lancedb.github.io/lance/) is a pluggable base file format selected per table via
+`hoodie.table.base.file.format = 'lance'`. Hudi manages the table layer
+(timeline, metadata, schema, file groups, table services); Lance is the on-disk file format for
+base files. Log files for MOR tables remain Avro; log compaction merges Avro logs into Lance base
+files.
+
+```sql
+-- COW
+CREATE TABLE my_ai_table (
+ id STRING,
+ embedding VECTOR(768),
Review Comment:
π€ The Lance CREATE TABLE example sets `hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger'`. This config typically accepts a comma-separated list of merger class names. Is `DefaultSparkRecordMerger` the correct FQCN required here, and is it a required property for Lance-backed tables, or only recommended? A short sentence clarifying why this needs to be set explicitly (and that it's a single value here, not the full list) would help users not familiar with the merger API. @yihua could you confirm the recommended value here?
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/sql_queries.md:
##########
@@ -0,0 +1,980 @@
+---
+title: SQL Queries
+summary: "In this page, we go over querying Hudi tables using SQL"
+toc: true
+last_modified_at: 2026-05-29T00:00:00-00:00
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
+This page will show how to issue different queries and discuss any specific instructions for each query engine.
+
+## Spark SQL
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+
+:::tip Setting Hudi read options at the session level
+Hudi 1.2.0 supports setting read options at the **Spark session level** using the `spark.hoodie.*` prefix.
+Any `spark.hoodie.X` config set via `spark.conf.set` or `--conf` is treated equivalently to `hoodie.X`.
+
+Config precedence (low β high):
+1. Global DFS properties
+2. `spark.hoodie.*` session-level configs (normalized to `hoodie.*`)
+3. Explicit `hoodie.*` data source options or per-table `SET` commands
+
+```sql
+-- Apply a Hudi read option for the entire session
+SET spark.hoodie.metadata.column.stats.enable = true;
+SELECT * FROM hudi_table WHERE price BETWEEN 10.0 AND 50.0;
+```
+
+If both `spark.hoodie.X` and `hoodie.X` are set, the explicit `hoodie.X` value takes precedence.
+:::
+
+### Snapshot Query
+Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+Using session properties, you can specify options around indexing to optimize query performance, as shown below.
+
+```sql
+-- You can turn on relevant options for indexing.
+
+-- Turn on use of column stat index, to perform range queries.
+SET hoodie.metadata.column.stats.enable=true;
+SELECT * FROM hudi_table
+WHERE price > 1.0 and price < 10.0
+
+-- Turn on use of record level index, to perform point queries.
+SET hoodie.metadata.record.index.enable=true;
+SELECT * FROM hudi_table
+WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa'
+```
+
+:::note Integration with Spark
+Users are encouraged to migrate to Hudi versions > 0.12.x, for the best spark experience and discouraged from using any older approaches
+using path filters. We expect that native integration with Spark's optimized table readers along with Hudi's automatic table
+management will yield great performance benefits in those versions.
+:::
+
+### Snapshot Query with Index Acceleration
+
+In this section we would go over the various indexes and how they help in data skipping in Hudi. We will first create
+a hudi table without any index.
+
+```sql
+-- Create a table with primary key
+CREATE TABLE hudi_indexed_table (
+ ts BIGINT,
+ uuid STRING,
+ rider STRING,
+ driver STRING,
+ fare DOUBLE,
+ city STRING
+) USING HUDI
+options(
+ primaryKey ='uuid',
+ hoodie.write.record.merge.mode = "COMMIT_TIME_ORDERING"
+)
+PARTITIONED BY (city);
+
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
+(1695091554,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
+(1695046462,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
+(1695332066,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
+(1695516137,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo' ),
+(1695376420,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo' ),
+(1695173887,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai' ),
+(1695115999,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
+UPDATE hudi_indexed_table SET rider = 'rider-B', driver = 'driver-N', ts = '1697516137' WHERE rider = 'rider-A';
+```
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data. Let's create a secondary index on the rider column.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+```
+
+
+<p align = "left">Figure: Query pruning without secondary index</p>
+
+### Query using Secondary Index
+
+We will run the query again after creating secondary index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+:::note
+Please note in order to create secondary index:
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
+2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
+:::
+
+```sql
+-- We will first create a record index since secondary index is dependent upon it
+CREATE INDEX record_index ON hudi_indexed_table (uuid);
+-- We create a secondary index on rider column
+CREATE INDEX idx_rider ON hudi_indexed_table (rider);
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';
+DROP INDEX record_index on hudi_indexed_table;
+DROP INDEX secondary_index_idx_rider on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with secondary index</p>
+
+### Query using Bloom Filter Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet on the `driver` column.
+All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+```
+
+
+<p align = "left">Figure: Query pruning without bloom filter expression index</p>
+
+We will run the query again after creating bloom filter expression index on rider column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a bloom filter expression index on driver column
+CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
+-- We run the same query again
+SELECT * FROM hudi_indexed_table WHERE driver = 'driver-N';
+DROP INDEX expr_index_idx_bloom_driver on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with bloom filter expression index</p>
+
+### Query using Column Stats Expression Index
+
+With the query run below, we will see no data skipping or pruning since there is no index created yet in the table as can
+be seen in the image below. All the files are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+```
+
+
+<p align = "left">Figure: Query pruning without column stat expression index</p>
+
+We will run the query again after creating column stat expression index on ts column. The query would now
+show the files scanned as 1 compared to 3 files scanned without index.
+
+```sql
+-- We create a column stat expression index on ts column
+CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(expr='from_unixtime', format = 'yyyy-MM-dd');
+-- We run the same query again
+SELECT uuid, rider FROM hudi_indexed_table WHERE from_unixtime(ts, 'yyyy-MM-dd') = '2023-10-17';
+DROP INDEX expr_index_idx_column_ts on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with column stat expression index</p>
+
+### Query using Partition Stats Index
+
+With the query run below, we will see no data skipping or pruning since there is no partition stats index created yet in the table as can
+be seen in the image below. All the partitions are scanned in the table to fetch the data.
+
+```sql
+SHOW INDEXES FROM hudi_indexed_table;
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+```
+
+
+<p align = "left">Figure: Query pruning without partition stats index</p>
+
+We will run the query again after creating partition stats index. The query would now show the partitions scanned as 1
+compared to 3 partitions scanned without index.
+
+```sql
+-- We will need to enable column stats as well since partition stats index leverages it
+SET hoodie.metadata.index.partition.stats.enable=true;
+SET hoodie.metadata.index.column.stats.enable=true;
+INSERT INTO hudi_indexed_table
+VALUES
+(1695159649,'854g46e0-8355-45cc-97c6-c31daf0df330','rider-H','driver-T',19.10,'chennai');
+-- Run the query again on the table with partition stats index
+SELECT * FROM hudi_indexed_table WHERE rider >= 'rider-H';
+DROP INDEX column_stats on hudi_indexed_table;
+DROP INDEX partition_stats on hudi_indexed_table;
+```
+
+
+<p align = "left">Figure: Query pruning with partition stats index</p>
+
+### Snapshot Query with Event Time Ordering
+
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
+time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
+event time ordering merge mode.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'EVENT_TIME_ORDERING'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but lower ts
+INSERT INTO hudi_table_merge_mode VALUES (1, 'a1', 900, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=1000, price=10.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode;
+```
+
+With `EVENT_TIME_ORDERING`, the record with the larger event time (specified via `orderingFields`) overwrites the record with the
+smaller event time on the same key, regardless of transaction time.
+
+### Snapshot Query with Custom Merge Mode
+
+Users can set `CUSTOM` mode to provide their own merge logic. With `CUSTOM` merge mode, you also need to provide your
+payload class that implements the merge logic. For example, you can use `PartialUpdateAvroPayload` to merge the records
+as below.
+
+```sql
+CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
+ id INT,
+ name STRING,
+ ts LONG,
+ price DOUBLE
+) USING hudi
+TBLPROPERTIES (
+ type = 'mor',
+ primaryKey = 'id',
+ orderingFields = 'ts',
+ recordMergeMode = 'CUSTOM',
+ 'hoodie.datasource.write.payload.class' = 'org.apache.hudi.common.model.PartialUpdateAvroPayload'
+)
+LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
+
+-- insert a record
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, 'a1', 1000, 10.0);
+
+-- another record with the same key but set higher ts and name as null to show partial update
+INSERT INTO hudi_table_merge_mode_custom VALUES (1, null, 2000, 20.0);
+
+-- query the table, result should be id=1, name=a1, ts=2000, price=20.0
+SELECT id, name, ts, price FROM hudi_table_merge_mode_custom;
+```
+
+As you can see, not only the record with higher ordering field overwrites the record with lower ordering value, but also
+the name field is partially updated.
+
+### Time Travel Query
+
+You can also query the table at a specific commit time using the `AS OF` syntax. This is useful for debugging and auditing purposes, as well as for
+machine learning pipelines where you want to train models on a specific point in time.
+
+```sql
+SELECT * FROM <table name>
+TIMESTAMP AS OF '<timestamp in yyyy-MM-dd HH:mm:ss.SSS or yyyy-MM-dd or yyyyMMddHHmmssSSS>'
+WHERE <filter conditions>
+```
+
+### Change Data Capture
+
+Change Data Capture (CDC) queries are useful when you want to obtain all changes to a Hudi table within a given time window, along with before/after images and change operation
+of the changed records. Similar to many relational database counterparts, Hudi provides flexible ways of controlling supplemental logging levels, to balance storage/logging costs
+by materializing more versus compute costs of computing the changes on the fly, using `hoodie.table.cdc.supplemental.logging.mode` configuration.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
+ <pathToTable | tableName>,
+ 'cdc',
+ <'earliest' | <time to capture from>>
+ [, <time to capture to>]
+)
+```
+
+# add note on checkpoint translation from 0.x to 1.x. same for incremental query below
+:::note CDC Query Checkpointing between Hudi 0.x and 1.x
+In Hudi 1.0, we switch the incremental and CDC queries to used completion time, instead of requested instant time, to determine the
+range of commits to incrementally pull from. The checkpoint stored for Hudi incremental source and related sources is
+also changed to use completion time. To seamless migration without downtime or data duplication, Hudi does an automatic checkpoint
+translation from requested instant time to completion time depending on the source table version.
+:::
+
+### Incremental Query
+
+Incremental queries are useful when you want to obtain the latest values for all records that have changed after a given commit time. They help author incremental data pipelines with
+orders of magnitude efficiency over batch counterparts by only processing the changed records. Hudi users have realized [large gains](https://www.uber.com/blog/ubers-lakehouse-architecture/) in
+query efficiency by using incremental queries in this fashion. Hudi supports incremental queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
+
+```sql
+-- Supported through the hudi_table_changes TVF
+SELECT *
+FROM hudi_table_changes(
+ <pathToTable | tableName>,
+ 'latest_state',
+ <'earliest' | <time to capture from>>
+ [, <time to capture to>]
+)
+```
+
+:::info Incremental vs CDC Queries
+Incremental queries offer even better query efficiency than even the CDC queries above, since they amortize the cost of compactions across your data lake.
+For e.g the table has received 10 million modifications across 1 million records over a time window, incremental queries can fetch the latest value for
+1 million records using Hudi's record level metadata. On the other hand, the CDC queries will process 10 million records and useful in cases, where you want to
+see all changes in a given time window and not just the latest values.
+:::
+
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
+
+:::note Incremental Query Checkpointing between Hudi 0.x and 1.0.
+In Hudi 1.0, we switch the incremental and CDC query to used completion time, instead of instant time, to determine the
+range of commits to incrementally pull from. The checkpoint stored for Hudi incremental source and related sources is
+also changed to use completion time. To support compatiblity, Hudi does a checkpoint translation from requested instant
+time to completion time depending on the source table version.
+:::
+
+### Vector Similarity Search
+
+The `hudi_vector_search` table-valued function (TVF) runs top-K similarity search over a
+[`VECTOR`](sql_ddl.md#vector) column. It returns the `top_k` rows whose VECTOR column is closest to
+a query vector under the chosen distance metric.
+
+```sql
+SELECT *
+FROM hudi_vector_search(
+ table_name, -- STRING: registered table name or path
+ vector_column, -- STRING: VECTOR column name
+ query_vector, -- ARRAY: query embedding
+ top_k, -- INT: number of nearest neighbors
+ [distance_metric], -- STRING: 'cosine' (default), 'l2', 'dot_product'
+ [algorithm] -- STRING: 'brute_force' (default)
+)
+```
+
+Parameters:
+
+| Parameter | Type | Default | Description |
+|:----------|:-----|:--------|:------------|
+| `table_name` | STRING | (required) | Registered table name or table path. |
+| `vector_column` | STRING | (required) | Name of the VECTOR column. |
+| `query_vector` | ARRAY<FLOAT> | (required) | Query embedding; must match the column's dimension and element type. |
+| `top_k` | INT | (required) | Number of nearest neighbors. |
+| `distance_metric` | STRING | `'cosine'` | One of `'cosine'`, `'l2'`, `'dot_product'`. |
+| `algorithm` | STRING | `'brute_force'` | Only `'brute_force'` is currently supported. |
+
+Return schema: all columns from the source table (excluding the embedding column) plus
+`_hudi_distance DOUBLE`. Results are ordered by `_hudi_distance` ascending, so the closest matches
+come first.
+
+Distance metrics:
+
+| Metric | Formula | Range | Notes |
+|:-------|:--------|:------|:------|
+| `cosine` | 1 β cos(a, b), clamped to [0, 2] | [0, 2] | Returns 1.0 for zero vectors. |
+| `l2` | sqrt(sum((a[i] β b[i])Β²)) | [0, +β) | |
+| `dot_product` | β(a Β· b) | (ββ, +β) | Negated so ascending sort surfaces the most similar rows first. |
+
+```sql
+-- Find the 10 nearest neighbors to a query embedding
+SELECT product_id, name, _hudi_distance
+FROM hudi_vector_search('products', 'embedding',
+ ARRAY(0.12, -0.03, 0.87, /* ... */),
+ 10, 'cosine')
+ORDER BY _hudi_distance;
+```
+
+RAG context retrieval to apply a distance threshold to drop weak matches:
+
+```sql
+-- Retrieve the 5 most relevant document chunks for an LLM prompt
+SELECT chunk_id, text_content, _hudi_distance
+FROM hudi_vector_search(
+ 'document_chunks', 'embedding',
+ ARRAY(/* embedding of the user's question */),
+ 5, 'cosine'
+)
+WHERE _hudi_distance < 0.3;
+```
+
+Cross-modal search to query a corpus of image embeddings with a text embedding (e.g., CLIP):
+
+```sql
+SELECT image_id, caption, _hudi_distance
+FROM hudi_vector_search(
+ 'image_catalog', 'clip_embedding',
+ ARRAY(/* text embedding from CLIP */),
+ 20, 'cosine'
+);
+```
+
+The TVF also works on Lance-backed tables. Set `hoodie.table.base.file.format=lance` at table
+creation (see [Storage Layouts β Lance](storage_layouts.md#lance-base-file-format)); the call site
+is unchanged.
+
+:::note
+`cosine` distance computes `1 β cos(a, b)`. If embeddings are not L2-normalized before write,
+results reflect both vector direction and magnitude.
+:::
+
+#### `hudi_vector_search_batch`
+
+The batch variant runs many query vectors at once:
+
+```sql
+SELECT *
+FROM hudi_vector_search_batch(
+ corpus_table, -- STRING
+ corpus_embedding_col, -- STRING
+ query_table, -- STRING: table holding query vectors
+ query_embedding_col, -- STRING: VECTOR column in query table
+ top_k, -- INT: neighbors per query
+ [distance_metric],
+ [algorithm]
+)
+```
+
+Return schema: corpus columns + query columns + `_hudi_distance DOUBLE` +
+`_hudi_query_index LONG` (identifies the query that produced the row). If corpus and query share
+column names, query columns are prefixed with `_hudi_query_`.
+
+### Reading BLOB Columns
+
+The `read_blob()` SQL function materializes raw bytes from a [`BLOB`](sql_ddl.md#blob) column. It
+handles both storage modes uniformly: for `INLINE`, it extracts the embedded bytes; for
+`OUT_OF_LINE`, it reads `reference.length` bytes starting at `reference.offset` from
+`reference.external_path`.
+
+```sql
+SELECT asset_id, read_blob(content) AS raw_bytes
+FROM media_assets
+WHERE asset_id = 'asset_001';
+```
+
+Standard queries on a BLOB column return the underlying struct (descriptor + bytes), not raw bytes:
+
+```sql
Review Comment:
π€ Typo in the metadata column description: `The commit requence number of the record` β `The commit sequence number of the record`.
<sub><i>- AI-generated; verify before applying. React π/π to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org