You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by xu...@apache.org on 2025/10/29 19:28:55 UTC
(hudi) branch asf-site updated: docs(blog): add Hudi indexing subsystem blog part 1 of 2 (#14179)
This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 57c6c1161dfa docs(blog): add Hudi indexing subsystem blog part 1 of 2 (#14179)
57c6c1161dfa is described below
commit 57c6c1161dfac108f25ca1b9f4669c0f8a0410eb
Author: Shiyan Xu <27...@users.noreply.github.com>
AuthorDate: Wed Oct 29 12:28:48 2025 -0700
docs(blog): add Hudi indexing subsystem blog part 1 of 2 (#14179)
---
...ve-into-hudis-indexing-subsystem-part-1-of-2.md | 213 +++++++++++++++++++++
.../fig1.png | Bin 0 -> 57480 bytes
.../fig2.png | Bin 0 -> 34132 bytes
.../fig3.png | Bin 0 -> 62125 bytes
.../fig4.png | Bin 0 -> 63637 bytes
.../fig5.png | Bin 0 -> 55763 bytes
.../fig6.png | Bin 0 -> 55373 bytes
7 files changed, 213 insertions(+)
diff --git a/website/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2.md b/website/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2.md
new file mode 100644
index 000000000000..e1cd9a03d495
--- /dev/null
+++ b/website/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2.md
@@ -0,0 +1,213 @@
+---
+title: "Deep Dive Into Hudi’s Indexing Subsystem (Part 1 of 2)"
+excerpt: ""
+author: Shiyan Xu
+category: blog
+image: /assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig1.png
+tags:
+- hudi
+- indexing
+- data lakehouse
+- data skipping
+---
+
+For decades, databases have relied on indexes—specialized data structures—to dramatically improve read and write performance by quickly locating specific records. Apache Hudi extends this fundamental principle to the data lakehouse with a unique and powerful approach. Every Hudi table contains a self-managed metadata table that functions as an indexing subsystem, enabling efficient data skipping and fast record lookups across a wide range of read and write scenarios.
+
+This two-part series dives into Hudi’s indexing subsystem. Part 1 explains the internal layout and data-skipping capabilities. Part 2 covers advanced features—record, secondary, and expression indexes—and asynchronous index maintenance. By the end, you’ll know how to leverage Hudi’s multimodal index to build more efficient lakehouse tables.
+
+## The Metadata Table
+
+Within a Hudi table (the data table), the metadata table itself is a Hudi Merge-on-Read (MOR) table. Unlike a typical data table, it features a specialized layout. The table is physically partitioned by index type, with each partition containing the relevant index entries. For its physical storage, the metadata table uses HFile as the base file format. This choice is deliberate: HFile is exceptionally efficient at handling key lookups—the predominant query pattern for indexing. Let’s exp [...]
+
+### Multimodal indexing
+
+The metadata table is often referred to as a multimodal index because it houses a diverse range of index types, providing versatile capabilities to accelerate various query patterns. The following diagram illustrates the layout of the metadata table and its relationship with the main data table.
+
+
+
+The metadata table is located in the `.hoodie/metadata/` directory under the data table’s base path. It contains partitions for different indexes, such as the files index (under the `files/` partition) for tracking the data table’s partitions and files, and the column stats index (under the `column_stats/` partition) for tracking file-level statistics (e.g., min/max values) for specific columns. Each index partition stores mapping entries tailored to its specific purpose.
+
+This partitioned design provides great flexibility, allowing you to enable only the indexes that suit your workload. It also ensures extensibility, making it straightforward to support new index types in the future. For example, the [bitmap index](https://github.com/apache/hudi/blob/master/rfc/rfc-92/rfc-92.md) and the vector search index are on the [roadmap](https://hudi.apache.org/roadmap) and will be maintained in their own dedicated partitions.
+
+When committing to a data table, the metadata table is updated within the same transactional write. This crucial step ensures that index entries are always synchronized with data table records, upholding data integrity across the table. Therefore, choosing Merge-on-Read (MOR) as the table type for the metadata table is an obvious choice. MOR offers the advantage of absorbing high-frequency write operations, preventing the metadata table’s update process from becoming a bottleneck for ove [...]
+
+### HFile format
+
+The HFile format stores key-value pairs in a sorted, immutable, and block-indexed way, modeled after Google’s SSTable introduced by the [Bigtable paper](https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf). Here is the description of SSTable quoted from the paper:
+
+> An SSTable provides a persistent, ordered immutable map from keys to values, where both keys and values are arbitrary byte strings. Operations are provided to look up the value associated with a specified key, and to iterate over all key/value pairs in a specified key range. Internally, each SSTable contains a sequence of blocks (typically each block is 64KB in size, but this is configurable). A block index (stored at the end of the SSTable) is used to locate blocks; the index is loade [...]
+
+As you can tell, by implementing the SSTable, HFile is especially efficient at performing random access, which is the primary query pattern for indexing—given a specific piece of information, like a record key or a partition value, return matching results, such as the file ID that contains the record key, or the list of files that belong to the partition.
+
+
+
+Because the keys in an HFile are stored in lexicographic order, a batched lookup with a common key prefix is also highly efficient, requiring only a sequential read of nearby keys.
+
+### Default behaviors
+
+When a Hudi table is created, the metadata table will be enabled with three partitions by default: *files*, *column stats*, and *partition stats*:
+
+* **Files**: stores the list of all partitions and the lists of all base files and log files of each partition, located at the `files/` partition of the metadata table.
+* **Column stats**: stores file-level statistics like min, max, value count, and null count for specified columns, located at the `column_stats/` partition of the metadata table.
+* **Partition stats**: stores partition-level statistics like min, max, value count, and null count for specified columns, located at the `partition_stats/` partition of the metadata table.
+
+By default, when no column is specified for column\_stats and partition\_stats, Hudi will index the first 32 columns (controlled by `hoodie.metadata.index.column.stats.max.columns.to.index`) available in the table schema.
+
+Whenever a new write is performed on the data table, the metadata table will be updated accordingly. For any available index, new index entries will be upserted to its corresponding partition. For example, if the new write creates a new partition in the data table with some new base files, the files partition will be updated and contain the latest partition and file lists. Similarly, the column stats and partition stats partitions will receive new entries indicating the updated statistic [...]
+
+Note that by design, you cannot disable the files partition, as it is a fundamental index that serves both read and write processes. You can still, although not recommended, disable the entire metadata table by setting `hoodie.metadata.enable=false` during a write.
+
+We will discuss more details about how the default indexes work to improve read and write performance. We will also introduce more indexes supported by the metadata table with usage examples in the following sections.
+
+## Data Skipping with Files, Column Stats, and Partition Stats
+
+Data skipping is a core optimization technique that avoids unnecessary data scanning. Its most basic form is physical partitioning, where data is organized into directories based on columns like `order_date` in a customer order table. When a query filters on a partitioned column, the engine uses *partition pruning* to read only the relevant directories. More advanced techniques store lightweight statistics—such as min/max values—for data within each file. The query engine consults this m [...]
+
+### The data skipping process
+
+Hudi’s indexing subsystem implements a multi-level skipping strategy using a combination of indexes. Query engines like Spark or Trino can leverage Hudi’s files, partition stats, and column stats indexes to improve performance dramatically. The process, illustrated in the figure below, unfolds in several stages.
+
+
+
+First, the query engine parses the input SQL and extracts relevant filter predicates, such as `price >= 300`. These predicates are pushed down to Hudi’s integration component, which manages the index lookup process.
+
+The component then consults the files index to get an initial list of partitions. It prunes this list using the partition stats index, which holds partition-level statistics like min/max values. For example, any partition with a maximum price below 300 is skipped entirely.
+
+After this initial pruning, the component consults the files index again to retrieve the list of data files within the remaining partitions. This file list is pruned further using the column stats index, which provides the same min/max statistics at the file level.
+
+This multi-step process ensures that the query engine reads only the minimum set of files required to satisfy the query, significantly reducing the total amount of data processed.
+
+### SQL examples
+
+The following examples demonstrate data skipping in action. We will create a Hudi table and execute Spark SQL queries against it, starting with both partition and column stats disabled to establish a baseline.
+
+```sql
+CREATE TABLE orders (
+ order_id STRING,
+ price DECIMAL(12,2),
+ order_status STRING,
+ update_ts BIGINT,
+ shipping_date DATE,
+ shipping_country STRING
+) USING HUDI
+PARTITIONED BY (shipping_country)
+OPTIONS (
+ primaryKey = 'order_id',
+ preCombineField = 'update_ts',
+ hoodie.metadata.index.column.stats.enable = 'false',
+ hoodie.metadata.index.partition.stats.enable = 'false'
+);
+```
+
+And insert some sample data:
+
+```sql
+INSERT INTO orders VALUES
+('ORD001', 389.99, 'PENDING', 17495166353, DATE '2023-01-01', 'A'),
+('ORD002', 199.99, 'CONFIRMED', 17495167353, DATE '2023-01-01', 'A'),
+('ORD003', 59.50, 'SHIPPED', 17495168353, DATE '2023-01-11', 'B'),
+('ORD004', 99.00, 'PENDING', 17495169353, DATE '2023-02-09', 'B'),
+('ORD005', 19.99, 'PENDING', 17495170353, DATE '2023-06-12', 'C'),
+('ORD006', 5.99, 'SHIPPED', 17495171353, DATE '2023-07-31', 'C');
+```
+
+#### Only the files index
+
+With both column stats and partition stats disabled, only the files index is built during the insert operation. We’ll use the SQL below for our test:
+
+```sql
+SELECT order_id, price, shipping_country
+FROM orders
+WHERE price > 300;
+```
+
+This query looks for orders with price greater than 300, which only exist in partition 'A' (shipping\_country = 'A'). After running the SQL, here's what we see in the Spark UI:
+
+
+
+Spark read all 3 partitions and 3 files to find potential matches, but only 1 record from partition A actually satisfied the query condition.
+
+#### Enabling column stats
+
+Now let's enable column stats while keeping partition stats disabled. Note that we can't do it the other way around—partition stats requires column stats to be enabled first.
+
+```sql
+CREATE TABLE orders (
+ order_id STRING,
+ price DECIMAL(12,2),
+ order_status STRING,
+ update_ts BIGINT,
+ shipping_date DATE,
+ shipping_country STRING
+) USING HUDI
+PARTITIONED BY (shipping_country)
+OPTIONS (
+ primaryKey = 'order_id',
+ preCombineField = 'update_ts',
+ hoodie.metadata.index.column.stats.enable = 'true',
+ hoodie.metadata.index.partition.stats.enable = 'false'
+);
+```
+
+Running the same SQL gives us this in the Spark UI:
+
+
+
+Now it shows all 3 partitions but only 1 file was scanned. Without partition stats, the query engine couldn't prune partitions, but column stats successfully filtered out the non-matching files. The compute cost of examining those 2 irrelevant partitions and their files could have been avoided with partition stats enabled.
+
+#### Enabling column stats and partition stats
+
+Now let's enable partition stats as well. Since both indexes are enabled by default in Hudi 1.x, we can simply omit those additional configs from the CREATE statement:
+
+```sql
+CREATE TABLE orders (
+ order_id STRING,
+ price DECIMAL(12,2),
+ order_status STRING,
+ update_ts BIGINT,
+ shipping_date DATE,
+ shipping_country STRING
+) USING HUDI
+PARTITIONED BY (shipping_country)
+OPTIONS (
+ primaryKey = 'order_id',
+ preCombineField = 'update_ts'
+);
+```
+
+Running the same SQL gives us this in the Spark UI:
+
+
+
+Now we see the full pruning effect happened—only 1 relevant partition and 1 relevant file were scanned, thanks to both indexes working together. [This blog](https://hudi.apache.org/blog/2025/10/22/Partition_Stats_Enhancing_Column_Stats_in_Hudi_1.0/) shows a 93% reduction in query time running on a 1 TB dataset.
+
+### Configure relevant columns to be indexed
+
+By default, Hudi indexes the first 32 columns for both partition stats and column stats. This limit prevents excessive metadata overhead—each indexed column requires computing min, max, null-count, and value-count statistics for every partition and data file. In most cases, you only need to index a small subset of columns that are frequently used in query predicates. You can specify which columns to be indexed to reduce the maintenance costs:
+
+```sql
+CREATE TABLE orders (
+ order_id STRING,
+ price DECIMAL(12,2),
+ order_status STRING,
+ update_ts BIGINT,
+ shipping_date DATE,
+ shipping_country STRING
+) USING HUDI
+PARTITIONED BY (shipping_country)
+OPTIONS (
+ primaryKey = 'order_id',
+ preCombineField = 'update_ts',
+ 'hoodie.metadata.index.column.stats.column.list' = 'price,shipping_date'
+);
+```
+
+The config `hoodie.metadata.index.column.stats.column.list` applies to both partition stats and column stats. By indexing just the `price` and `shipping_date` columns, queries filtering on price comparisons or shipping date ranges will already see significant performance improvements.
+
+## Key Takeaways and What's Next
+
+Hudi’s metadata table is itself a Hudi Merge‑on‑Read (MOR) table that acts as a multimodal indexing subsystem. It is physically partitioned by index type (for example, `files/`, `column_stats/`, `partition_stats/`) and stores base files in the HFile (SSTable‑like) format. This layout provides fast point lookups and efficient batched scans by key prefix—exactly the access patterns indexing needs at lakehouse scale.
+
+Index maintenance happens transactionally alongside data writes, keeping index entries consistent with the data table. Periodic compaction merges log files into read‑optimized HFile base files to keep point lookups fast and predictable. On the read path, Hudi composes multiple indexes to minimize I/O: the files index enumerates candidates, partition stats prune irrelevant partitions, and column stats prune non‑matching files. In effect, the engine scans only the minimum set of files requ [...]
+
+In practice, the defaults are a strong starting point. Keep the metadata table enabled and explicitly list only the columns you frequently filter on via `hoodie.metadata.index.column.stats.column.list` to control metadata overhead. In Part 2, we’ll go deeper into accelerating equality‑matching and expression‑based predicates using the record, secondary, and expression indexes, and discuss how asynchronous index maintenance keeps writers unblocked while indexes build in the background.
diff --git a/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig1.png b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig1.png
new file mode 100644
index 000000000000..9c0da0c1802d
Binary files /dev/null and b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig1.png differ
diff --git a/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig2.png b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig2.png
new file mode 100644
index 000000000000..3ed598e50102
Binary files /dev/null and b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig2.png differ
diff --git a/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig3.png b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig3.png
new file mode 100644
index 000000000000..ffd98ec3b524
Binary files /dev/null and b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig3.png differ
diff --git a/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig4.png b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig4.png
new file mode 100644
index 000000000000..7a170b1db5ec
Binary files /dev/null and b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig4.png differ
diff --git a/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig5.png b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig5.png
new file mode 100644
index 000000000000..32a99550b488
Binary files /dev/null and b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig5.png differ
diff --git a/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig6.png b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig6.png
new file mode 100644
index 000000000000..f34cf0f8d996
Binary files /dev/null and b/website/static/assets/images/blog/2025-10-29-deep-dive-into-hudis-indexing-subsystem-part-1-of-2/fig6.png differ