You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 04:45:43 UTC
[I] Introduce Secondary Index to Improve HUDI Query Performance [hudi]
hudi-bot opened a new issue, #15051:
URL: https://github.com/apache/hudi/issues/15051
In query processing, we need to scan many data blocks in HUDI table. However, most of them may not match the query predicate even after using column statistic info in the metadata table, row group level or page level statistics in parquet files, etc.
The total data size of touched blocks determines the query speed, and how to save IO has become the key point to improving query performance. To address this problem, we introduce secondary index to improve hudi query performance.
In the initial implementation, secondary index based on lucene will be carried out.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3582
- Type: New Feature
- Epic: https://issues.apache.org/jira/browse/HUDI-466
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Introduce Secondary Index to Improve HUDI Query Performance [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15051:
URL: https://github.com/apache/hudi/issues/15051#issuecomment-4795422642
Secondary Index has been implemented in Apache Hudi via [RFC-77](https://github.com/apache/hudi/pull/10814) (HUDI-7146) and is available on master. Write path landed in #11146, reader integration in #11162, with many follow-on enhancements (nested fields, streaming MDT writes, table version 9 index format versioning, Trino support, etc.). Note: the final implementation is built on the Hudi metadata table rather than Lucene, but it achieves the goal of pruning files via secondary-key lookups. Closing as completed — please open a new issue if specific gaps remain. Relevant code: `hudi-common/src/main/java/org/apache/hudi/index/secondary/`, `hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SecondaryIndexSupport.scala`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Introduce Secondary Index to Improve HUDI Query Performance [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15051: Introduce Secondary Index to Improve HUDI Query Performance
URL: https://github.com/apache/hudi/issues/15051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org