You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Sagar Sumit (Jira)" <ji...@apache.org> on 2024/04/18 16:27:00 UTC
[jira] [Created] (HUDI-7639) Refactor HoodieFileIndex so that different indexes can be used via optimizer rules
Sagar Sumit created HUDI-7639:
---------------------------------
Summary: Refactor HoodieFileIndex so that different indexes can be used via optimizer rules
Key: HUDI-7639
URL: https://issues.apache.org/jira/browse/HUDI-7639
Project: Apache Hudi
Issue Type: Task
Reporter: Sagar Sumit
Fix For: 1.0.0
Currently, `HoodieFileIndex` is responsible for partition pruning as well as file skipping. All indexes are being used in [lookupCandidateFilesInMetadataTable|https://github.com/apache/hudi/blob/b5b14f7d4fa6224a6674b021664b510c6ae8afb9/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala#L333] method through if-else branches. This is not only hard to maintain as we add more indexes, but also induces a static hierarchy. Instead, we need more flexibility so that we can alter logical plan based on availability of indexes. For partition pruning in Spark, we already have [HoodiePruneFileSourcePartitions|https://github.com/apache/hudi/blob/b5b14f7d4fa6224a6674b021664b510c6ae8afb9/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodiePruneFileSourcePartitions.scala#L40] rule but it is injected during the operator optimization batch and it does not modify the result of the LogicalPlan. To be fully extensible, we should be able to rewrite the LogicalPlan. We should be able to inject rules after partition pruning after the operator optimization batch and before any CBO rules that depend on stats. Spark provides [injectPreCBORules|https://github.com/apache/spark/blob/6232085227ee2cc4e831996a1ac84c27868a1595/sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala#L304] API to do so, however it is only available in Spark 3.1.0 onwards.
The goal of this ticket is to refactor index hierarchy and create new rules such that Spark version < 3.1.0 still go via the old path, while later versions can modify the plan using an appropriate index and inject as a pre-CBO rule.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)