[jira] [Created] (HUDI-9787) HoodieFileIndex file listing perf issue

["Davis Zhang (Jira)" <ji...@apache.org>]

Davis Zhang created HUDI-9787:
---------------------------------

             Summary: HoodieFileIndex file listing perf issue
                 Key: HUDI-9787
                 URL: https://issues.apache.org/jira/browse/HUDI-9787
             Project: Apache Hudi
          Issue Type: Bug
          Components: index
            Reporter: Davis Zhang
             Fix For: 1.2.0
         Attachments: image-2025-09-04-10-23-53-374.png

org.apache.hudi.HoodieFileIndex#filterFileSlices

 
{code:java}
val prunedPartitionsAndFilteredFileSlices = prunedPartitionsAndFileSlices.map {
  case (partitionOpt, fileSlices) =>
    // Filter in candidate files based on the col-stats or record level index lookup
    val candidateFileSlices: Seq[FileSlice] = {
      fileSlices.filter(fs => {
        val fileSliceFiles = fs.getLogFiles.map[String](JFunction.toJavaFunction[HoodieLogFile, String](lf => lf.getPath.getName))
          .collect(Collectors.toSet[String])
        val baseFileStatusOpt = getBaseFileStatus(Option.apply(fs.getBaseFile.orElse(null)))
        baseFileStatusOpt.exists(f => fileSliceFiles.add(f.getPath.getName))
        // NOTE: This predicate is true when {@code Option} is empty         candidateFilesNamesOpt.forall(files => files.exists(elem => fileSliceFiles.contains(elem)))
      })
    }

    totalFileSliceSize += fileSlices.size
    candidateFileSliceSize += candidateFileSlices.size
    (partitionOpt, candidateFileSlices)
}

val skippingRatio =
  if (!areAllFileSlicesCached) -1
  else if (getAllFiles().nonEmpty && totalFileSliceSize > 0)
    (totalFileSliceSize - candidateFileSliceSize) / totalFileSliceSize.toDouble
  else 0

logInfo(s"Total file slices: $totalFileSliceSize; " +
  s"candidate file slices after data skipping: $candidateFileSliceSize; " +
  s"skipping percentage $skippingRatio")
 {code}
This is doing a nested for loop like processing between 2 scala list candidateFilesNamesOpt and prunedPartitionsAndFileSlices to figure out the overlap.

 

In production use case (0.14.1 based branch), we saw that 
!image-2025-09-04-10-23-53-374.png!

It means both list are of size 55.8k. The total iteration the above for loop is 55.8k*55.8k which is not trivial.

We noticed on a cluster with 32 GB memory with no CPU throttling (this logic is anyway single threaded), in this set up the hoodie file index choose to use 
COLUMN_STATS_INDEX_PROCESSING_MODE_IN_MEMORY which means everything is on driver.
 
That nested for loop took ~10 min to complete. During this time, no spark tasks are running because this is driver only logic and there is no spark tasks available for execution. The performance impact could be more profound if the auto scaling logic of user's spark application choose to toss away executors and later scale them back. This churning would make the e2e spark application perf worse.
 
 
 
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)