You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 04:47:32 UTC
[I] Optimize FileStatus construction in file listing from metadata table [hudi]
hudi-bot opened a new issue, #15071:
URL: https://github.com/apache/hudi/issues/15071
In BaseTableMetadata::getAllFilesInPartitions, we construct FileStatus instances for the later use of reading files. This can be deferred, or the necessary information can be stored in simple POJO instead of FileStatus. The instantiation of FileStatus instances can also be parallelized.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4073
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-1292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize FileStatus construction in file listing from metadata table [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15071:
URL: https://github.com/apache/hudi/issues/15071#issuecomment-4795310218
This appears to be addressed on master. `BaseTableMetadata.getAllFilesInPartitions` no longer constructs `FileStatus` — it now returns `Map<String, List<StoragePathInfo>>`, where `StoragePathInfo` is a lightweight Hudi POJO described in its javadoc as `FileStatus` "with simplification based on what Hudi needs." The replacement was done in commit `bce71996eac7` ([HUDI-6497] Replace FileSystem, Path, and FileStatus usage in hudi-common module, #10591), which directly delivers the "simple POJO instead of FileStatus" optimization called out in this JIRA. Suggest closing as resolved by HUDI-6497 unless there is a remaining concern (e.g., parallelizing the per-partition payload->StoragePathInfo conversion in `fetchAllFilesInPartitionPaths`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize FileStatus construction in file listing from metadata table [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15071:
URL: https://github.com/apache/hudi/issues/15071#issuecomment-3593860129
This issue belongs to a closed JIRA Epic: https://issues.apache.org/jira/browse/HUDI-1292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize FileStatus construction in file listing from metadata table [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15071: Optimize FileStatus construction in file listing from metadata table
URL: https://github.com/apache/hudi/issues/15071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org