You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 08:32:50 UTC
[I] getAllPartitionPaths perf fix need to account for parquet/orc partition path meta files [hudi]
hudi-bot opened a new issue, #16124:
URL: https://github.com/apache/hudi/issues/16124
[https://github.com/apache/hudi/pull/9121/files?diff=split&w=0#r1263994796]
we might need to follow up to ensure we dont' break the parquet/orc partition meta file flows.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6591
- Type: Bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] getAllPartitionPaths perf fix need to account for parquet/orc partition path meta files [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16124:
URL: https://github.com/apache/hudi/issues/16124#issuecomment-4795539115
This appears to have been addressed in [PR #9366](https://github.com/apache/hudi/pull/9366) (commit `d600e98de63a`), which re-implemented the `getAllPartitionPaths` perf improvement after the original #9121 was reverted.
In the current code at [`FileSystemBackedTableMetadata.getPartitionPathWithPathPrefixUsingFilterExpression`](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/metadata/FileSystemBackedTableMetadata.java#L198-L219), parquet/orc partition meta files are accounted for in two places:
1. **Directory branch**: `HoodiePartitionMetadata.hasPartitionMetadata(...)` checks both text and base format (parquet/orc) via `baseFormatMetaPathIfExists` (`HoodiePartitionMetadata.java:222-258`).
2. **File branch**: For non-directory entries, `path.getName().startsWith(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE_PREFIX)` matches `.hoodie_partition_metadata`, `.hoodie_partition_metadata.parquet`, and `.hoodie_partition_metadata.orc` — covering the non-partitioned table case where the meta file appears as a direct child of the base path.
If I'm missing a scenario you intended to flag, could you point at the specific flow you're worried about? Otherwise this can likely be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] getAllPartitionPaths perf fix need to account for parquet/orc partition path meta files [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16124: getAllPartitionPaths perf fix need to account for parquet/orc partition path meta files
URL: https://github.com/apache/hudi/issues/16124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org