You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2026/01/04 06:15:39 UTC
[PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
prashantwason opened a new pull request, #17775:
URL: https://github.com/apache/hudi/pull/17775
### Describe the issue this Pull Request addresses
This PR adds a new configuration option to enable listing partition paths from HMS (Hive MetaStore) instead of the file system in the Spark HoodieFileIndex.
When MDT (Metadata Table) is not enabled on the reader path, the list of partitions is fetched using FileSystemBackedTableMetadata which recursively lists all partitions from storage. This can be slow for large tables with hundreds of partitions and generates a large number of listing calls.
When the new config is enabled, the partition listing is retrieved from the HMS and pruned on the driver side, providing better performance than FileSystemBackedTableMetadata.
### Summary and Changelog
**New Config Added:**
- `hoodie.datasource.read.file.index.list.partitions.from.hms` (default: false)
**Changes:**
- Added new boolean config `FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED` in `DataSourceOptions.scala`
- Added `getMatchingPartitionPathsFromCatalog` method in `BaseHoodieTableFileIndex.java` that can be overridden by implementations
- Implemented `getMatchingPartitionPathsFromCatalog` in `SparkHoodieTableFileIndex.scala` to fetch partitions from Spark's external catalog (HMS)
- Updated `HoodieFileIndex.scala` to pass the new config property
### Impact
- New user-facing configuration option for Spark-based queries
- No breaking changes to existing APIs
- When enabled, partition listing is obtained from HMS instead of file system
### Risk Level
low - This is an opt-in feature with default value of false. The existing behavior remains unchanged unless explicitly enabled.
### Documentation Update
The config description is self-documenting with proper documentation string:
- `hoodie.datasource.read.file.index.list.partitions.from.hms`: Controls whether partition listing is obtained from the HMS instead of listing the file system to avoid recursively listing of large number of directories. Enabling this can reduce large amount of listing calls and speed up the queries for very large tables. This is only necessary when MDT is not enabled on the dataset as otherwise the MDT can provide the partition listing faster and without any actual listing on the file system.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3855788108
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d85848f4881fe7a71cd149e5d4545865c293e14 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391)
* 785469de431bfc4eb60f88ed72173817f51635b6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3947443391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3887734681",
"triggerType" : "MANUAL"
}, {
"hash" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234",
"triggerID" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"triggerType" : "PUSH"
}, {
"hash" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3947439778",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
* 6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3947420769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3887734681",
"triggerType" : "MANUAL"
}, {
"hash" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234",
"triggerID" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
* 6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-6024132681
Closing as superseded by #18265, which landed the same capability (catalog-backed partition listing when MDT is unavailable) via `CatalogBackedTableMetadata` and the `hoodie.datasource.read.file.index.list.partitions.from.catalog` config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #17775:
URL: https://github.com/apache/hudi/pull/17775#discussion_r2770789667
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -353,6 +361,27 @@ private HoodieTimeline findInstantsInRange() {
}
}
+ /**
+ * List partition paths matching path prefixes from the Catalog.
+ *
+ * File Index implementations can override this method to fetch partition paths from the Catalog. This may be faster
+ * than listing all partition paths from the table metadata and filtering them. This is definitely faster than
+ * listing all partition paths from the file system when metadata table may not be enabled.
+ *
+ * Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.
+ * <p>
+ * E.g., Table has partition 4 partitions:
+ * year=2022/month=08/day=30, year=2022/month=08/day=31, year=2022/month=07/day=03, year=2022/month=07/day=04
+ * The relative path "year=2022" returns all partitions, while the relative path
+ * "year=2022/month=07" returns only two partitions.
+ *
+ * @param relativePathPrefixes The prefixes to relative partition paths that must match
+ * @return null if not supported by File Index implementation
+ */
+ protected List<String> getMatchingPartitionPathsFromCatalog(List<String> relativePathPrefixes) {
Review Comment:
Done. Added `isPartitionListingViaCatalogEnabled()` boolean method in both the base class and SparkHoodieTableFileIndex. The code now uses this method instead of checking for null return values.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala:
##########
@@ -437,6 +439,56 @@ class SparkHoodieTableFileIndex(spark: SparkSession,
private def arePartitionPathsUrlEncoded: Boolean =
metaClient.getTableConfig.getUrlEncodePartitioning.toBoolean
+ /**
+ * List partition paths matching path prefixes from the Catalog.
+ *
+ * File Index implementations can override this method to fetch partition paths from the Catalog. This may be faster
+ * than listing all partition paths from the table metadata and filtering them. This is definitely faster than
+ * listing all partition paths from the file system when metadata table may not be enabled.
+ *
+ * Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.
+ * <p>
+ * E.g., Table has partition 4 partitions:
+ * year=2022/month=08/day=30, year=2022/month=08/day=31, year=2022/month=07/day=03, year=2022/month=07/day=04
+ * The relative path "year=2022" returns all partitions, while the relative path
+ * "year=2022/month=07" returns only two partitions.
+ *
+ * @param relativePathPrefixes The prefixes to relative partition paths that must match
+ * @return null if not supported by File Index implementation
+ */
+ override protected def getMatchingPartitionPathsFromCatalog(relativePathPrefixes: List[String]): List[String] = {
+ // If listing from the catalog is disabled, or if MDT is available (which is faster), return null
+ if (!configProperties.getBoolean(FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.key, FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.defaultValue())
+ || metaClient.getTableConfig.isMetadataTableAvailable) {
+ null
+ } else {
+ // Retrieve all the partition paths from the catalog
+ logInfo("Listing partition paths from the catalog using path prefixes " + relativePathPrefixes.toString)
+ val databaseName = metaClient.getTableConfig.getDatabaseName
+ val tableName = metaClient.getTableConfig.getTableName
+ val basePath = metaClient.getBasePath
+ val allPartitionPaths: Seq[String] = spark.sessionState.catalog.externalCatalog
Review Comment:
Done. Refactored the branching logic to separate the case when relativePathPrefixes is empty (return all partitions) from the case when filtering is needed. Also renamed the variable to `filteredPartitionPaths` as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3855774458
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d85848f4881fe7a71cd149e5d4545865c293e14 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391)
* 785469de431bfc4eb60f88ed72173817f51635b6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3887744707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3887734681",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3707778701
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b1819fdd2e9b999677b83e66c9f66ae89b242e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3803551638
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b1819fdd2e9b999677b83e66c9f66ae89b242e2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781)
* 3d85848f4881fe7a71cd149e5d4545865c293e14 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3857110723
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 785469de431bfc4eb60f88ed72173817f51635b6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704)
* 7fc33532dabd4e7a632669c054ded781908e8075 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3857119295
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 785469de431bfc4eb60f88ed72173817f51635b6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704)
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3857450702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 785469de431bfc4eb60f88ed72173817f51635b6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704)
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3857558630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3947439778
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3947776310
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/17775?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `25.00000%` with `6 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 45.42%. Comparing base ([`ec04479`](https://app.codecov.io/gh/apache/hudi/commit/ec044790fe71f04710472ad63e7859f1b29e3985?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6ee5d7e`](https://app.codecov.io/gh/apache/hudi/commit/6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 988 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/17775?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/BaseHoodieTableFileIndex.java](https://app.codecov.io/gh/apache/hudi/pull/17775?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2FBaseHoodieTableFileIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvQmFzZUhvb2RpZVRhYmxlRmlsZUluZGV4LmphdmE=) | 25.00% | [5 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/17775?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (ec04479) and HEAD (6ee5d7e). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (ec04479) | HEAD (6ee5d7e) |
>|------|------|------|
>||1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17775 +/- ##
=============================================
- Coverage 61.43% 45.42% -16.01%
+ Complexity 23082 8374 -14708
=============================================
Files 2108 1177 -931
Lines 127636 60245 -67391
Branches 14534 6415 -8119
=============================================
- Hits 78409 27365 -51044
+ Misses 42873 29906 -12967
+ Partials 6354 2974 -3380
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/17775/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/17775/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.42% <25.00%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/17775?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...java/org/apache/hudi/BaseHoodieTableFileIndex.java](https://app.codecov.io/gh/apache/hudi/pull/17775?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2FBaseHoodieTableFileIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvQmFzZUhvb2RpZVRhYmxlRmlsZUluZGV4LmphdmE=) | `59.89% <25.00%> (ø)` | |
... and [3242 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/17775/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3820214762
hey @prashantwason : looks like there are CI failures. can you follow up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3803539150
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b1819fdd2e9b999677b83e66c9f66ae89b242e2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781)
* 3d85848f4881fe7a71cd149e5d4545865c293e14 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #17775:
URL: https://github.com/apache/hudi/pull/17775#discussion_r2743470256
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -216,6 +216,16 @@ object DataSourceReadOptions {
" by carefully analyzing provided partition-column predicates and deducing corresponding partition-path prefix from " +
" them (if possible).")
+ val FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED: ConfigProperty[Boolean] =
+ ConfigProperty.key("hoodie.datasource.read.file.index.list.partitions.from.hms")
Review Comment:
how about
`hoodie.datasource.read.file.index.list.partitions.from.catalog`
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -353,6 +361,27 @@ private HoodieTimeline findInstantsInRange() {
}
}
+ /**
+ * List partition paths matching path prefixes from the Catalog.
+ *
+ * File Index implementations can override this method to fetch partition paths from the Catalog. This may be faster
+ * than listing all partition paths from the table metadata and filtering them. This is definitely faster than
+ * listing all partition paths from the file system when metadata table may not be enabled.
+ *
+ * Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.
+ * <p>
+ * E.g., Table has partition 4 partitions:
+ * year=2022/month=08/day=30, year=2022/month=08/day=31, year=2022/month=07/day=03, year=2022/month=07/day=04
+ * The relative path "year=2022" returns all partitions, while the relative path
+ * "year=2022/month=07" returns only two partitions.
+ *
+ * @param relativePathPrefixes The prefixes to relative partition paths that must match
+ * @return null if not supported by File Index implementation
+ */
+ protected List<String> getMatchingPartitionPathsFromCatalog(List<String> relativePathPrefixes) {
Review Comment:
can we also add a boolean method. Not a fan of relying on null return values.
then we cold do something like
```
boolean isPartitionListingViaCatalogEnabled = isPartitionListingViaCatalogEnabled()
if (isPartitionListingViaCatalogEnabled) {
matchedPartitionPaths = getMatchingPartitionPathsFromCatalog(...)
} else. {
matchedPartitionPaths = tableMetadata.getPartitionPathWithPathPrefixUsingFilterExpression(relativePartitionPaths,
partitionFields, partitionColumnPredicates);
}
```
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -527,6 +527,12 @@ object HoodieFileIndex extends Logging {
properties.setProperty(DataSourceReadOptions.FILE_INDEX_LISTING_MODE_OVERRIDE.key, listingModeOverride)
}
+ var hmsListingEnabled = getConfigValue(options, sqlConf,
Review Comment:
how about `partitionListingViaCatalog`
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala:
##########
@@ -437,6 +439,56 @@ class SparkHoodieTableFileIndex(spark: SparkSession,
private def arePartitionPathsUrlEncoded: Boolean =
metaClient.getTableConfig.getUrlEncodePartitioning.toBoolean
+ /**
+ * List partition paths matching path prefixes from the Catalog.
+ *
+ * File Index implementations can override this method to fetch partition paths from the Catalog. This may be faster
+ * than listing all partition paths from the table metadata and filtering them. This is definitely faster than
+ * listing all partition paths from the file system when metadata table may not be enabled.
+ *
+ * Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.
+ * <p>
+ * E.g., Table has partition 4 partitions:
+ * year=2022/month=08/day=30, year=2022/month=08/day=31, year=2022/month=07/day=03, year=2022/month=07/day=04
+ * The relative path "year=2022" returns all partitions, while the relative path
+ * "year=2022/month=07" returns only two partitions.
+ *
+ * @param relativePathPrefixes The prefixes to relative partition paths that must match
+ * @return null if not supported by File Index implementation
+ */
+ override protected def getMatchingPartitionPathsFromCatalog(relativePathPrefixes: List[String]): List[String] = {
+ // If listing from the catalog is disabled, or if MDT is available (which is faster), return null
+ if (!configProperties.getBoolean(FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.key, FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.defaultValue())
+ || metaClient.getTableConfig.isMetadataTableAvailable) {
+ null
+ } else {
+ // Retrieve all the partition paths from the catalog
+ logInfo("Listing partition paths from the catalog using path prefixes " + relativePathPrefixes.toString)
+ val databaseName = metaClient.getTableConfig.getDatabaseName
+ val tableName = metaClient.getTableConfig.getTableName
+ val basePath = metaClient.getBasePath
+ val allPartitionPaths: Seq[String] = spark.sessionState.catalog.externalCatalog
Review Comment:
we can branch this one out
```
if (relativePathPrefixes is not empty) {
.. filter and return.
// here lets name the var as filteredPartitionPaths instead of allPartitionPaths
} else {
return all partitons.
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3857431526
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 785469de431bfc4eb60f88ed72173817f51635b6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704)
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3947684274
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3887734681",
"triggerType" : "MANUAL"
}, {
"hash" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234",
"triggerID" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"triggerType" : "PUSH"
}, {
"hash" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234",
"triggerID" : "3947439778",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3887757015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3887734681",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3855769212
Addressed all review comments and pushed the changes. The CI should re-run automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #17775:
URL: https://github.com/apache/hudi/pull/17775#discussion_r2730609262
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala:
##########
@@ -437,6 +439,55 @@ class SparkHoodieTableFileIndex(spark: SparkSession,
private def arePartitionPathsUrlEncoded: Boolean =
metaClient.getTableConfig.getUrlEncodePartitioning.toBoolean
+ /**
+ * List partition paths matching path prefixes from the Catalog.
+ *
+ * File Index implementations can override this method to fetch partition paths from the Catalog. This may be faster
+ * than listing all partition paths from the table metadata and filtering them. This is definitely faster than
+ * listing all partition paths from the file system when metadata table may not be enabled.
+ *
+ * Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.
+ * <p>
+ * E.g., Table has partition 4 partitions:
+ * year=2022/month=08/day=30, year=2022/month=08/day=31, year=2022/month=07/day=03, year=2022/month=07/day=04
+ * The relative path "year=2022" returns all partitions, while the relative path
+ * "year=2022/month=07" returns only two partitions.
+ *
+ * @param relativePathPrefixes The prefixes to relative partition paths that must match
+ * @return null if not supported by File Index implementation
+ */
+ override protected def getMatchingPartitionPathsFromCatalog(relativePathPrefixes: List[String]): List[String] = {
+ // If listing from the catalog is disabled, return null
+ if (!configProperties.getBoolean(FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.key, FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.defaultValue())) {
+ null
+ } else {
+ // Retrieve all the partition paths from the catalog
Review Comment:
Good point! I've updated the logic to also check if MDT is available. Now the HMS listing path is only used when:
1. The config `FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED` is explicitly enabled, AND
2. MDT is NOT available (`isMetadataTableAvailable` returns false)
This ensures we prefer the faster MDT-based listing when it's available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3803977236
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d85848f4881fe7a71cd149e5d4545865c293e14 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3803534505
@nsivabalan Thanks for the review! I've addressed your comments:
**Tests Added:** Created `TestSparkHoodieTableFileIndexHmsListing.scala` with tests that verify:
- Config wiring is correct (HMS listing only used when enabled)
- When config is enabled and MDT is disabled, partitions come from catalog
- When MDT is available, HMS listing is skipped (prefers faster MDT-based listing)
- Partition prefix filtering works correctly
- End-to-end partition pruning works with HMS listing
**MDT Check Added:** Updated `getMatchingPartitionPathsFromCatalog` to also check `isMetadataTableAvailable`. The HMS listing path is now only used when the config is enabled AND MDT is not available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3857428442
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3887734681
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3947454098
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3887734681",
"triggerType" : "MANUAL"
}, {
"hash" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234",
"triggerID" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"triggerType" : "PUSH"
}, {
"hash" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234",
"triggerID" : "3947439778",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
* 6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12234)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3947408631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "7fc33532dabd4e7a632669c054ded781908e8075",
"triggerType" : "PUSH"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3857428442",
"triggerType" : "MANUAL"
}, {
"hash" : "7fc33532dabd4e7a632669c054ded781908e8075",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730",
"triggerID" : "3887734681",
"triggerType" : "MANUAL"
}, {
"hash" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fc33532dabd4e7a632669c054ded781908e8075 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11730)
* 6ee5d7ee3e69c81dfca5cc0df331003cdf2d68a4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #17775:
URL: https://github.com/apache/hudi/pull/17775#discussion_r2729653208
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala:
##########
@@ -437,6 +439,55 @@ class SparkHoodieTableFileIndex(spark: SparkSession,
private def arePartitionPathsUrlEncoded: Boolean =
metaClient.getTableConfig.getUrlEncodePartitioning.toBoolean
+ /**
+ * List partition paths matching path prefixes from the Catalog.
+ *
+ * File Index implementations can override this method to fetch partition paths from the Catalog. This may be faster
+ * than listing all partition paths from the table metadata and filtering them. This is definitely faster than
+ * listing all partition paths from the file system when metadata table may not be enabled.
+ *
+ * Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.
+ * <p>
+ * E.g., Table has partition 4 partitions:
+ * year=2022/month=08/day=30, year=2022/month=08/day=31, year=2022/month=07/day=03, year=2022/month=07/day=04
+ * The relative path "year=2022" returns all partitions, while the relative path
+ * "year=2022/month=07" returns only two partitions.
+ *
+ * @param relativePathPrefixes The prefixes to relative partition paths that must match
+ * @return null if not supported by File Index implementation
+ */
+ override protected def getMatchingPartitionPathsFromCatalog(relativePathPrefixes: List[String]): List[String] = {
+ // If listing from the catalog is disabled, return null
+ if (!configProperties.getBoolean(FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.key, FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.defaultValue())) {
+ null
+ } else {
+ // Retrieve all the partition paths from the catalog
Review Comment:
should we also take this route only when mdt is disabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #17775:
URL: https://github.com/apache/hudi/pull/17775#discussion_r2770788740
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -216,6 +216,16 @@ object DataSourceReadOptions {
" by carefully analyzing provided partition-column predicates and deducing corresponding partition-path prefix from " +
" them (if possible).")
+ val FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED: ConfigProperty[Boolean] =
+ ConfigProperty.key("hoodie.datasource.read.file.index.list.partitions.from.hms")
Review Comment:
Done. Renamed to `hoodie.datasource.read.file.index.list.partitions.from.catalog`
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -527,6 +527,12 @@ object HoodieFileIndex extends Logging {
properties.setProperty(DataSourceReadOptions.FILE_INDEX_LISTING_MODE_OVERRIDE.key, listingModeOverride)
}
+ var hmsListingEnabled = getConfigValue(options, sqlConf,
Review Comment:
Done. Renamed to `partitionListingViaCatalog`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3855802230
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
}, {
"hash" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11391",
"triggerID" : "3d85848f4881fe7a71cd149e5d4545865c293e14",
"triggerType" : "PUSH"
}, {
"hash" : "785469de431bfc4eb60f88ed72173817f51635b6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704",
"triggerID" : "785469de431bfc4eb60f88ed72173817f51635b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 785469de431bfc4eb60f88ed72173817f51635b6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11704)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3707779606
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b1819fdd2e9b999677b83e66c9f66ae89b242e2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17775:
URL: https://github.com/apache/hudi/pull/17775#issuecomment-3707835727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781",
"triggerID" : "2b1819fdd2e9b999677b83e66c9f66ae89b242e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b1819fdd2e9b999677b83e66c9f66ae89b242e2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10781)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): add config to list partition paths from HMS in file index [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason closed pull request #17775: feat(spark): add config to list partition paths from HMS in file index
URL: https://github.com/apache/hudi/pull/17775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org