You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2025/10/02 00:50:58 UTC
[PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
yihua opened a new pull request, #14031:
URL: https://github.com/apache/hudi/pull/14031
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402139687
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -1552,13 +1565,27 @@ private static FileSlice mergeCompactionPendingFileSlices(FileSlice lastSlice, F
*
* @param fileGroup File Group for which the file slice belongs to
* @param fileSlice File Slice which needs to be merged
- */
- private FileSlice fetchMergedFileSlice(HoodieFileGroup fileGroup, FileSlice fileSlice) {
+ * @param currentInstantTime Instant time of the current transaction
+ * @param includeBaseFileUnderInflightCompaction whether to include base file under inflight compaction
+ * when merging file slices
+ */
+ private FileSlice fetchMergedFileSlice(HoodieFileGroup fileGroup,
+ FileSlice fileSlice,
+ String currentInstantTime,
+ boolean includeBaseFileUnderInflightCompaction) {
// if the file-group is under construction, pick the latest before compaction instant time.
Option<Pair<String, CompactionOperation>> compactionOpWithInstant =
getPendingCompactionOperationWithInstant(fileGroup.getFileGroupId());
if (compactionOpWithInstant.isPresent()) {
String compactionInstantTime = compactionOpWithInstant.get().getKey();
+ // Only if includeBaseFileUnderInflightCompaction is true, i.e., the base file from inflight
+ // compaction should be considered in the latest merged file slice, we check the file slice.
Review Comment:
Synced offline and it is understood now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3364854295
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* dfb771fd1781b21adae727ac32fa4552bcca259c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047)
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 7de5bf44a2282bf19d81036ec5eb3f944a5d0407 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057)
* 28f94ed13d59cfe6ab3c77773168c69327e90965 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2396800440
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
+ Map<String, List<HoodieWriteStat>> partitionedWriteStats = allWriteStats.stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath));
int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
boolean shouldScanColStatsForTightBound = isShouldScanColStatsForTightBound(dataMetaClient);
- HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata = engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
- final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
- // Step 1: Collect Column Metadata for Each File part of current commit metadata
- List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
- .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
-
- if (shouldScanColStatsForTightBound) {
- checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
- // Collect Column Metadata for Each File part of active file system view of latest snapshot
- // Get all file names, including log files, in a set from the file slices
- Set<String> fileNames = getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.empty(), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
- // Fetch metadata table COLUMN_STATS partition records for above files
- List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
- .getRecordsByKeyPrefixes(
- HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
- MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
- // schema and properties are ignored in getInsertValue, so simply pass as null
- .map(record -> ((HoodieMetadataPayload)record.getData()).getColumnStatMetadata())
- .filter(Option::isPresent)
- .map(colStatsOpt -> colStatsOpt.get())
- .filter(stats -> fileNames.contains(stats.getFileName()))
- .map(HoodieColumnRangeMetadata::fromColumnStats).collectAsList();
- if (!partitionColumnMetadata.isEmpty()) {
- // incase of shouldScanColStatsForTightBound = true, we compute stats for the partition of interest for all files from getLatestFileSlice() excluding current commit here
- // already fileColumnMetadata contains stats for files from the current infliht commit. so, we are adding both together and sending it to collectAndProcessColumnMetadata
- fileColumnMetadata.addAll(partitionColumnMetadata);
- }
- }
+ List<StoragePathInfo> consolidatedPathInfos = new ArrayList<>();
+ //final Map<String, Stream<FileSlice>> consolidatedFileSliceMap;
+
+ // TODO(yihua): refactor usage of shouldScanColStatsForTightBound
+ // if (shouldScanColStatsForTightBound) {
+
+ // TODO(yihua): seems not possible to directly get latest merged file slices without constructing another FSV
+ tableMetadata.getAllFilesInPartitions(
Review Comment:
`tableMetadata` or `HoodieTableMetadata` does not expose API to get latest file slices; only the file system view provides such logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2398818053
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
+ Map<String, List<HoodieWriteStat>> partitionedWriteStats = allWriteStats.stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath));
int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
boolean shouldScanColStatsForTightBound = isShouldScanColStatsForTightBound(dataMetaClient);
- HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata = engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
- final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
- // Step 1: Collect Column Metadata for Each File part of current commit metadata
- List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
- .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
-
- if (shouldScanColStatsForTightBound) {
- checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
- // Collect Column Metadata for Each File part of active file system view of latest snapshot
- // Get all file names, including log files, in a set from the file slices
- Set<String> fileNames = getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.empty(), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
- // Fetch metadata table COLUMN_STATS partition records for above files
- List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
- .getRecordsByKeyPrefixes(
- HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
- MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
- // schema and properties are ignored in getInsertValue, so simply pass as null
- .map(record -> ((HoodieMetadataPayload)record.getData()).getColumnStatMetadata())
- .filter(Option::isPresent)
- .map(colStatsOpt -> colStatsOpt.get())
- .filter(stats -> fileNames.contains(stats.getFileName()))
- .map(HoodieColumnRangeMetadata::fromColumnStats).collectAsList();
- if (!partitionColumnMetadata.isEmpty()) {
- // incase of shouldScanColStatsForTightBound = true, we compute stats for the partition of interest for all files from getLatestFileSlice() excluding current commit here
- // already fileColumnMetadata contains stats for files from the current infliht commit. so, we are adding both together and sending it to collectAndProcessColumnMetadata
- fileColumnMetadata.addAll(partitionColumnMetadata);
- }
- }
+ List<StoragePathInfo> consolidatedPathInfos = new ArrayList<>();
+ //final Map<String, Stream<FileSlice>> consolidatedFileSliceMap;
+
+ // TODO(yihua): refactor usage of shouldScanColStatsForTightBound
+ // if (shouldScanColStatsForTightBound) {
+
+ // TODO(yihua): seems not possible to directly get latest merged file slices without constructing another FSV
+ tableMetadata.getAllFilesInPartitions(
Review Comment:
As discussed, we're going to use the same file system view as the regular data table writer, i.e., remote file system view through the timeline server, so that the listing can be parallelized across affected partition, part of the commit metadata, to reduce the memory pressure on the driver compared to the approach of collecting and consolidating all files of all affected partitions on the driver.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363847406
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4dd726d631f4fc3749017f264d4b45c4c73795a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046)
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* dfb771fd1781b21adae727ac32fa4552bcca259c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3362957897
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43e107dd015030d5f10317b8180fc99d031561a4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3365088812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 7de5bf44a2282bf19d81036ec5eb3f944a5d0407 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057)
* 28f94ed13d59cfe6ab3c77773168c69327e90965 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402353914
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -1043,22 +1043,35 @@ public final Map<String, Stream<FileSlice>> getAllLatestFileSlicesBeforeOrOn(Str
@Override
public final Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionStr, String maxInstantTime) {
+ return getLatestMergedFileSliceBeforeOrOnInternal(partitionStr, maxInstantTime, maxInstantTime, false);
+ }
+
+ @Override
+ public final Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOnIncludingInflight(String partitionStr, String maxInstantTime, String currentInstantTime) {
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402702145
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/view/TestHoodieTableFileSystemView.java:
##########
@@ -2506,6 +2506,184 @@ private HoodieWriteStat getHoodieWriteStat(String partitionPath, String fileId,
return writeStat;
}
+ @Test
+ public void testGetLatestMergedFileSlicesBeforeOrOnIncludingInflight() throws IOException {
+ String partitionPath = "2023/01/01";
+ String fileId = UUID.randomUUID().toString();
+ String instantTime1 = "001";
+ String instantTime2 = "002";
+ String compactionInstant = "003";
+ String instantTime4 = "004";
+ String instantTime5 = "005";
+
+ new File(basePath + "/" + partitionPath).mkdirs();
+ HoodieActiveTimeline commitTimeline = metaClient.getActiveTimeline();
+
+ // Create base file at instant1
+ String baseFileName = FSUtils.makeBaseFileName(instantTime1, TEST_WRITE_TOKEN, fileId, BASE_FILE_EXTENSION);
+ new File(basePath + "/" + partitionPath + "/" + baseFileName).createNewFile();
+ HoodieInstant instant1 = INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, instantTime1);
+ saveAsComplete(commitTimeline, instant1, new HoodieCommitMetadata());
+
+ // Add log files at instant2
+ HoodieInstant instant2 = INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, instantTime2);
+ String logFileName2 = FSUtils.makeLogFileName(fileId, HoodieLogFile.DELTA_EXTENSION, instantTime2, 1, TEST_WRITE_TOKEN);
+ new File(basePath + "/" + partitionPath + "/" + logFileName2).createNewFile();
+ saveAsComplete(commitTimeline, instant2, new HoodieCommitMetadata());
+
+ refreshFsView();
+
+ List<FileSlice> actual = fsView.getLatestMergedFileSlicesBeforeOrOn(partitionPath, instantTime2)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ FileSlice fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(1, fileSlice.getLogFiles().count());
+ assertEquals(logFileName2, fileSlice.getLogFiles().collect(Collectors.toList()).get(0).getFileName());
+
+ actual = fsView.getLatestMergedFileSlicesBeforeOrOnIncludingInflight(partitionPath, instantTime2, instantTime2)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(1, fileSlice.getLogFiles().count());
+ assertEquals(logFileName2, fileSlice.getLogFiles().collect(Collectors.toList()).get(0).getFileName());
+
+ // Schedule compaction at instant3
+ List<FileSlice> fileSlices = rtView.getLatestFileSlices(partitionPath).collect(Collectors.toList());
+ List<Pair<String, FileSlice>> partitionFileSlicesPairs = new ArrayList<>();
+ partitionFileSlicesPairs.add(Pair.of(partitionPath, fileSlices.get(0)));
+ HoodieCompactionPlan compactionPlan = CompactionUtils.buildFromFileSlices(
+ partitionFileSlicesPairs, Option.empty(), Option.empty());
+ HoodieInstant compactionRequestedInstant = INSTANT_GENERATOR.createNewInstant(State.REQUESTED, HoodieTimeline.COMPACTION_ACTION, compactionInstant);
+ commitTimeline.createNewInstant(compactionRequestedInstant);
+ commitTimeline.saveToCompactionRequested(compactionRequestedInstant, compactionPlan);
+ commitTimeline.transitionRequestedToInflight(compactionRequestedInstant, Option.empty());
+ String baseFileName2 = FSUtils.makeBaseFileName(compactionInstant, TEST_WRITE_TOKEN, fileId, BASE_FILE_EXTENSION);
+ new File(basePath + "/" + partitionPath + "/" + baseFileName2).createNewFile();
+
+ // Add log files at instant4
+ HoodieInstant instant4 = INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, instantTime4);
+ String logFileName4 = FSUtils.makeLogFileName(fileId, HoodieLogFile.DELTA_EXTENSION, instantTime4, 1, TEST_WRITE_TOKEN);
+ new File(basePath + "/" + partitionPath + "/" + logFileName4).createNewFile();
+ saveAsComplete(commitTimeline, instant4, new HoodieCommitMetadata());
+
+ // Add log files at instant5
+ String logFileName5 = FSUtils.makeLogFileName(fileId, HoodieLogFile.DELTA_EXTENSION, instantTime5, 1, TEST_WRITE_TOKEN);
+ new File(basePath + "/" + partitionPath + "/" + logFileName5).createNewFile();
+ commitTimeline.createNewInstant(INSTANT_GENERATOR.createNewInstant(State.REQUESTED, HoodieTimeline.DELTA_COMMIT_ACTION, instantTime5));
+ commitTimeline.createNewInstant(INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, instantTime5));
+
+ refreshFsView();
+ actual = fsView.getLatestMergedFileSlicesBeforeOrOn(partitionPath, instantTime5)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(2, fileSlice.getLogFiles().count());
+ assertEquals(
+ Arrays.asList(logFileName2, logFileName4),
+ fileSlice.getLogFiles().map(HoodieLogFile::getFileName).sorted().collect(Collectors.toList()));
+
+ actual = fsView.getLatestMergedFileSlicesBeforeOrOnIncludingInflight(partitionPath, instantTime5, instantTime5)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(3, fileSlice.getLogFiles().count());
+ assertEquals(
+ Arrays.asList(logFileName2, logFileName4, logFileName5),
+ fileSlice.getLogFiles().map(HoodieLogFile::getFileName).sorted().collect(Collectors.toList()));
+
+ actual = fsView.getLatestMergedFileSlicesBeforeOrOnIncludingInflight(partitionPath, instantTime5, compactionInstant)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName2, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(2, fileSlice.getLogFiles().count());
+ assertEquals(
+ Arrays.asList(logFileName4, logFileName5),
+ fileSlice.getLogFiles().map(HoodieLogFile::getFileName).sorted().collect(Collectors.toList()));
+
+ commitTimeline.saveAsComplete(INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.COMPACTION_ACTION, compactionInstant), Option.empty());
Review Comment:
Fixed alongside removing a failing test that is redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2396872799
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
+ Map<String, List<HoodieWriteStat>> partitionedWriteStats = allWriteStats.stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath));
int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
boolean shouldScanColStatsForTightBound = isShouldScanColStatsForTightBound(dataMetaClient);
- HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata = engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
- final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
- // Step 1: Collect Column Metadata for Each File part of current commit metadata
- List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
- .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
-
- if (shouldScanColStatsForTightBound) {
- checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
- // Collect Column Metadata for Each File part of active file system view of latest snapshot
- // Get all file names, including log files, in a set from the file slices
- Set<String> fileNames = getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.empty(), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
- // Fetch metadata table COLUMN_STATS partition records for above files
- List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
- .getRecordsByKeyPrefixes(
- HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
- MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
- // schema and properties are ignored in getInsertValue, so simply pass as null
- .map(record -> ((HoodieMetadataPayload)record.getData()).getColumnStatMetadata())
- .filter(Option::isPresent)
- .map(colStatsOpt -> colStatsOpt.get())
- .filter(stats -> fileNames.contains(stats.getFileName()))
- .map(HoodieColumnRangeMetadata::fromColumnStats).collectAsList();
- if (!partitionColumnMetadata.isEmpty()) {
- // incase of shouldScanColStatsForTightBound = true, we compute stats for the partition of interest for all files from getLatestFileSlice() excluding current commit here
- // already fileColumnMetadata contains stats for files from the current infliht commit. so, we are adding both together and sending it to collectAndProcessColumnMetadata
- fileColumnMetadata.addAll(partitionColumnMetadata);
- }
- }
+ List<StoragePathInfo> consolidatedPathInfos = new ArrayList<>();
+ //final Map<String, Stream<FileSlice>> consolidatedFileSliceMap;
+
+ // TODO(yihua): refactor usage of shouldScanColStatsForTightBound
+ // if (shouldScanColStatsForTightBound) {
+
+ // TODO(yihua): seems not possible to directly get latest merged file slices without constructing another FSV
+ tableMetadata.getAllFilesInPartitions(
Review Comment:
Wrt to the latest list of latest file slices, I guess you're referring to `getLazyLatestMergedPartitionFileSliceList()`. It currently is not stored as a class member variable and only reused when initializing MDT partitions from the file system. For subsequent commits after MDT partitions are initialized, the method may not be called, and the code path of updates to partition stats index do not have the file slice list computed (i.e., `getLazyLatestMergedPartitionFileSliceList()`), so it has to be fetched somewhere, either in `HoodieTableMetadataUtil#convertMetadataToPartitionStatRecords` or the caller in `HoodieBackedTableMetadataWriter `.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
+ Map<String, List<HoodieWriteStat>> partitionedWriteStats = allWriteStats.stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath));
int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
boolean shouldScanColStatsForTightBound = isShouldScanColStatsForTightBound(dataMetaClient);
- HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata = engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
- final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
- // Step 1: Collect Column Metadata for Each File part of current commit metadata
- List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
- .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
-
- if (shouldScanColStatsForTightBound) {
- checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
- // Collect Column Metadata for Each File part of active file system view of latest snapshot
- // Get all file names, including log files, in a set from the file slices
- Set<String> fileNames = getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.empty(), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
- // Fetch metadata table COLUMN_STATS partition records for above files
- List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
- .getRecordsByKeyPrefixes(
- HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
- MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
- // schema and properties are ignored in getInsertValue, so simply pass as null
- .map(record -> ((HoodieMetadataPayload)record.getData()).getColumnStatMetadata())
- .filter(Option::isPresent)
- .map(colStatsOpt -> colStatsOpt.get())
- .filter(stats -> fileNames.contains(stats.getFileName()))
- .map(HoodieColumnRangeMetadata::fromColumnStats).collectAsList();
- if (!partitionColumnMetadata.isEmpty()) {
- // incase of shouldScanColStatsForTightBound = true, we compute stats for the partition of interest for all files from getLatestFileSlice() excluding current commit here
- // already fileColumnMetadata contains stats for files from the current infliht commit. so, we are adding both together and sending it to collectAndProcessColumnMetadata
- fileColumnMetadata.addAll(partitionColumnMetadata);
- }
- }
+ List<StoragePathInfo> consolidatedPathInfos = new ArrayList<>();
+ //final Map<String, Stream<FileSlice>> consolidatedFileSliceMap;
+
+ // TODO(yihua): refactor usage of shouldScanColStatsForTightBound
+ // if (shouldScanColStatsForTightBound) {
+
+ // TODO(yihua): seems not possible to directly get latest merged file slices without constructing another FSV
+ tableMetadata.getAllFilesInPartitions(
Review Comment:
Wrt to the latest list of latest file slices, I guess you're referring to `getLazyLatestMergedPartitionFileSliceList()`. It currently is not stored as a class member variable and only reused when initializing MDT partitions from the file system. For subsequent commits after MDT partitions are initialized, the method may not be called, and the code path of updates to partition stats index do not have the file slice list computed (i.e., `getLazyLatestMergedPartitionFileSliceList()`), so it has to be fetched somewhere, either in `HoodieTableMetadataUtil#convertMetadataToPartitionStatRecords` or the caller in `HoodieBackedTableMetadataWriter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402105973
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/hadoop/fs/HoodieWrapperFileSystem.java:
##########
@@ -374,6 +374,7 @@ public boolean delete(Path f, boolean recursive) throws IOException {
@Override
public FileStatus[] listStatus(Path f) throws IOException {
+ System.out.println("!!!!!!! listStatus for " + f);
Review Comment:
Cleaned up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363470104
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43e107dd015030d5f10317b8180fc99d031561a4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035)
* 1566c8ee1e7fa736b3eac4e6228d742daba5a49e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3364525034
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* dfb771fd1781b21adae727ac32fa4552bcca259c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047)
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 7de5bf44a2282bf19d81036ec5eb3f944a5d0407 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363852704
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4dd726d631f4fc3749017f264d4b45c4c73795a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046)
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* dfb771fd1781b21adae727ac32fa4552bcca259c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3366064350
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
}, {
"hash" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"triggerType" : "PUSH"
}, {
"hash" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066",
"triggerID" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 28f94ed13d59cfe6ab3c77773168c69327e90965 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063)
* 85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2 UNKNOWN
* 288355f230fdcc0b1535d87590a21dbebd50feb6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3366171188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
}, {
"hash" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"triggerType" : "PUSH"
}, {
"hash" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066",
"triggerID" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"triggerType" : "PUSH"
}, {
"hash" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9067",
"triggerID" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2 UNKNOWN
* 288355f230fdcc0b1535d87590a21dbebd50feb6 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066)
* 8d0033e1c0448317dfd66724e7322d5d693fdad0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9067)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3366148821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
}, {
"hash" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"triggerType" : "PUSH"
}, {
"hash" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066",
"triggerID" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"triggerType" : "PUSH"
}, {
"hash" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 28f94ed13d59cfe6ab3c77773168c69327e90965 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063)
* 85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2 UNKNOWN
* 288355f230fdcc0b1535d87590a21dbebd50feb6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066)
* 8d0033e1c0448317dfd66724e7322d5d693fdad0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363946193
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* dfb771fd1781b21adae727ac32fa4552bcca259c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records and fix correctness issue for partition stats index [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #14031:
URL: https://github.com/apache/hudi/pull/14031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3362650008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ed4ec66734b182c328495851b8d9f6d2cb1b70f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031)
* 894ea561d160481482bffbb0f1edccbfb555bcfc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402107157
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/storage/hadoop/HoodieHadoopStorage.java:
##########
@@ -194,6 +200,9 @@ public List<StoragePathInfo> listFiles(StoragePath path) throws IOException {
@Override
public List<StoragePathInfo> listDirectEntries(List<StoragePath> pathList) throws IOException {
+ if (pathList.stream().anyMatch(path -> path.getParent().getName().equals("test_table"))) {
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2398802695
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3361179133
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e83ac0429e7d295f64f69eb0326cbf68a87d732 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007)
* 47806303eeed26d7036970ce52c2c56572a47a64 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3362787348
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 894ea561d160481482bffbb0f1edccbfb555bcfc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034)
* 43e107dd015030d5f10317b8180fc99d031561a4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3358643379
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e83ac0429e7d295f64f69eb0326cbf68a87d732 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363815587
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a73bca8f8d48d18c65ba000bb8e09540767fbf83 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040)
* c4dd726d631f4fc3749017f264d4b45c4c73795a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046)
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363831068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a73bca8f8d48d18c65ba000bb8e09540767fbf83 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040)
* c4dd726d631f4fc3749017f264d4b45c4c73795a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046)
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* dfb771fd1781b21adae727ac32fa4552bcca259c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3365297668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 28f94ed13d59cfe6ab3c77773168c69327e90965 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3365838815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
}, {
"hash" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 28f94ed13d59cfe6ab3c77773168c69327e90965 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063)
* 85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2396791463
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
+ Map<String, List<HoodieWriteStat>> partitionedWriteStats = allWriteStats.stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath));
int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
boolean shouldScanColStatsForTightBound = isShouldScanColStatsForTightBound(dataMetaClient);
- HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata = engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
- final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
- // Step 1: Collect Column Metadata for Each File part of current commit metadata
- List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
- .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
-
- if (shouldScanColStatsForTightBound) {
- checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
- // Collect Column Metadata for Each File part of active file system view of latest snapshot
- // Get all file names, including log files, in a set from the file slices
- Set<String> fileNames = getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.empty(), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
- // Fetch metadata table COLUMN_STATS partition records for above files
- List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
- .getRecordsByKeyPrefixes(
- HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
- MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
- // schema and properties are ignored in getInsertValue, so simply pass as null
- .map(record -> ((HoodieMetadataPayload)record.getData()).getColumnStatMetadata())
- .filter(Option::isPresent)
- .map(colStatsOpt -> colStatsOpt.get())
- .filter(stats -> fileNames.contains(stats.getFileName()))
- .map(HoodieColumnRangeMetadata::fromColumnStats).collectAsList();
- if (!partitionColumnMetadata.isEmpty()) {
- // incase of shouldScanColStatsForTightBound = true, we compute stats for the partition of interest for all files from getLatestFileSlice() excluding current commit here
- // already fileColumnMetadata contains stats for files from the current infliht commit. so, we are adding both together and sending it to collectAndProcessColumnMetadata
- fileColumnMetadata.addAll(partitionColumnMetadata);
- }
- }
+ List<StoragePathInfo> consolidatedPathInfos = new ArrayList<>();
+ //final Map<String, Stream<FileSlice>> consolidatedFileSliceMap;
+
+ // TODO(yihua): refactor usage of shouldScanColStatsForTightBound
+ // if (shouldScanColStatsForTightBound) {
+
+ // TODO(yihua): seems not possible to directly get latest merged file slices without constructing another FSV
+ tableMetadata.getAllFilesInPartitions(
+ partitionedWriteStats.keySet().stream()
+ .map(partition -> new StoragePath(dataMetaClient.getBasePath(), partition).toString()).collect(toList()))
+ .values().forEach(consolidatedPathInfos::addAll);
+ allWriteStats.forEach(
+ stat -> consolidatedPathInfos.add(
+ new StoragePathInfo(new StoragePath(dataMetaClient.getBasePath(), stat.getPath()), stat.getFileSizeInBytes(), false, (short) 0, 0, 0)));
+ // TODO(yihua): check config constructed here
+ // TODO(yihua): explore this approach vs RDD based data partitions and fetching files in partition from table metadata
+ // and construct FSV per partition (this alleviates driver memory at the cost of increasing FS calls to MDT?)
+ SpillableMapBasedFileSystemView fileSystemView = new SpillableMapBasedFileSystemView(
+ tableMetadata, dataMetaClient, dataMetaClient.getActiveTimeline(),
+ consolidatedPathInfos, FileSystemViewStorageConfig.newBuilder().fromProperties(metadataConfig.getProps()).build(),
+ HoodieCommonConfig.newBuilder().fromProperties(metadataConfig.getProps()).build());
+ Map<String, List<FileSlice>> consolidatedFileSliceMap = partitionedWriteStats.keySet().stream()
+ .collect(Collectors.toMap(
+ e -> e,
+ partition -> fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partition, instantTime).collect(toList())));
+
+ HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata =
+ engineContext.parallelize(partitionedWriteStats.entrySet().stream().collect(Collectors.toList()), parallelism).mapToPair(entry -> {
+ List<HoodieWriteStat> partitionedWriteStat = entry.getValue();
+ final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
+ // Step 1: Collect Column Metadata for Each File part of current commit metadata
+ List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
+ .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
+ Set<String> filesWithColumnStats = partitionedWriteStat.stream().map(stat -> stat.getPath()).collect(Collectors.toSet());
+
+ if (shouldScanColStatsForTightBound) {
+ checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
+ // Collect Column Metadata for Each File part of active file system view of latest snapshot
+ // Get all file names, including log files, in a set from the file slices
+ Set<String> fileNames = consolidatedFileSliceMap.get(partitionName).stream()
+ .flatMap(fileSlice -> Stream.concat(
+ Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
+ fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
+ .filter(e -> Objects.nonNull(e) && !filesWithColumnStats.contains(e))
+ .collect(Collectors.toSet());
+ // Fetch metadata table COLUMN_STATS partition records for above files
+ List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
+ .getRecordsByKeyPrefixes(
+ HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
Review Comment:
I'm actually worried about the size of exact lookup keys given the combination of columns and files (e.g., 32 columns into 10K files at a large scale would result in 320K lookup keys). Since this is not a necessary change for reducing the list calls, I haven't put it in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3361275279
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47806303eeed26d7036970ce52c2c56572a47a64 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3358645418
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e83ac0429e7d295f64f69eb0326cbf68a87d732 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363473956
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43e107dd015030d5f10317b8180fc99d031561a4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035)
* 1566c8ee1e7fa736b3eac4e6228d742daba5a49e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3366530761
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
}, {
"hash" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"triggerType" : "PUSH"
}, {
"hash" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066",
"triggerID" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"triggerType" : "PUSH"
}, {
"hash" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9067",
"triggerID" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"triggerType" : "PUSH"
}, {
"hash" : "9be92869aad74c787c68bca353982d00048dff7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9be92869aad74c787c68bca353982d00048dff7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2 UNKNOWN
* 8d0033e1c0448317dfd66724e7322d5d693fdad0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9067)
* 9be92869aad74c787c68bca353982d00048dff7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3365859983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
}, {
"hash" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"triggerType" : "PUSH"
}, {
"hash" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 28f94ed13d59cfe6ab3c77773168c69327e90965 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063)
* 85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2 UNKNOWN
* 288355f230fdcc0b1535d87590a21dbebd50feb6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402100607
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/TableFileSystemView.java:
##########
@@ -166,6 +166,24 @@ Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath, String max
*/
Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime);
+ /**
+ * Stream all "merged" file slices before on an instant time including files under inflight instants.
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3366221127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
}, {
"hash" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"triggerType" : "PUSH"
}, {
"hash" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066",
"triggerID" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"triggerType" : "PUSH"
}, {
"hash" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9067",
"triggerID" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2 UNKNOWN
* 8d0033e1c0448317dfd66724e7322d5d693fdad0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9067)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402451592
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/view/TestHoodieTableFileSystemView.java:
##########
@@ -2506,6 +2506,184 @@ private HoodieWriteStat getHoodieWriteStat(String partitionPath, String fileId,
return writeStat;
}
+ @Test
+ public void testGetLatestMergedFileSlicesBeforeOrOnIncludingInflight() throws IOException {
+ String partitionPath = "2023/01/01";
+ String fileId = UUID.randomUUID().toString();
+ String instantTime1 = "001";
+ String instantTime2 = "002";
+ String compactionInstant = "003";
+ String instantTime4 = "004";
+ String instantTime5 = "005";
+
+ new File(basePath + "/" + partitionPath).mkdirs();
+ HoodieActiveTimeline commitTimeline = metaClient.getActiveTimeline();
+
+ // Create base file at instant1
+ String baseFileName = FSUtils.makeBaseFileName(instantTime1, TEST_WRITE_TOKEN, fileId, BASE_FILE_EXTENSION);
+ new File(basePath + "/" + partitionPath + "/" + baseFileName).createNewFile();
+ HoodieInstant instant1 = INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, instantTime1);
+ saveAsComplete(commitTimeline, instant1, new HoodieCommitMetadata());
+
+ // Add log files at instant2
+ HoodieInstant instant2 = INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, instantTime2);
+ String logFileName2 = FSUtils.makeLogFileName(fileId, HoodieLogFile.DELTA_EXTENSION, instantTime2, 1, TEST_WRITE_TOKEN);
+ new File(basePath + "/" + partitionPath + "/" + logFileName2).createNewFile();
+ saveAsComplete(commitTimeline, instant2, new HoodieCommitMetadata());
+
+ refreshFsView();
+
+ List<FileSlice> actual = fsView.getLatestMergedFileSlicesBeforeOrOn(partitionPath, instantTime2)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ FileSlice fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(1, fileSlice.getLogFiles().count());
+ assertEquals(logFileName2, fileSlice.getLogFiles().collect(Collectors.toList()).get(0).getFileName());
+
+ actual = fsView.getLatestMergedFileSlicesBeforeOrOnIncludingInflight(partitionPath, instantTime2, instantTime2)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(1, fileSlice.getLogFiles().count());
+ assertEquals(logFileName2, fileSlice.getLogFiles().collect(Collectors.toList()).get(0).getFileName());
+
+ // Schedule compaction at instant3
+ List<FileSlice> fileSlices = rtView.getLatestFileSlices(partitionPath).collect(Collectors.toList());
+ List<Pair<String, FileSlice>> partitionFileSlicesPairs = new ArrayList<>();
+ partitionFileSlicesPairs.add(Pair.of(partitionPath, fileSlices.get(0)));
+ HoodieCompactionPlan compactionPlan = CompactionUtils.buildFromFileSlices(
+ partitionFileSlicesPairs, Option.empty(), Option.empty());
+ HoodieInstant compactionRequestedInstant = INSTANT_GENERATOR.createNewInstant(State.REQUESTED, HoodieTimeline.COMPACTION_ACTION, compactionInstant);
+ commitTimeline.createNewInstant(compactionRequestedInstant);
+ commitTimeline.saveToCompactionRequested(compactionRequestedInstant, compactionPlan);
+ commitTimeline.transitionRequestedToInflight(compactionRequestedInstant, Option.empty());
+ String baseFileName2 = FSUtils.makeBaseFileName(compactionInstant, TEST_WRITE_TOKEN, fileId, BASE_FILE_EXTENSION);
+ new File(basePath + "/" + partitionPath + "/" + baseFileName2).createNewFile();
+
+ // Add log files at instant4
+ HoodieInstant instant4 = INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, instantTime4);
+ String logFileName4 = FSUtils.makeLogFileName(fileId, HoodieLogFile.DELTA_EXTENSION, instantTime4, 1, TEST_WRITE_TOKEN);
+ new File(basePath + "/" + partitionPath + "/" + logFileName4).createNewFile();
+ saveAsComplete(commitTimeline, instant4, new HoodieCommitMetadata());
+
+ // Add log files at instant5
+ String logFileName5 = FSUtils.makeLogFileName(fileId, HoodieLogFile.DELTA_EXTENSION, instantTime5, 1, TEST_WRITE_TOKEN);
+ new File(basePath + "/" + partitionPath + "/" + logFileName5).createNewFile();
+ commitTimeline.createNewInstant(INSTANT_GENERATOR.createNewInstant(State.REQUESTED, HoodieTimeline.DELTA_COMMIT_ACTION, instantTime5));
+ commitTimeline.createNewInstant(INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, instantTime5));
+
+ refreshFsView();
+ actual = fsView.getLatestMergedFileSlicesBeforeOrOn(partitionPath, instantTime5)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(2, fileSlice.getLogFiles().count());
+ assertEquals(
+ Arrays.asList(logFileName2, logFileName4),
+ fileSlice.getLogFiles().map(HoodieLogFile::getFileName).sorted().collect(Collectors.toList()));
+
+ actual = fsView.getLatestMergedFileSlicesBeforeOrOnIncludingInflight(partitionPath, instantTime5, instantTime5)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(3, fileSlice.getLogFiles().count());
+ assertEquals(
+ Arrays.asList(logFileName2, logFileName4, logFileName5),
+ fileSlice.getLogFiles().map(HoodieLogFile::getFileName).sorted().collect(Collectors.toList()));
+
+ actual = fsView.getLatestMergedFileSlicesBeforeOrOnIncludingInflight(partitionPath, instantTime5, compactionInstant)
+ .collect(Collectors.toList());
+ assertEquals(1, actual.size());
+ fileSlice = actual.get(0);
+ assertTrue(fileSlice.getBaseFile().isPresent());
+ assertEquals(baseFileName2, fileSlice.getBaseFile().get().getFileName());
+ assertEquals(2, fileSlice.getLogFiles().count());
+ assertEquals(
+ Arrays.asList(logFileName4, logFileName5),
+ fileSlice.getLogFiles().map(HoodieLogFile::getFileName).sorted().collect(Collectors.toList()));
+
+ commitTimeline.saveAsComplete(INSTANT_GENERATOR.createNewInstant(State.INFLIGHT, HoodieTimeline.COMPACTION_ACTION, compactionInstant), Option.empty());
Review Comment:
add java docs that we are completing the compaction here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records for partition stats index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3366561585
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
}, {
"hash" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"triggerType" : "PUSH"
}, {
"hash" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066",
"triggerID" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"triggerType" : "PUSH"
}, {
"hash" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9067",
"triggerID" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"triggerType" : "PUSH"
}, {
"hash" : "9be92869aad74c787c68bca353982d00048dff7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9072",
"triggerID" : "9be92869aad74c787c68bca353982d00048dff7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2 UNKNOWN
* 8d0033e1c0448317dfd66724e7322d5d693fdad0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9067)
* 9be92869aad74c787c68bca353982d00048dff7e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records for partition stats index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3366831871
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9063",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
}, {
"hash" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2",
"triggerType" : "PUSH"
}, {
"hash" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9066",
"triggerID" : "288355f230fdcc0b1535d87590a21dbebd50feb6",
"triggerType" : "PUSH"
}, {
"hash" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9067",
"triggerID" : "8d0033e1c0448317dfd66724e7322d5d693fdad0",
"triggerType" : "PUSH"
}, {
"hash" : "9be92869aad74c787c68bca353982d00048dff7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9072",
"triggerID" : "9be92869aad74c787c68bca353982d00048dff7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 85d84a4ac72fd3ecd17b172c42e6bc28bfd20bc2 UNKNOWN
* 9be92869aad74c787c68bca353982d00048dff7e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2396635104
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
Review Comment:
`shouldScanColStatsForTightBound` needs to be cleaned up.
it was lingering from previous attempt at partition stats.
currently, everything is tight bound only. so, we can clean it up.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
+ Map<String, List<HoodieWriteStat>> partitionedWriteStats = allWriteStats.stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath));
int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
boolean shouldScanColStatsForTightBound = isShouldScanColStatsForTightBound(dataMetaClient);
- HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata = engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
- final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
- // Step 1: Collect Column Metadata for Each File part of current commit metadata
- List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
- .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
-
- if (shouldScanColStatsForTightBound) {
- checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
- // Collect Column Metadata for Each File part of active file system view of latest snapshot
- // Get all file names, including log files, in a set from the file slices
- Set<String> fileNames = getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.empty(), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
- // Fetch metadata table COLUMN_STATS partition records for above files
- List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
- .getRecordsByKeyPrefixes(
- HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
- MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
- // schema and properties are ignored in getInsertValue, so simply pass as null
- .map(record -> ((HoodieMetadataPayload)record.getData()).getColumnStatMetadata())
- .filter(Option::isPresent)
- .map(colStatsOpt -> colStatsOpt.get())
- .filter(stats -> fileNames.contains(stats.getFileName()))
- .map(HoodieColumnRangeMetadata::fromColumnStats).collectAsList();
- if (!partitionColumnMetadata.isEmpty()) {
- // incase of shouldScanColStatsForTightBound = true, we compute stats for the partition of interest for all files from getLatestFileSlice() excluding current commit here
- // already fileColumnMetadata contains stats for files from the current infliht commit. so, we are adding both together and sending it to collectAndProcessColumnMetadata
- fileColumnMetadata.addAll(partitionColumnMetadata);
- }
- }
+ List<StoragePathInfo> consolidatedPathInfos = new ArrayList<>();
+ //final Map<String, Stream<FileSlice>> consolidatedFileSliceMap;
+
+ // TODO(yihua): refactor usage of shouldScanColStatsForTightBound
+ // if (shouldScanColStatsForTightBound) {
+
+ // TODO(yihua): seems not possible to directly get latest merged file slices without constructing another FSV
+ tableMetadata.getAllFilesInPartitions(
+ partitionedWriteStats.keySet().stream()
+ .map(partition -> new StoragePath(dataMetaClient.getBasePath(), partition).toString()).collect(toList()))
+ .values().forEach(consolidatedPathInfos::addAll);
+ allWriteStats.forEach(
+ stat -> consolidatedPathInfos.add(
+ new StoragePathInfo(new StoragePath(dataMetaClient.getBasePath(), stat.getPath()), stat.getFileSizeInBytes(), false, (short) 0, 0, 0)));
+ // TODO(yihua): check config constructed here
+ // TODO(yihua): explore this approach vs RDD based data partitions and fetching files in partition from table metadata
+ // and construct FSV per partition (this alleviates driver memory at the cost of increasing FS calls to MDT?)
+ SpillableMapBasedFileSystemView fileSystemView = new SpillableMapBasedFileSystemView(
+ tableMetadata, dataMetaClient, dataMetaClient.getActiveTimeline(),
+ consolidatedPathInfos, FileSystemViewStorageConfig.newBuilder().fromProperties(metadataConfig.getProps()).build(),
+ HoodieCommonConfig.newBuilder().fromProperties(metadataConfig.getProps()).build());
+ Map<String, List<FileSlice>> consolidatedFileSliceMap = partitionedWriteStats.keySet().stream()
+ .collect(Collectors.toMap(
+ e -> e,
+ partition -> fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partition, instantTime).collect(toList())));
+
+ HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata =
+ engineContext.parallelize(partitionedWriteStats.entrySet().stream().collect(Collectors.toList()), parallelism).mapToPair(entry -> {
+ List<HoodieWriteStat> partitionedWriteStat = entry.getValue();
+ final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
+ // Step 1: Collect Column Metadata for Each File part of current commit metadata
+ List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
+ .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
+ Set<String> filesWithColumnStats = partitionedWriteStat.stream().map(stat -> stat.getPath()).collect(Collectors.toSet());
+
+ if (shouldScanColStatsForTightBound) {
+ checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
+ // Collect Column Metadata for Each File part of active file system view of latest snapshot
+ // Get all file names, including log files, in a set from the file slices
+ Set<String> fileNames = consolidatedFileSliceMap.get(partitionName).stream()
+ .flatMap(fileSlice -> Stream.concat(
+ Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
+ fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
+ .filter(e -> Objects.nonNull(e) && !filesWithColumnStats.contains(e))
+ .collect(Collectors.toSet());
+ // Fetch metadata table COLUMN_STATS partition records for above files
+ List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
+ .getRecordsByKeyPrefixes(
+ HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
Review Comment:
as we discussed yesterday, we already have list of file names we are interested in. So, we should include those as well. currently, we are only looking up for col name and partition name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3361698123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47806303eeed26d7036970ce52c2c56572a47a64 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027)
* 7ed4ec66734b182c328495851b8d9f6d2cb1b70f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363573314
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1566c8ee1e7fa736b3eac4e6228d742daba5a49e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039)
* a73bca8f8d48d18c65ba000bb8e09540767fbf83 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363544066
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43e107dd015030d5f10317b8180fc99d031561a4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035)
* 1566c8ee1e7fa736b3eac4e6228d742daba5a49e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039)
* a73bca8f8d48d18c65ba000bb8e09540767fbf83 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363773419
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a73bca8f8d48d18c65ba000bb8e09540767fbf83 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040)
* c4dd726d631f4fc3749017f264d4b45c4c73795a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2396739994
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
+ Map<String, List<HoodieWriteStat>> partitionedWriteStats = allWriteStats.stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath));
int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
boolean shouldScanColStatsForTightBound = isShouldScanColStatsForTightBound(dataMetaClient);
- HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata = engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
- final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
- // Step 1: Collect Column Metadata for Each File part of current commit metadata
- List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
- .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
-
- if (shouldScanColStatsForTightBound) {
- checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
- // Collect Column Metadata for Each File part of active file system view of latest snapshot
- // Get all file names, including log files, in a set from the file slices
- Set<String> fileNames = getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.empty(), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
- // Fetch metadata table COLUMN_STATS partition records for above files
- List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
- .getRecordsByKeyPrefixes(
- HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
- MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
- // schema and properties are ignored in getInsertValue, so simply pass as null
- .map(record -> ((HoodieMetadataPayload)record.getData()).getColumnStatMetadata())
- .filter(Option::isPresent)
- .map(colStatsOpt -> colStatsOpt.get())
- .filter(stats -> fileNames.contains(stats.getFileName()))
- .map(HoodieColumnRangeMetadata::fromColumnStats).collectAsList();
- if (!partitionColumnMetadata.isEmpty()) {
- // incase of shouldScanColStatsForTightBound = true, we compute stats for the partition of interest for all files from getLatestFileSlice() excluding current commit here
- // already fileColumnMetadata contains stats for files from the current infliht commit. so, we are adding both together and sending it to collectAndProcessColumnMetadata
- fileColumnMetadata.addAll(partitionColumnMetadata);
- }
- }
+ List<StoragePathInfo> consolidatedPathInfos = new ArrayList<>();
+ //final Map<String, Stream<FileSlice>> consolidatedFileSliceMap;
+
+ // TODO(yihua): refactor usage of shouldScanColStatsForTightBound
+ // if (shouldScanColStatsForTightBound) {
+
+ // TODO(yihua): seems not possible to directly get latest merged file slices without constructing another FSV
+ tableMetadata.getAllFilesInPartitions(
Review Comment:
in HoodieBackedTableMetadataWriter, we already have a lazy list of latest file slices. we can pass it as an argument to this method and avoid fetching it again.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2808,49 +2812,78 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndexSchemaMap.keySet());
// Group by partitionPath and then gather write stats lists,
// where each inner list contains HoodieWriteStat objects that have the same partitionPath.
- List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(allWriteStats.stream()
- .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
- .values());
+ // TODO(yihua): only invoke FSV resolution if needed based on shouldScanColStatsForTightBound?
+ Map<String, List<HoodieWriteStat>> partitionedWriteStats = allWriteStats.stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath));
int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
boolean shouldScanColStatsForTightBound = isShouldScanColStatsForTightBound(dataMetaClient);
- HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadata = engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
- final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
- // Step 1: Collect Column Metadata for Each File part of current commit metadata
- List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata = partitionedWriteStat.stream()
- .flatMap(writeStat -> translateWriteStatToFileStats(writeStat, dataMetaClient, colsToIndex, partitionStatsIndexVersion).stream()).collect(toList());
-
- if (shouldScanColStatsForTightBound) {
- checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
- // Collect Column Metadata for Each File part of active file system view of latest snapshot
- // Get all file names, including log files, in a set from the file slices
- Set<String> fileNames = getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.empty(), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
- // Fetch metadata table COLUMN_STATS partition records for above files
- List<HoodieColumnRangeMetadata<Comparable>> partitionColumnMetadata = tableMetadata
- .getRecordsByKeyPrefixes(
- HoodieListData.lazy(generateColumnStatsKeys(colsToIndex, partitionName)),
- MetadataPartitionType.COLUMN_STATS.getPartitionPath(), false)
- // schema and properties are ignored in getInsertValue, so simply pass as null
- .map(record -> ((HoodieMetadataPayload)record.getData()).getColumnStatMetadata())
- .filter(Option::isPresent)
- .map(colStatsOpt -> colStatsOpt.get())
- .filter(stats -> fileNames.contains(stats.getFileName()))
- .map(HoodieColumnRangeMetadata::fromColumnStats).collectAsList();
- if (!partitionColumnMetadata.isEmpty()) {
- // incase of shouldScanColStatsForTightBound = true, we compute stats for the partition of interest for all files from getLatestFileSlice() excluding current commit here
- // already fileColumnMetadata contains stats for files from the current infliht commit. so, we are adding both together and sending it to collectAndProcessColumnMetadata
- fileColumnMetadata.addAll(partitionColumnMetadata);
- }
- }
+ List<StoragePathInfo> consolidatedPathInfos = new ArrayList<>();
+ //final Map<String, Stream<FileSlice>> consolidatedFileSliceMap;
+
+ // TODO(yihua): refactor usage of shouldScanColStatsForTightBound
+ // if (shouldScanColStatsForTightBound) {
+
+ // TODO(yihua): seems not possible to directly get latest merged file slices without constructing another FSV
+ tableMetadata.getAllFilesInPartitions(
Review Comment:
why `getAllFilesInPartitions`. we should only poll for `getLatestfileSlice` right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3361184669
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e83ac0429e7d295f64f69eb0326cbf68a87d732 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007)
* 47806303eeed26d7036970ce52c2c56572a47a64 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402108670
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala:
##########
@@ -763,6 +763,8 @@ class TestCOWDataSource extends HoodieSparkClientTestBase with ScalaAssertionSup
.options(writeOpts)
.option(DataSourceWriteOptions.OPERATION.key, DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL)
.option("hoodie.write.merge.handle.class", "org.apache.hudi.io.FileGroupReaderBasedMergeHandle")
+ .option("hoodie.index.type","SIMPLE")
+ .option("hoodie.metadata.enable","false")
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3361684880
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47806303eeed26d7036970ce52c2c56572a47a64 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027)
* 7ed4ec66734b182c328495851b8d9f6d2cb1b70f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3358717071
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e83ac0429e7d295f64f69eb0326cbf68a87d732 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3362728963
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 894ea561d160481482bffbb0f1edccbfb555bcfc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3364656605
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* dfb771fd1781b21adae727ac32fa4552bcca259c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047)
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 7de5bf44a2282bf19d81036ec5eb3f944a5d0407 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2401202570
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -1552,13 +1565,27 @@ private static FileSlice mergeCompactionPendingFileSlices(FileSlice lastSlice, F
*
* @param fileGroup File Group for which the file slice belongs to
* @param fileSlice File Slice which needs to be merged
- */
- private FileSlice fetchMergedFileSlice(HoodieFileGroup fileGroup, FileSlice fileSlice) {
+ * @param currentInstantTime Instant time of the current transaction
+ * @param includeBaseFileUnderInflightCompaction whether to include base file under inflight compaction
+ * when merging file slices
+ */
+ private FileSlice fetchMergedFileSlice(HoodieFileGroup fileGroup,
+ FileSlice fileSlice,
+ String currentInstantTime,
+ boolean includeBaseFileUnderInflightCompaction) {
// if the file-group is under construction, pick the latest before compaction instant time.
Option<Pair<String, CompactionOperation>> compactionOpWithInstant =
getPendingCompactionOperationWithInstant(fileGroup.getFileGroupId());
if (compactionOpWithInstant.isPresent()) {
String compactionInstantTime = compactionOpWithInstant.get().getKey();
+ // Only if includeBaseFileUnderInflightCompaction is true, i.e., the base file from inflight
+ // compaction should be considered in the latest merged file slice, we check the file slice.
Review Comment:
can you help me understand the need here.
does the base file refer to last but one file slice's base file? or the phantom base file?
if its former, doesn't typical `getLatestMergedFileSlice` already return this.
if its later, why do we need to return a base file which is not even created yet?
can you help throw some light please
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/SecondaryIndexRecordGenerationUtils.java:
##########
@@ -114,20 +124,60 @@ public static <T> HoodieData<HoodieRecord> convertWriteStatsToSecondaryIndexReco
String fileId = writeStatsByFileIdEntry.getKey();
List<HoodieWriteStat> writeStats = writeStatsByFileIdEntry.getValue();
String partition = writeStats.get(0).getPartitionPath();
- FileSlice previousFileSliceForFileId = fsView.getLatestFileSlice(partition, fileId).orElse(null);
+ StoragePath basePath = dataMetaClient.getBasePath();
+
+ // validate that for a given fileId, either we have 1 parquet file or N log files.
+ AtomicInteger totalParquetFiles = new AtomicInteger();
+ AtomicInteger totalLogFiles = new AtomicInteger();
+ writeStats.stream().forEach(writeStat -> {
+ if (FSUtils.isLogFile(new StoragePath(basePath, writeStat.getPath()))) {
+ totalLogFiles.getAndIncrement();
+ } else {
+ totalParquetFiles.getAndIncrement();
+ }
+ });
+
+ ValidationUtils.checkArgument(!(totalParquetFiles.get() > 0 && totalLogFiles.get() > 0), "Only either of base file or log files are expected for a given file group. "
+ + "Partition " + partition + ", fileId " + fileId);
+ if (totalParquetFiles.get() > 0) {
+ // we should expect only 1 parquet file
+ ValidationUtils.checkArgument(writeStats.size() == 1, "Only one new parquet file expected per file group per commit");
+ }
+ // Instantiate Remote table FSV
+ TableFileSystemView.SliceView sliceView = getSliceView(writeConfig, dataMetaClient);
+ Option<FileSlice> fileSliceOption = sliceView.getLatestMergedFileSliceBeforeOrOn(partition, instantTime, fileId);
Map<String, String> recordKeyToSecondaryKeyForPreviousFileSlice;
- if (previousFileSliceForFileId == null) {
- // new file slice, so empty mapping for previous slice
- recordKeyToSecondaryKeyForPreviousFileSlice = Collections.emptyMap();
- } else {
+ Map<String, String> recordKeyToSecondaryKeyForCurrentFileSlice;
+ if (fileSliceOption.isPresent()) { // if previous file slice is present.
recordKeyToSecondaryKeyForPreviousFileSlice =
- getRecordKeyToSecondaryKey(dataMetaClient, readerContextFactory.getContext(), previousFileSliceForFileId, tableSchema, indexDefinition, instantTime, props, false);
+ getRecordKeyToSecondaryKey(dataMetaClient, readerContextFactory.getContext(), fileSliceOption.get(), tableSchema, indexDefinition, instantTime, props, false);
Review Comment:
oh, looks like I also pushed my change here by mistake.
I have a separate patch for secondary index here https://github.com/apache/hudi/pull/14045/
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala:
##########
@@ -763,6 +763,8 @@ class TestCOWDataSource extends HoodieSparkClientTestBase with ScalaAssertionSup
.options(writeOpts)
.option(DataSourceWriteOptions.OPERATION.key, DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL)
.option("hoodie.write.merge.handle.class", "org.apache.hudi.io.FileGroupReaderBasedMergeHandle")
+ .option("hoodie.index.type","SIMPLE")
+ .option("hoodie.metadata.enable","false")
Review Comment:
to be cleaned up
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/TableFileSystemView.java:
##########
@@ -166,6 +166,24 @@ Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath, String max
*/
Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime);
+ /**
+ * Stream all "merged" file slices before on an instant time including files under inflight instants.
Review Comment:
minor.
`Stream all latest merged .."
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -1043,22 +1043,35 @@ public final Map<String, Stream<FileSlice>> getAllLatestFileSlicesBeforeOrOn(Str
@Override
public final Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionStr, String maxInstantTime) {
+ return getLatestMergedFileSliceBeforeOrOnInternal(partitionStr, maxInstantTime, maxInstantTime, false);
+ }
+
+ @Override
+ public final Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOnIncludingInflight(String partitionStr, String maxInstantTime, String currentInstantTime) {
Review Comment:
do we have UTs for these
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/hadoop/fs/HoodieWrapperFileSystem.java:
##########
@@ -374,6 +374,7 @@ public boolean delete(Path f, boolean recursive) throws IOException {
@Override
public FileStatus[] listStatus(Path f) throws IOException {
+ System.out.println("!!!!!!! listStatus for " + f);
Review Comment:
all these need to be cleaned up
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/metadata/TestHoodieTableMetadataUtil.java:
##########
@@ -856,4 +856,260 @@ private static Stream<Arguments> mapKeyNoSeparatorToFileGroupIndexTestCases() {
)
);
}
+
+ /*
+ @Disabled
+ @Test
+ public void testConvertMetadataToPartitionStatRecordsWithoutColumnStats() throws Exception {
Review Comment:
why disabled?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/TableFileSystemView.java:
##########
@@ -166,6 +166,24 @@ Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath, String max
*/
Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime);
+ /**
+ * Stream all "merged" file slices before on an instant time including files under inflight instants.
Review Comment:
`before or on ...`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/TableFileSystemView.java:
##########
@@ -166,6 +166,24 @@ Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath, String max
*/
Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime);
+ /**
+ * Stream all "merged" file slices before on an instant time including files under inflight instants.
+ * If a filegroup has a pending compaction request,
+ * (1) if the base file from compaction is not present, the file slice before and after
+ * compaction request instant is merged and returned;
+ * (2) if the base file from compaction is present, and the compaction is inflight, only if the
Review Comment:
now I get the rational for the previous fix.
So, essentially, if current client is actually the compaction, and is polling `getLatestMergedFileSlice including inflight` , we should return the base file created by inflght compaction and ignore entire previous file slice (prev base file + all logs from previous slice).
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/storage/hadoop/HoodieHadoopStorage.java:
##########
@@ -194,6 +200,9 @@ public List<StoragePathInfo> listFiles(StoragePath path) throws IOException {
@Override
public List<StoragePathInfo> listDirectEntries(List<StoragePath> pathList) throws IOException {
+ if (pathList.stream().anyMatch(path -> path.getParent().getName().equals("test_table"))) {
Review Comment:
same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3361714839
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ed4ec66734b182c328495851b8d9f6d2cb1b70f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3362761618
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 894ea561d160481482bffbb0f1edccbfb555bcfc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034)
* 43e107dd015030d5f10317b8180fc99d031561a4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363548745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1566c8ee1e7fa736b3eac4e6228d742daba5a49e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039)
* a73bca8f8d48d18c65ba000bb8e09540767fbf83 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363655601
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a73bca8f8d48d18c65ba000bb8e09540767fbf83 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3363775915
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a73bca8f8d48d18c65ba000bb8e09540767fbf83 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040)
* c4dd726d631f4fc3749017f264d4b45c4c73795a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: [draft] Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3362654435
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ed4ec66734b182c328495851b8d9f6d2cb1b70f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031)
* 894ea561d160481482bffbb0f1edccbfb555bcfc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3364517240
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* dfb771fd1781b21adae727ac32fa4552bcca259c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047)
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14031:
URL: https://github.com/apache/hudi/pull/14031#issuecomment-3364877248
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9007",
"triggerID" : "9e83ac0429e7d295f64f69eb0326cbf68a87d732",
"triggerType" : "PUSH"
}, {
"hash" : "47806303eeed26d7036970ce52c2c56572a47a64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9027",
"triggerID" : "47806303eeed26d7036970ce52c2c56572a47a64",
"triggerType" : "PUSH"
}, {
"hash" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9031",
"triggerID" : "7ed4ec66734b182c328495851b8d9f6d2cb1b70f",
"triggerType" : "PUSH"
}, {
"hash" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9034",
"triggerID" : "894ea561d160481482bffbb0f1edccbfb555bcfc",
"triggerType" : "PUSH"
}, {
"hash" : "43e107dd015030d5f10317b8180fc99d031561a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9035",
"triggerID" : "43e107dd015030d5f10317b8180fc99d031561a4",
"triggerType" : "PUSH"
}, {
"hash" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9039",
"triggerID" : "1566c8ee1e7fa736b3eac4e6228d742daba5a49e",
"triggerType" : "PUSH"
}, {
"hash" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9040",
"triggerID" : "a73bca8f8d48d18c65ba000bb8e09540767fbf83",
"triggerType" : "PUSH"
}, {
"hash" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9046",
"triggerID" : "c4dd726d631f4fc3749017f264d4b45c4c73795a",
"triggerType" : "PUSH"
}, {
"hash" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "413d8fe64791290766f6a92a3ade80cc28775e12",
"triggerType" : "PUSH"
}, {
"hash" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9047",
"triggerID" : "dfb771fd1781b21adae727ac32fa4552bcca259c",
"triggerType" : "PUSH"
}, {
"hash" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db4854b7cdb97412abc0e5f9bb2b25a7db700928",
"triggerType" : "PUSH"
}, {
"hash" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057",
"triggerID" : "7de5bf44a2282bf19d81036ec5eb3f944a5d0407",
"triggerType" : "PUSH"
}, {
"hash" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28f94ed13d59cfe6ab3c77773168c69327e90965",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 413d8fe64791290766f6a92a3ade80cc28775e12 UNKNOWN
* db4854b7cdb97412abc0e5f9bb2b25a7db700928 UNKNOWN
* 7de5bf44a2282bf19d81036ec5eb3f944a5d0407 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9057)
* 28f94ed13d59cfe6ab3c77773168c69327e90965 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402094226
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/SecondaryIndexRecordGenerationUtils.java:
##########
@@ -114,20 +124,60 @@ public static <T> HoodieData<HoodieRecord> convertWriteStatsToSecondaryIndexReco
String fileId = writeStatsByFileIdEntry.getKey();
List<HoodieWriteStat> writeStats = writeStatsByFileIdEntry.getValue();
String partition = writeStats.get(0).getPartitionPath();
- FileSlice previousFileSliceForFileId = fsView.getLatestFileSlice(partition, fileId).orElse(null);
+ StoragePath basePath = dataMetaClient.getBasePath();
+
+ // validate that for a given fileId, either we have 1 parquet file or N log files.
+ AtomicInteger totalParquetFiles = new AtomicInteger();
+ AtomicInteger totalLogFiles = new AtomicInteger();
+ writeStats.stream().forEach(writeStat -> {
+ if (FSUtils.isLogFile(new StoragePath(basePath, writeStat.getPath()))) {
+ totalLogFiles.getAndIncrement();
+ } else {
+ totalParquetFiles.getAndIncrement();
+ }
+ });
+
+ ValidationUtils.checkArgument(!(totalParquetFiles.get() > 0 && totalLogFiles.get() > 0), "Only either of base file or log files are expected for a given file group. "
+ + "Partition " + partition + ", fileId " + fileId);
+ if (totalParquetFiles.get() > 0) {
+ // we should expect only 1 parquet file
+ ValidationUtils.checkArgument(writeStats.size() == 1, "Only one new parquet file expected per file group per commit");
+ }
+ // Instantiate Remote table FSV
+ TableFileSystemView.SliceView sliceView = getSliceView(writeConfig, dataMetaClient);
+ Option<FileSlice> fileSliceOption = sliceView.getLatestMergedFileSliceBeforeOrOn(partition, instantTime, fileId);
Map<String, String> recordKeyToSecondaryKeyForPreviousFileSlice;
- if (previousFileSliceForFileId == null) {
- // new file slice, so empty mapping for previous slice
- recordKeyToSecondaryKeyForPreviousFileSlice = Collections.emptyMap();
- } else {
+ Map<String, String> recordKeyToSecondaryKeyForCurrentFileSlice;
+ if (fileSliceOption.isPresent()) { // if previous file slice is present.
recordKeyToSecondaryKeyForPreviousFileSlice =
- getRecordKeyToSecondaryKey(dataMetaClient, readerContextFactory.getContext(), previousFileSliceForFileId, tableSchema, indexDefinition, instantTime, props, false);
+ getRecordKeyToSecondaryKey(dataMetaClient, readerContextFactory.getContext(), fileSliceOption.get(), tableSchema, indexDefinition, instantTime, props, false);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf: Remove file system based file listing when building records of indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14031:
URL: https://github.com/apache/hudi/pull/14031#discussion_r2402144654
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/metadata/TestHoodieTableMetadataUtil.java:
##########
@@ -856,4 +856,260 @@ private static Stream<Arguments> mapKeyNoSeparatorToFileGroupIndexTestCases() {
)
);
}
+
+ /*
+ @Disabled
+ @Test
+ public void testConvertMetadataToPartitionStatRecordsWithoutColumnStats() throws Exception {
Review Comment:
Removed since these are temporary tests before the util method is moved to another class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org