You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2024/09/10 18:00:54 UTC
[PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
lokeshj1703 opened a new pull request, #11921:
URL: https://github.com/apache/hudi/pull/11921
### Change Logs
In this Jira, we are adding validation support for partition stat index in HoodieMetadataTableValidator. Partition stat index is a new index added in Hudi 1.0.
### Impact
NA
### Risk level (write none, low medium or high below)
low
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2371205341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1406591fc5056b3f0fa0ad6cb88f71abe37ec90f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723)
* 94a0979f0d5093dc443420999362367793859ee6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2367864392
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9343f333e6a8a67fbbad483af7272efd4d078c9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375)
* 2e7c03f87334b8da9f14017b0855af5f531874f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2341831807
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9343f333e6a8a67fbbad483af7272efd4d078c9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11921:
URL: https://github.com/apache/hudi/pull/11921#discussion_r1774687269
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
+ PartitionStatsIndexSupport partitionStatsIndexSupport =
+ new PartitionStatsIndexSupport(engineContext.getSqlContext().sparkSession(), AvroConversionUtils.convertAvroSchemaToStructType(schemaResolver.getTableAvroSchema()), metadataConfig, metaClient,
+ false);
+ HoodieData<HoodieMetadataColumnStats> partitionStats =
+ partitionStatsIndexSupport.loadColumnStatsIndexRecords(JavaConverters.asScalaBufferConverter(metadataTableBasedContext.allColumnNameList).asScala().toSeq(), false);
+ JavaRDD<HoodieMetadataColumnStats> diffRDD = HoodieJavaRDD.getJavaRDD(partitionStats).subtract(HoodieJavaRDD.getJavaRDD(partitionStatsUsingColStats));
+ if (!diffRDD.isEmpty()) {
+ List<HoodieMetadataColumnStats> diff = diffRDD.collect();
+ Set<String> partitionPaths = diff.stream().map(HoodieMetadataColumnStats::getFileName).collect(Collectors.toSet());
+ StringBuilder statDiffMsg = new StringBuilder();
+ for (String partitionPath : partitionPaths) {
+ List<HoodieMetadataColumnStats> diffPartitionStatsColStats = partitionStatsUsingColStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ List<HoodieMetadataColumnStats> diffPartitionStats = partitionStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ statDiffMsg.append(String.format("Partition stats from MDT: %s from colstats: %s", Arrays.toString(diffPartitionStats.toArray()), Arrays.toString(diffPartitionStatsColStats.toArray())));
+ }
+ throw new HoodieValidationException(String.format("Partition stats validation failed diff: %s", statDiffMsg));
+ }
+ }
+
+ private HoodieData<HoodieMetadataColumnStats> getPartitionStatsUsingColStats(HoodieMetadataValidationContext metadataTableBasedContext, Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions, HoodieSparkEngineContext engineContext) {
+ return engineContext.parallelize(allPartitions).flatMap(partitionPath -> {
+ List<FileSlice> latestFileSlicesFromMetadataTable = filterFileSliceBasedOnInflightCleaning(metadataTableBasedContext.getSortedLatestFileSliceList(partitionPath),
+ baseDataFilesForCleaning);
+ List<String> latestFileNames = new ArrayList<>();
+ latestFileSlicesFromMetadataTable.stream().filter(fs -> fs.getBaseFile().isPresent()).forEach(fs -> {
+ latestFileNames.add(fs.getBaseFile().get().getFileName());
+ latestFileNames.addAll(fs.getLogFiles().map(HoodieLogFile::getFileName).collect(Collectors.toList()));
+ });
+ List<HoodieColumnRangeMetadata<Comparable>> colStats = metadataTableBasedContext
+ .getSortedColumnStatsList(partitionPath, latestFileNames);
Review Comment:
Does this method take care of collecting stats from log files as well?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
Review Comment:
let's follow the usual style - if all args fit within 200 chars then no need to break. If there is a need to break, then keep the first arg in the same line as the method, and then each arg on new line indented to the same position as the first arg. I think if you install our checkstyle in the IDE, it should automatically format that way.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
+ PartitionStatsIndexSupport partitionStatsIndexSupport =
+ new PartitionStatsIndexSupport(engineContext.getSqlContext().sparkSession(), AvroConversionUtils.convertAvroSchemaToStructType(schemaResolver.getTableAvroSchema()), metadataConfig, metaClient,
+ false);
+ HoodieData<HoodieMetadataColumnStats> partitionStats =
+ partitionStatsIndexSupport.loadColumnStatsIndexRecords(JavaConverters.asScalaBufferConverter(metadataTableBasedContext.allColumnNameList).asScala().toSeq(), false);
+ JavaRDD<HoodieMetadataColumnStats> diffRDD = HoodieJavaRDD.getJavaRDD(partitionStats).subtract(HoodieJavaRDD.getJavaRDD(partitionStatsUsingColStats));
+ if (!diffRDD.isEmpty()) {
+ List<HoodieMetadataColumnStats> diff = diffRDD.collect();
+ Set<String> partitionPaths = diff.stream().map(HoodieMetadataColumnStats::getFileName).collect(Collectors.toSet());
+ StringBuilder statDiffMsg = new StringBuilder();
+ for (String partitionPath : partitionPaths) {
+ List<HoodieMetadataColumnStats> diffPartitionStatsColStats = partitionStatsUsingColStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ List<HoodieMetadataColumnStats> diffPartitionStats = partitionStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ statDiffMsg.append(String.format("Partition stats from MDT: %s from colstats: %s", Arrays.toString(diffPartitionStats.toArray()), Arrays.toString(diffPartitionStatsColStats.toArray())));
+ }
+ throw new HoodieValidationException(String.format("Partition stats validation failed diff: %s", statDiffMsg));
+ }
+ }
+
+ private HoodieData<HoodieMetadataColumnStats> getPartitionStatsUsingColStats(HoodieMetadataValidationContext metadataTableBasedContext, Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions, HoodieSparkEngineContext engineContext) {
+ return engineContext.parallelize(allPartitions).flatMap(partitionPath -> {
+ List<FileSlice> latestFileSlicesFromMetadataTable = filterFileSliceBasedOnInflightCleaning(metadataTableBasedContext.getSortedLatestFileSliceList(partitionPath),
+ baseDataFilesForCleaning);
+ List<String> latestFileNames = new ArrayList<>();
+ latestFileSlicesFromMetadataTable.stream().filter(fs -> fs.getBaseFile().isPresent()).forEach(fs -> {
+ latestFileNames.add(fs.getBaseFile().get().getFileName());
+ latestFileNames.addAll(fs.getLogFiles().map(HoodieLogFile::getFileName).collect(Collectors.toList()));
+ });
+ List<HoodieColumnRangeMetadata<Comparable>> colStats = metadataTableBasedContext
+ .getSortedColumnStatsList(partitionPath, latestFileNames);
+
+ TreeSet<HoodieColumnRangeMetadata<Comparable>> aggregatedColumnStats = aggregateColumnStats(partitionPath, colStats);
+ List<HoodieRecord> partitionStatRecords = HoodieMetadataPayload.createPartitionStatsRecords(partitionPath, new ArrayList<>(aggregatedColumnStats), false)
+ .collect(Collectors.toList());
+ return partitionStatRecords.stream()
+ .map(record -> {
+ try {
+ return ((HoodieMetadataPayload) record.getData()).getInsertValue(null, null)
+ .map(metadataRecord -> ((HoodieMetadataRecord) metadataRecord).getColumnStatsMetadata());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ })
+ .filter(Option::isPresent)
+ .map(Option::get)
+ .collect(Collectors.toList())
+ .iterator();
+ });
+ }
+
+ /**
+ * Generates aggregated column stats which also signify as partition stat for the particular partition
+ * path.
+ *
+ * @param partitionPath Provided partition path
+ * @param colStats Column stat records for the partition
+ */
+ private static TreeSet<HoodieColumnRangeMetadata<Comparable>> aggregateColumnStats(String partitionPath, List<HoodieColumnRangeMetadata<Comparable>> colStats) {
+ TreeSet<HoodieColumnRangeMetadata<Comparable>> aggregatedColumnStats = new TreeSet<>(Comparator.comparing(HoodieColumnRangeMetadata::getColumnName));
+ for (HoodieColumnRangeMetadata<Comparable> colStat : colStats) {
Review Comment:
Check if you can `reduce` colStats using `HoodieColumnRangeMetadata.merge`. Then we may not need this method.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
+ PartitionStatsIndexSupport partitionStatsIndexSupport =
+ new PartitionStatsIndexSupport(engineContext.getSqlContext().sparkSession(), AvroConversionUtils.convertAvroSchemaToStructType(schemaResolver.getTableAvroSchema()), metadataConfig, metaClient,
+ false);
+ HoodieData<HoodieMetadataColumnStats> partitionStats =
+ partitionStatsIndexSupport.loadColumnStatsIndexRecords(JavaConverters.asScalaBufferConverter(metadataTableBasedContext.allColumnNameList).asScala().toSeq(), false);
+ JavaRDD<HoodieMetadataColumnStats> diffRDD = HoodieJavaRDD.getJavaRDD(partitionStats).subtract(HoodieJavaRDD.getJavaRDD(partitionStatsUsingColStats));
+ if (!diffRDD.isEmpty()) {
+ List<HoodieMetadataColumnStats> diff = diffRDD.collect();
+ Set<String> partitionPaths = diff.stream().map(HoodieMetadataColumnStats::getFileName).collect(Collectors.toSet());
+ StringBuilder statDiffMsg = new StringBuilder();
+ for (String partitionPath : partitionPaths) {
+ List<HoodieMetadataColumnStats> diffPartitionStatsColStats = partitionStatsUsingColStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ List<HoodieMetadataColumnStats> diffPartitionStats = partitionStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ statDiffMsg.append(String.format("Partition stats from MDT: %s from colstats: %s", Arrays.toString(diffPartitionStats.toArray()), Arrays.toString(diffPartitionStatsColStats.toArray())));
+ }
+ throw new HoodieValidationException(String.format("Partition stats validation failed diff: %s", statDiffMsg));
+ }
+ }
+
+ private HoodieData<HoodieMetadataColumnStats> getPartitionStatsUsingColStats(HoodieMetadataValidationContext metadataTableBasedContext, Set<String> baseDataFilesForCleaning,
Review Comment:
Should we rely on colstats for partition stats validation? What if colstats itself is wrong? Maybe we can compute colstats from scratch for the latest file slices. Since this is a validator we can probably afford to incur higher overhead.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
Review Comment:
Please check if we need to extract the schema resolver as a member variable. I see it's getting instantiated at multiple places in the class.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -600,6 +634,8 @@ public boolean doMetadataTableValidation() {
LOG.warn("Metadata table validation failed ({}).", taskLabels);
return false;
}
+ } catch (HoodieValidationException e) {
+ throw e;
Review Comment:
Should we return false with a warn message?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
Review Comment:
This should be part of `HoodieMetadataValidationContext`
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieMetadataTableValidator.java:
##########
@@ -139,6 +139,50 @@ public void testMetadataTableValidation(String viewStorageTypeForFSListing, Stri
assertTrue(validator.getThrowables().isEmpty());
}
+ @Test
+ public void testPartitionStatsValidation() {
+ // TODO: Add validation for compaction and clustering cases
Review Comment:
you can add it with a relaxed validation - partition stats range is same or wider than that after aggregating stats for all files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2377465094
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
}, {
"hash" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795",
"triggerID" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"triggerType" : "PUSH"
}, {
"hash" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=799",
"triggerID" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"triggerType" : "PUSH"
}, {
"hash" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=799)
* c5d2797cb26fae800efd4937ba5ea33b697f5768 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2376886780
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
}, {
"hash" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795",
"triggerID" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"triggerType" : "PUSH"
}, {
"hash" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* faed95611cba8f5b510a37a0aff0f9f1d2f287f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795)
* 05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11921:
URL: https://github.com/apache/hudi/pull/11921#discussion_r1777440510
##########
hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java:
##########
@@ -71,7 +72,7 @@
* Helper class to read schema from data files and log files and to convert it between different formats.
*/
@ThreadSafe
-public class TableSchemaResolver {
+public class TableSchemaResolver implements Serializable {
Review Comment:
Good catch! Addressed the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11921:
URL: https://github.com/apache/hudi/pull/11921#discussion_r1775705839
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
+ PartitionStatsIndexSupport partitionStatsIndexSupport =
+ new PartitionStatsIndexSupport(engineContext.getSqlContext().sparkSession(), AvroConversionUtils.convertAvroSchemaToStructType(schemaResolver.getTableAvroSchema()), metadataConfig, metaClient,
+ false);
+ HoodieData<HoodieMetadataColumnStats> partitionStats =
+ partitionStatsIndexSupport.loadColumnStatsIndexRecords(JavaConverters.asScalaBufferConverter(metadataTableBasedContext.allColumnNameList).asScala().toSeq(), false);
+ JavaRDD<HoodieMetadataColumnStats> diffRDD = HoodieJavaRDD.getJavaRDD(partitionStats).subtract(HoodieJavaRDD.getJavaRDD(partitionStatsUsingColStats));
+ if (!diffRDD.isEmpty()) {
+ List<HoodieMetadataColumnStats> diff = diffRDD.collect();
+ Set<String> partitionPaths = diff.stream().map(HoodieMetadataColumnStats::getFileName).collect(Collectors.toSet());
+ StringBuilder statDiffMsg = new StringBuilder();
+ for (String partitionPath : partitionPaths) {
+ List<HoodieMetadataColumnStats> diffPartitionStatsColStats = partitionStatsUsingColStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ List<HoodieMetadataColumnStats> diffPartitionStats = partitionStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ statDiffMsg.append(String.format("Partition stats from MDT: %s from colstats: %s", Arrays.toString(diffPartitionStats.toArray()), Arrays.toString(diffPartitionStatsColStats.toArray())));
+ }
+ throw new HoodieValidationException(String.format("Partition stats validation failed diff: %s", statDiffMsg));
+ }
+ }
+
+ private HoodieData<HoodieMetadataColumnStats> getPartitionStatsUsingColStats(HoodieMetadataValidationContext metadataTableBasedContext, Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions, HoodieSparkEngineContext engineContext) {
+ return engineContext.parallelize(allPartitions).flatMap(partitionPath -> {
+ List<FileSlice> latestFileSlicesFromMetadataTable = filterFileSliceBasedOnInflightCleaning(metadataTableBasedContext.getSortedLatestFileSliceList(partitionPath),
+ baseDataFilesForCleaning);
+ List<String> latestFileNames = new ArrayList<>();
+ latestFileSlicesFromMetadataTable.stream().filter(fs -> fs.getBaseFile().isPresent()).forEach(fs -> {
+ latestFileNames.add(fs.getBaseFile().get().getFileName());
+ latestFileNames.addAll(fs.getLogFiles().map(HoodieLogFile::getFileName).collect(Collectors.toList()));
+ });
+ List<HoodieColumnRangeMetadata<Comparable>> colStats = metadataTableBasedContext
+ .getSortedColumnStatsList(partitionPath, latestFileNames);
Review Comment:
Yes, it does. latestFileNames considers both base files and log files.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
+ PartitionStatsIndexSupport partitionStatsIndexSupport =
+ new PartitionStatsIndexSupport(engineContext.getSqlContext().sparkSession(), AvroConversionUtils.convertAvroSchemaToStructType(schemaResolver.getTableAvroSchema()), metadataConfig, metaClient,
+ false);
+ HoodieData<HoodieMetadataColumnStats> partitionStats =
+ partitionStatsIndexSupport.loadColumnStatsIndexRecords(JavaConverters.asScalaBufferConverter(metadataTableBasedContext.allColumnNameList).asScala().toSeq(), false);
+ JavaRDD<HoodieMetadataColumnStats> diffRDD = HoodieJavaRDD.getJavaRDD(partitionStats).subtract(HoodieJavaRDD.getJavaRDD(partitionStatsUsingColStats));
+ if (!diffRDD.isEmpty()) {
+ List<HoodieMetadataColumnStats> diff = diffRDD.collect();
+ Set<String> partitionPaths = diff.stream().map(HoodieMetadataColumnStats::getFileName).collect(Collectors.toSet());
+ StringBuilder statDiffMsg = new StringBuilder();
+ for (String partitionPath : partitionPaths) {
+ List<HoodieMetadataColumnStats> diffPartitionStatsColStats = partitionStatsUsingColStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ List<HoodieMetadataColumnStats> diffPartitionStats = partitionStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ statDiffMsg.append(String.format("Partition stats from MDT: %s from colstats: %s", Arrays.toString(diffPartitionStats.toArray()), Arrays.toString(diffPartitionStatsColStats.toArray())));
+ }
+ throw new HoodieValidationException(String.format("Partition stats validation failed diff: %s", statDiffMsg));
+ }
+ }
+
+ private HoodieData<HoodieMetadataColumnStats> getPartitionStatsUsingColStats(HoodieMetadataValidationContext metadataTableBasedContext, Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions, HoodieSparkEngineContext engineContext) {
+ return engineContext.parallelize(allPartitions).flatMap(partitionPath -> {
+ List<FileSlice> latestFileSlicesFromMetadataTable = filterFileSliceBasedOnInflightCleaning(metadataTableBasedContext.getSortedLatestFileSliceList(partitionPath),
+ baseDataFilesForCleaning);
+ List<String> latestFileNames = new ArrayList<>();
+ latestFileSlicesFromMetadataTable.stream().filter(fs -> fs.getBaseFile().isPresent()).forEach(fs -> {
+ latestFileNames.add(fs.getBaseFile().get().getFileName());
+ latestFileNames.addAll(fs.getLogFiles().map(HoodieLogFile::getFileName).collect(Collectors.toList()));
+ });
+ List<HoodieColumnRangeMetadata<Comparable>> colStats = metadataTableBasedContext
+ .getSortedColumnStatsList(partitionPath, latestFileNames);
+
+ TreeSet<HoodieColumnRangeMetadata<Comparable>> aggregatedColumnStats = aggregateColumnStats(partitionPath, colStats);
+ List<HoodieRecord> partitionStatRecords = HoodieMetadataPayload.createPartitionStatsRecords(partitionPath, new ArrayList<>(aggregatedColumnStats), false)
+ .collect(Collectors.toList());
+ return partitionStatRecords.stream()
+ .map(record -> {
+ try {
+ return ((HoodieMetadataPayload) record.getData()).getInsertValue(null, null)
+ .map(metadataRecord -> ((HoodieMetadataRecord) metadataRecord).getColumnStatsMetadata());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ })
+ .filter(Option::isPresent)
+ .map(Option::get)
+ .collect(Collectors.toList())
+ .iterator();
+ });
+ }
+
+ /**
+ * Generates aggregated column stats which also signify as partition stat for the particular partition
+ * path.
+ *
+ * @param partitionPath Provided partition path
+ * @param colStats Column stat records for the partition
+ */
+ private static TreeSet<HoodieColumnRangeMetadata<Comparable>> aggregateColumnStats(String partitionPath, List<HoodieColumnRangeMetadata<Comparable>> colStats) {
+ TreeSet<HoodieColumnRangeMetadata<Comparable>> aggregatedColumnStats = new TreeSet<>(Comparator.comparing(HoodieColumnRangeMetadata::getColumnName));
+ for (HoodieColumnRangeMetadata<Comparable> colStat : colStats) {
Review Comment:
This is still needed since I am changing the file name to paritition path in the column stat record
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
Review Comment:
Addressed
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11921:
URL: https://github.com/apache/hudi/pull/11921#discussion_r1775705064
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
+ PartitionStatsIndexSupport partitionStatsIndexSupport =
+ new PartitionStatsIndexSupport(engineContext.getSqlContext().sparkSession(), AvroConversionUtils.convertAvroSchemaToStructType(schemaResolver.getTableAvroSchema()), metadataConfig, metaClient,
+ false);
+ HoodieData<HoodieMetadataColumnStats> partitionStats =
+ partitionStatsIndexSupport.loadColumnStatsIndexRecords(JavaConverters.asScalaBufferConverter(metadataTableBasedContext.allColumnNameList).asScala().toSeq(), false);
+ JavaRDD<HoodieMetadataColumnStats> diffRDD = HoodieJavaRDD.getJavaRDD(partitionStats).subtract(HoodieJavaRDD.getJavaRDD(partitionStatsUsingColStats));
+ if (!diffRDD.isEmpty()) {
+ List<HoodieMetadataColumnStats> diff = diffRDD.collect();
+ Set<String> partitionPaths = diff.stream().map(HoodieMetadataColumnStats::getFileName).collect(Collectors.toSet());
+ StringBuilder statDiffMsg = new StringBuilder();
+ for (String partitionPath : partitionPaths) {
+ List<HoodieMetadataColumnStats> diffPartitionStatsColStats = partitionStatsUsingColStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ List<HoodieMetadataColumnStats> diffPartitionStats = partitionStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ statDiffMsg.append(String.format("Partition stats from MDT: %s from colstats: %s", Arrays.toString(diffPartitionStats.toArray()), Arrays.toString(diffPartitionStatsColStats.toArray())));
+ }
+ throw new HoodieValidationException(String.format("Partition stats validation failed diff: %s", statDiffMsg));
+ }
+ }
+
+ private HoodieData<HoodieMetadataColumnStats> getPartitionStatsUsingColStats(HoodieMetadataValidationContext metadataTableBasedContext, Set<String> baseDataFilesForCleaning,
Review Comment:
We need to rely on colstats so that log files are also accounted for. Created https://issues.apache.org/jira/browse/HUDI-8260 to fix the validation logic in col stats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2376014374
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
}, {
"hash" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795",
"triggerID" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* faed95611cba8f5b510a37a0aff0f9f1d2f287f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11921:
URL: https://github.com/apache/hudi/pull/11921#discussion_r1775706138
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
Review Comment:
Addressed
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -600,6 +634,8 @@ public boolean doMetadataTableValidation() {
LOG.warn("Metadata table validation failed ({}).", taskLabels);
return false;
}
+ } catch (HoodieValidationException e) {
+ throw e;
Review Comment:
This is an error where HoodieValidationException is never thrown even if ignoreFailed is false. For example if validation exception is thrown by partition stat validation logic, it will be caught and will lead to validation being successful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2374979882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4403271f24e75fcb2b5d3c06d6eb84267c7c351 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2375875355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
}, {
"hash" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795",
"triggerID" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4403271f24e75fcb2b5d3c06d6eb84267c7c351 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787)
* faed95611cba8f5b510a37a0aff0f9f1d2f287f2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2368003826
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e7c03f87334b8da9f14017b0855af5f531874f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11921:
URL: https://github.com/apache/hudi/pull/11921#discussion_r1777309592
##########
hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java:
##########
@@ -71,7 +72,7 @@
* Helper class to read schema from data files and log files and to convert it between different formats.
*/
@ThreadSafe
-public class TableSchemaResolver {
+public class TableSchemaResolver implements Serializable {
Review Comment:
Why does `TableSchemaResolver` have to be Serializable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2377602693
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
}, {
"hash" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795",
"triggerID" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"triggerType" : "PUSH"
}, {
"hash" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=799",
"triggerID" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"triggerType" : "PUSH"
}, {
"hash" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=802",
"triggerID" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5d2797cb26fae800efd4937ba5ea33b697f5768 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=802)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2368058477
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e7c03f87334b8da9f14017b0855af5f531874f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720)
* 1406591fc5056b3f0fa0ad6cb88f71abe37ec90f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11921:
URL: https://github.com/apache/hudi/pull/11921#discussion_r1775700225
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieMetadataTableValidator.java:
##########
@@ -139,6 +139,50 @@ public void testMetadataTableValidation(String viewStorageTypeForFSListing, Stri
assertTrue(validator.getThrowables().isEmpty());
}
+ @Test
+ public void testPartitionStatsValidation() {
+ // TODO: Add validation for compaction and clustering cases
Review Comment:
I can probably address this in HUDI-8208. Added a note in the jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2367867357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9343f333e6a8a67fbbad483af7272efd4d078c9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375)
* 2e7c03f87334b8da9f14017b0855af5f531874f2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2368061335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e7c03f87334b8da9f14017b0855af5f531874f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720)
* 1406591fc5056b3f0fa0ad6cb88f71abe37ec90f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2376890293
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
}, {
"hash" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795",
"triggerID" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"triggerType" : "PUSH"
}, {
"hash" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=799",
"triggerID" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* faed95611cba8f5b510a37a0aff0f9f1d2f287f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795)
* 05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=799)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2374788198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94a0979f0d5093dc443420999362367793859ee6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742)
* c4403271f24e75fcb2b5d3c06d6eb84267c7c351 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2341631736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9343f333e6a8a67fbbad483af7272efd4d078c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11921:
URL: https://github.com/apache/hudi/pull/11921#discussion_r1776588105
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
+ PartitionStatsIndexSupport partitionStatsIndexSupport =
+ new PartitionStatsIndexSupport(engineContext.getSqlContext().sparkSession(), AvroConversionUtils.convertAvroSchemaToStructType(schemaResolver.getTableAvroSchema()), metadataConfig, metaClient,
+ false);
+ HoodieData<HoodieMetadataColumnStats> partitionStats =
+ partitionStatsIndexSupport.loadColumnStatsIndexRecords(JavaConverters.asScalaBufferConverter(metadataTableBasedContext.allColumnNameList).asScala().toSeq(), false);
+ JavaRDD<HoodieMetadataColumnStats> diffRDD = HoodieJavaRDD.getJavaRDD(partitionStats).subtract(HoodieJavaRDD.getJavaRDD(partitionStatsUsingColStats));
+ if (!diffRDD.isEmpty()) {
+ List<HoodieMetadataColumnStats> diff = diffRDD.collect();
+ Set<String> partitionPaths = diff.stream().map(HoodieMetadataColumnStats::getFileName).collect(Collectors.toSet());
+ StringBuilder statDiffMsg = new StringBuilder();
+ for (String partitionPath : partitionPaths) {
+ List<HoodieMetadataColumnStats> diffPartitionStatsColStats = partitionStatsUsingColStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ List<HoodieMetadataColumnStats> diffPartitionStats = partitionStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ statDiffMsg.append(String.format("Partition stats from MDT: %s from colstats: %s", Arrays.toString(diffPartitionStats.toArray()), Arrays.toString(diffPartitionStatsColStats.toArray())));
+ }
+ throw new HoodieValidationException(String.format("Partition stats validation failed diff: %s", statDiffMsg));
+ }
+ }
+
+ private HoodieData<HoodieMetadataColumnStats> getPartitionStatsUsingColStats(HoodieMetadataValidationContext metadataTableBasedContext, Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions, HoodieSparkEngineContext engineContext) {
+ return engineContext.parallelize(allPartitions).flatMap(partitionPath -> {
+ List<FileSlice> latestFileSlicesFromMetadataTable = filterFileSliceBasedOnInflightCleaning(metadataTableBasedContext.getSortedLatestFileSliceList(partitionPath),
+ baseDataFilesForCleaning);
+ List<String> latestFileNames = new ArrayList<>();
+ latestFileSlicesFromMetadataTable.stream().filter(fs -> fs.getBaseFile().isPresent()).forEach(fs -> {
+ latestFileNames.add(fs.getBaseFile().get().getFileName());
+ latestFileNames.addAll(fs.getLogFiles().map(HoodieLogFile::getFileName).collect(Collectors.toList()));
+ });
+ List<HoodieColumnRangeMetadata<Comparable>> colStats = metadataTableBasedContext
+ .getSortedColumnStatsList(partitionPath, latestFileNames);
+
+ TreeSet<HoodieColumnRangeMetadata<Comparable>> aggregatedColumnStats = aggregateColumnStats(partitionPath, colStats);
+ List<HoodieRecord> partitionStatRecords = HoodieMetadataPayload.createPartitionStatsRecords(partitionPath, new ArrayList<>(aggregatedColumnStats), false)
+ .collect(Collectors.toList());
+ return partitionStatRecords.stream()
+ .map(record -> {
+ try {
+ return ((HoodieMetadataPayload) record.getData()).getInsertValue(null, null)
+ .map(metadataRecord -> ((HoodieMetadataRecord) metadataRecord).getColumnStatsMetadata());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ })
+ .filter(Option::isPresent)
+ .map(Option::get)
+ .collect(Collectors.toList())
+ .iterator();
+ });
+ }
+
+ /**
+ * Generates aggregated column stats which also signify as partition stat for the particular partition
+ * path.
+ *
+ * @param partitionPath Provided partition path
+ * @param colStats Column stat records for the partition
+ */
+ private static TreeSet<HoodieColumnRangeMetadata<Comparable>> aggregateColumnStats(String partitionPath, List<HoodieColumnRangeMetadata<Comparable>> colStats) {
+ TreeSet<HoodieColumnRangeMetadata<Comparable>> aggregatedColumnStats = new TreeSet<>(Comparator.comparing(HoodieColumnRangeMetadata::getColumnName));
+ for (HoodieColumnRangeMetadata<Comparable> colStat : colStats) {
Review Comment:
let's UT this method.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -914,6 +950,97 @@ private void validateAllColumnStats(
validate(metadataBasedColStats, fsBasedColStats, partitionPath, "column stats");
}
+ private void validatePartitionStats(
+ HoodieMetadataValidationContext metadataTableBasedContext,
+ Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions) throws Exception {
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieData<HoodieMetadataColumnStats> partitionStatsUsingColStats = getPartitionStatsUsingColStats(metadataTableBasedContext,
+ baseDataFilesForCleaning, allPartitions, engineContext);
+
+ TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
+ boolean enableMetadataTable = true;
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(enableMetadataTable)
+ .withMetadataIndexBloomFilter(enableMetadataTable)
+ .withMetadataIndexColumnStats(enableMetadataTable)
+ .withEnableRecordIndex(enableMetadataTable)
+ .build();
+ PartitionStatsIndexSupport partitionStatsIndexSupport =
+ new PartitionStatsIndexSupport(engineContext.getSqlContext().sparkSession(), AvroConversionUtils.convertAvroSchemaToStructType(schemaResolver.getTableAvroSchema()), metadataConfig, metaClient,
+ false);
+ HoodieData<HoodieMetadataColumnStats> partitionStats =
+ partitionStatsIndexSupport.loadColumnStatsIndexRecords(JavaConverters.asScalaBufferConverter(metadataTableBasedContext.allColumnNameList).asScala().toSeq(), false);
+ JavaRDD<HoodieMetadataColumnStats> diffRDD = HoodieJavaRDD.getJavaRDD(partitionStats).subtract(HoodieJavaRDD.getJavaRDD(partitionStatsUsingColStats));
+ if (!diffRDD.isEmpty()) {
+ List<HoodieMetadataColumnStats> diff = diffRDD.collect();
+ Set<String> partitionPaths = diff.stream().map(HoodieMetadataColumnStats::getFileName).collect(Collectors.toSet());
+ StringBuilder statDiffMsg = new StringBuilder();
+ for (String partitionPath : partitionPaths) {
+ List<HoodieMetadataColumnStats> diffPartitionStatsColStats = partitionStatsUsingColStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ List<HoodieMetadataColumnStats> diffPartitionStats = partitionStats.filter(stat -> stat.getFileName().equals(partitionPath)).collectAsList();
+ statDiffMsg.append(String.format("Partition stats from MDT: %s from colstats: %s", Arrays.toString(diffPartitionStats.toArray()), Arrays.toString(diffPartitionStatsColStats.toArray())));
+ }
+ throw new HoodieValidationException(String.format("Partition stats validation failed diff: %s", statDiffMsg));
+ }
+ }
+
+ private HoodieData<HoodieMetadataColumnStats> getPartitionStatsUsingColStats(HoodieMetadataValidationContext metadataTableBasedContext, Set<String> baseDataFilesForCleaning,
+ List<String> allPartitions, HoodieSparkEngineContext engineContext) {
+ return engineContext.parallelize(allPartitions).flatMap(partitionPath -> {
+ List<FileSlice> latestFileSlicesFromMetadataTable = filterFileSliceBasedOnInflightCleaning(metadataTableBasedContext.getSortedLatestFileSliceList(partitionPath),
+ baseDataFilesForCleaning);
+ List<String> latestFileNames = new ArrayList<>();
+ latestFileSlicesFromMetadataTable.stream().filter(fs -> fs.getBaseFile().isPresent()).forEach(fs -> {
+ latestFileNames.add(fs.getBaseFile().get().getFileName());
+ latestFileNames.addAll(fs.getLogFiles().map(HoodieLogFile::getFileName).collect(Collectors.toList()));
+ });
+ List<HoodieColumnRangeMetadata<Comparable>> colStats = metadataTableBasedContext
+ .getSortedColumnStatsList(partitionPath, latestFileNames);
Review Comment:
The argument is named as if only base files are being considered - https://github.com/apache/hudi/blob/30f077d0ded790cdd908a84247c265189683c743/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java#L1486
Can you check again that log files are considered? I think if colstats is present then it's fine otherwise it only considers base files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2377467819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
}, {
"hash" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795",
"triggerID" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"triggerType" : "PUSH"
}, {
"hash" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=799",
"triggerID" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"triggerType" : "PUSH"
}, {
"hash" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=802",
"triggerID" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=799)
* c5d2797cb26fae800efd4937ba5ea33b697f5768 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=802)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2378325603
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=802",
"triggerID" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5d2797cb26fae800efd4937ba5ea33b697f5768 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=802)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2375874096
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
}, {
"hash" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4403271f24e75fcb2b5d3c06d6eb84267c7c351 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787)
* faed95611cba8f5b510a37a0aff0f9f1d2f287f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2371421205
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94a0979f0d5093dc443420999362367793859ee6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2341637095
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9343f333e6a8a67fbbad483af7272efd4d078c9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2377117093
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
}, {
"hash" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=795",
"triggerID" : "faed95611cba8f5b510a37a0aff0f9f1d2f287f2",
"triggerType" : "PUSH"
}, {
"hash" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=799",
"triggerID" : "05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 05fb2196a6f89de0d7fa02b42d1b3ddad3a41d97 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=799)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2368317552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1406591fc5056b3f0fa0ad6cb88f71abe37ec90f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2371202165
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1406591fc5056b3f0fa0ad6cb88f71abe37ec90f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723)
* 94a0979f0d5093dc443420999362367793859ee6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2374792864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=375",
"triggerID" : "a9343f333e6a8a67fbbad483af7272efd4d078c9",
"triggerType" : "PUSH"
}, {
"hash" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=720",
"triggerID" : "2e7c03f87334b8da9f14017b0855af5f531874f2",
"triggerType" : "PUSH"
}, {
"hash" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=723",
"triggerID" : "1406591fc5056b3f0fa0ad6cb88f71abe37ec90f",
"triggerType" : "PUSH"
}, {
"hash" : "94a0979f0d5093dc443420999362367793859ee6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742",
"triggerID" : "94a0979f0d5093dc443420999362367793859ee6",
"triggerType" : "PUSH"
}, {
"hash" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787",
"triggerID" : "c4403271f24e75fcb2b5d3c06d6eb84267c7c351",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94a0979f0d5093dc443420999362367793859ee6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=742)
* c4403271f24e75fcb2b5d3c06d6eb84267c7c351 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11921:
URL: https://github.com/apache/hudi/pull/11921#issuecomment-2378324740
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5d2797cb26fae800efd4937ba5ea33b697f5768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5d2797cb26fae800efd4937ba5ea33b697f5768 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #11921:
URL: https://github.com/apache/hudi/pull/11921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org