You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2024/10/24 17:28:33 UTC
[PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
lokeshj1703 opened a new pull request, #12159:
URL: https://github.com/apache/hudi/pull/12159
### Change Logs
Currently log files are not validated while validating column stats. The Jira aims to fix the validation logic so that log files are also validated along with base files.
Also if col stats validation fails, partition stats validation can be skipped since partition stats validation depends on col stats validation.
### Impact
NA
### Risk level (write none, low medium or high below)
low
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12159:
URL: https://github.com/apache/hudi/pull/12159#discussion_r1818893117
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -1730,13 +1743,28 @@ public List<HoodieColumnRangeMetadata<Comparable>> getSortedColumnStatsList(
} else {
FileFormatUtils formatUtils = HoodieIOFactory.getIOFactory(metaClient.getStorage())
.getFileFormatUtils(HoodieFileFormat.PARQUET);
- return fileNames.stream().flatMap(filename ->
- formatUtils.readColumnStatsFromMetadata(
- metaClient.getStorage(),
- new StoragePath(FSUtils.constructAbsolutePath(metaClient.getBasePath(), partitionPath), filename),
- allColumnNameList).stream())
- .sorted(new HoodieColumnRangeMetadataComparator())
- .collect(Collectors.toList());
+ Schema readerSchema = new TableSchemaResolver(metaClient).getTableAvroSchema();
Review Comment:
can we reuse the schema available in the validation context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #12159:
URL: https://github.com/apache/hudi/pull/12159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12159:
URL: https://github.com/apache/hudi/pull/12159#issuecomment-2441565105
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397",
"triggerID" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"triggerType" : "PUSH"
}, {
"hash" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1415",
"triggerID" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"triggerType" : "PUSH"
}, {
"hash" : "24dcbf67a0731d6213a2b8ae92241784e5da1c3f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1436",
"triggerID" : "24dcbf67a0731d6213a2b8ae92241784e5da1c3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24dcbf67a0731d6213a2b8ae92241784e5da1c3f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1436)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12159:
URL: https://github.com/apache/hudi/pull/12159#issuecomment-2437506315
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397",
"triggerID" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"triggerType" : "PUSH"
}, {
"hash" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1415",
"triggerID" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec6b68586cac25b5f7f2f56f2f4c42e916b89681 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397)
* c43dafc362c3ca8ee69b09171392dd6d768e9b92 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1415)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12159:
URL: https://github.com/apache/hudi/pull/12159#issuecomment-2435914852
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec6b68586cac25b5f7f2f56f2f4c42e916b89681 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12159:
URL: https://github.com/apache/hudi/pull/12159#issuecomment-2441367127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397",
"triggerID" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"triggerType" : "PUSH"
}, {
"hash" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1415",
"triggerID" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"triggerType" : "PUSH"
}, {
"hash" : "24dcbf67a0731d6213a2b8ae92241784e5da1c3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24dcbf67a0731d6213a2b8ae92241784e5da1c3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c43dafc362c3ca8ee69b09171392dd6d768e9b92 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1415)
* 24dcbf67a0731d6213a2b8ae92241784e5da1c3f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12159:
URL: https://github.com/apache/hudi/pull/12159#issuecomment-2436101496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397",
"triggerID" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec6b68586cac25b5f7f2f56f2f4c42e916b89681 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12159:
URL: https://github.com/apache/hudi/pull/12159#issuecomment-2435917904
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397",
"triggerID" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec6b68586cac25b5f7f2f56f2f4c42e916b89681 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12159:
URL: https://github.com/apache/hudi/pull/12159#issuecomment-2437503783
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397",
"triggerID" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"triggerType" : "PUSH"
}, {
"hash" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec6b68586cac25b5f7f2f56f2f4c42e916b89681 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397)
* c43dafc362c3ca8ee69b09171392dd6d768e9b92 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12159:
URL: https://github.com/apache/hudi/pull/12159#issuecomment-2441370120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397",
"triggerID" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"triggerType" : "PUSH"
}, {
"hash" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1415",
"triggerID" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"triggerType" : "PUSH"
}, {
"hash" : "24dcbf67a0731d6213a2b8ae92241784e5da1c3f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1436",
"triggerID" : "24dcbf67a0731d6213a2b8ae92241784e5da1c3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c43dafc362c3ca8ee69b09171392dd6d768e9b92 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1415)
* 24dcbf67a0731d6213a2b8ae92241784e5da1c3f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1436)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12159:
URL: https://github.com/apache/hudi/pull/12159#discussion_r1818923083
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -1730,13 +1743,28 @@ public List<HoodieColumnRangeMetadata<Comparable>> getSortedColumnStatsList(
} else {
FileFormatUtils formatUtils = HoodieIOFactory.getIOFactory(metaClient.getStorage())
.getFileFormatUtils(HoodieFileFormat.PARQUET);
- return fileNames.stream().flatMap(filename ->
- formatUtils.readColumnStatsFromMetadata(
- metaClient.getStorage(),
- new StoragePath(FSUtils.constructAbsolutePath(metaClient.getBasePath(), partitionPath), filename),
- allColumnNameList).stream())
- .sorted(new HoodieColumnRangeMetadataComparator())
- .collect(Collectors.toList());
+ Schema readerSchema = new TableSchemaResolver(metaClient).getTableAvroSchema();
Review Comment:
fixed this in my latest revision https://github.com/apache/hudi/pull/12159/commits/24dcbf67a0731d6213a2b8ae92241784e5da1c3f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8260] Fix col stats metadata validation so that log files are also validated [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12159:
URL: https://github.com/apache/hudi/pull/12159#issuecomment-2437650190
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1397",
"triggerID" : "ec6b68586cac25b5f7f2f56f2f4c42e916b89681",
"triggerType" : "PUSH"
}, {
"hash" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1415",
"triggerID" : "c43dafc362c3ca8ee69b09171392dd6d768e9b92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c43dafc362c3ca8ee69b09171392dd6d768e9b92 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1415)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org