You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/23 03:41:43 UTC
[GitHub] [hudi] zhangyue19921010 opened a new pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
zhangyue19921010 opened a new pull request #5100:
URL: https://github.com/apache/hudi/pull/5100
Please refer https://issues.apache.org/jira/browse/HUDI-3635 for more details.
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1075883416
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1078565923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
}, {
"hash" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213)
* aca2c3e03ef2d33d22a3391e0c24f04529cee6c5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1075883416
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1075884653
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1078725760
This patch is tested on local. Also CI passed. I believe it's ready for review :) cc @yihua
With out this patch, HoodieMetadataTableValidator may fail because of new partition is creating but not committed.
```
3801 [main] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Size of file spilled to disk => 0
3802 [main] INFO org.apache.hudi.metadata.HoodieBackedTableMetadata - Opened 3 metadata log files (dataset instant=20220323111740608, metadata instant=20220323111740608) in 469 ms
3819 [main] INFO org.apache.hudi.metadata.BaseTableMetadata - Listed partitions from metadata: #partitions=1
3819 [main] ERROR org.apache.hudi.utilities.HoodieMetadataTableValidator - Compare Partitions Failed! AllPartitionPathsFromFS : [20210623/0/20210623, 20210623/0/20210825] and allPartitionPathsMeta : [20210623/0/20210825]
3820 [main] ERROR org.apache.hudi.utilities.HoodieMetadataTableValidator - Metadata table validation failed to HoodieValidationException
org.apache.hudi.exception.HoodieValidationException: Compare Partitions Failed! AllPartitionPathsFromFS : [20210623/0/20210623, 20210623/0/20210825] and allPartitionPathsMeta : [20210623/0/20210825]
at org.apache.hudi.utilities.HoodieMetadataTableValidator.validatePartitions(HoodieMetadataTableValidator.java:435)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doMetadataTableValidation(HoodieMetadataTableValidator.java:388)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doHoodieMetadataTableValidationOnce(HoodieMetadataTableValidator.java:336)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.run(HoodieMetadataTableValidator.java:322)
at MetaDataTable.HoodieMetadataTableValidatorTest.run(HoodieMetadataTableValidatorTest.java:41)
at MetaDataTable.HoodieMetadataTableValidatorTest.main(HoodieMetadataTableValidatorTest.java:11)
Exception in thread "main" org.apache.hudi.exception.HoodieException: Unable to do hoodie metadata table validation in file:///Users/yuezhang/tmp/hudiAfTable/forecast_agg
at org.apache.hudi.utilities.HoodieMetadataTableValidator.run(HoodieMetadataTableValidator.java:325)
at MetaDataTable.HoodieMetadataTableValidatorTest.run(HoodieMetadataTableValidatorTest.java:41)
at MetaDataTable.HoodieMetadataTableValidatorTest.main(HoodieMetadataTableValidatorTest.java:11)
Caused by: org.apache.hudi.exception.HoodieValidationException: Compare Partitions Failed! AllPartitionPathsFromFS : [20210623/0/20210623, 20210623/0/20210825] and allPartitionPathsMeta : [20210623/0/20210825]
at org.apache.hudi.utilities.HoodieMetadataTableValidator.validatePartitions(HoodieMetadataTableValidator.java:435)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doMetadataTableValidation(HoodieMetadataTableValidator.java:388)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doHoodieMetadataTableValidationOnce(HoodieMetadataTableValidator.java:336)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.run(HoodieMetadataTableValidator.java:322)
... 2 more
3824 [Thread-1] INFO org.apache.spark.SparkContext - Invoking stop() from shutdown hook
3832 [Thread-1] INFO org.spark_project.jetty.server.AbstractConnector - Stopped Spark@7e70bd39{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
3834 [Thread-1] INFO org.apache.spark.ui.SparkUI - Stopped Spark web UI at http://localhost:4040
3844 [dispatcher-event-loop-11] INFO org.apache.spark.MapOutputTrackerMasterEndpoint - MapOutputTrackerMasterEndpoint stopped!
3856 [Thread-1] INFO org.apache.spark.storage.memory.MemoryStore - MemoryStore cleared
3856 [Thread-1] INFO org.apache.spark.storage.BlockManager - BlockManager stopped
3858 [Thread-1] INFO org.apache.spark.storage.BlockManagerMaster - BlockManagerMaster stopped
3860 [dispatcher-event-loop-3] INFO org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint - OutputCommitCoordinator stopped!
3869 [Thread-1] INFO org.apache.spark.SparkContext - Successfully stopped SparkContext
3869 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Shutdown hook called
3870 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Deleting directory /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-2b6036a2-c3d4-461b-8ec0-334aa6ea68e8
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1078567412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
}, {
"hash" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7329",
"triggerID" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213)
* aca2c3e03ef2d33d22a3391e0c24f04529cee6c5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7329)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1078567412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
}, {
"hash" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7329",
"triggerID" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213)
* aca2c3e03ef2d33d22a3391e0c24f04529cee6c5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7329)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1075914300
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1075914300
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1075884653
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1078565923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
}, {
"hash" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 32c70db6937f1c7875d50e9d3136adcef89eedc1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213)
* aca2c3e03ef2d33d22a3391e0c24f04529cee6c5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1078707858
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7213",
"triggerID" : "32c70db6937f1c7875d50e9d3136adcef89eedc1",
"triggerType" : "PUSH"
}, {
"hash" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7329",
"triggerID" : "aca2c3e03ef2d33d22a3391e0c24f04529cee6c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aca2c3e03ef2d33d22a3391e0c24f04529cee6c5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7329)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1078725760
This patch is tested on local. Also CI passed. I believe it's ready for review :)
With out this patch, HoodieMetadataTableValidator may fail because of new partition is creating but not committed.
```
3801 [main] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Size of file spilled to disk => 0
3802 [main] INFO org.apache.hudi.metadata.HoodieBackedTableMetadata - Opened 3 metadata log files (dataset instant=20220323111740608, metadata instant=20220323111740608) in 469 ms
3819 [main] INFO org.apache.hudi.metadata.BaseTableMetadata - Listed partitions from metadata: #partitions=1
3819 [main] ERROR org.apache.hudi.utilities.HoodieMetadataTableValidator - Compare Partitions Failed! AllPartitionPathsFromFS : [20210623/0/20210623, 20210623/0/20210825] and allPartitionPathsMeta : [20210623/0/20210825]
3820 [main] ERROR org.apache.hudi.utilities.HoodieMetadataTableValidator - Metadata table validation failed to HoodieValidationException
org.apache.hudi.exception.HoodieValidationException: Compare Partitions Failed! AllPartitionPathsFromFS : [20210623/0/20210623, 20210623/0/20210825] and allPartitionPathsMeta : [20210623/0/20210825]
at org.apache.hudi.utilities.HoodieMetadataTableValidator.validatePartitions(HoodieMetadataTableValidator.java:435)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doMetadataTableValidation(HoodieMetadataTableValidator.java:388)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doHoodieMetadataTableValidationOnce(HoodieMetadataTableValidator.java:336)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.run(HoodieMetadataTableValidator.java:322)
at MetaDataTable.HoodieMetadataTableValidatorTest.run(HoodieMetadataTableValidatorTest.java:41)
at MetaDataTable.HoodieMetadataTableValidatorTest.main(HoodieMetadataTableValidatorTest.java:11)
Exception in thread "main" org.apache.hudi.exception.HoodieException: Unable to do hoodie metadata table validation in file:///Users/yuezhang/tmp/hudiAfTable/forecast_agg
at org.apache.hudi.utilities.HoodieMetadataTableValidator.run(HoodieMetadataTableValidator.java:325)
at MetaDataTable.HoodieMetadataTableValidatorTest.run(HoodieMetadataTableValidatorTest.java:41)
at MetaDataTable.HoodieMetadataTableValidatorTest.main(HoodieMetadataTableValidatorTest.java:11)
Caused by: org.apache.hudi.exception.HoodieValidationException: Compare Partitions Failed! AllPartitionPathsFromFS : [20210623/0/20210623, 20210623/0/20210825] and allPartitionPathsMeta : [20210623/0/20210825]
at org.apache.hudi.utilities.HoodieMetadataTableValidator.validatePartitions(HoodieMetadataTableValidator.java:435)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doMetadataTableValidation(HoodieMetadataTableValidator.java:388)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doHoodieMetadataTableValidationOnce(HoodieMetadataTableValidator.java:336)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.run(HoodieMetadataTableValidator.java:322)
... 2 more
3824 [Thread-1] INFO org.apache.spark.SparkContext - Invoking stop() from shutdown hook
3832 [Thread-1] INFO org.spark_project.jetty.server.AbstractConnector - Stopped Spark@7e70bd39{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
3834 [Thread-1] INFO org.apache.spark.ui.SparkUI - Stopped Spark web UI at http://localhost:4040
3844 [dispatcher-event-loop-11] INFO org.apache.spark.MapOutputTrackerMasterEndpoint - MapOutputTrackerMasterEndpoint stopped!
3856 [Thread-1] INFO org.apache.spark.storage.memory.MemoryStore - MemoryStore cleared
3856 [Thread-1] INFO org.apache.spark.storage.BlockManager - BlockManager stopped
3858 [Thread-1] INFO org.apache.spark.storage.BlockManagerMaster - BlockManagerMaster stopped
3860 [dispatcher-event-loop-3] INFO org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint - OutputCommitCoordinator stopped!
3869 [Thread-1] INFO org.apache.spark.SparkContext - Successfully stopped SparkContext
3869 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Shutdown hook called
3870 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Deleting directory /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-2b6036a2-c3d4-461b-8ec0-334aa6ea68e8
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #5100:
URL: https://github.com/apache/hudi/pull/5100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org