You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/25 07:15:22 UTC
[GitHub] [hudi] zhangyue19921010 commented on pull request #5100: [HUDI-3635]Fix HoodieMetadataTableValidator around comparison of partition listing
zhangyue19921010 commented on pull request #5100:
URL: https://github.com/apache/hudi/pull/5100#issuecomment-1078725760
This patch is tested on local. Also CI passed. I believe it's ready for review :)
With out this patch, HoodieMetadataTableValidator may fail because of new partition is creating but not committed.
```
3801 [main] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Size of file spilled to disk => 0
3802 [main] INFO org.apache.hudi.metadata.HoodieBackedTableMetadata - Opened 3 metadata log files (dataset instant=20220323111740608, metadata instant=20220323111740608) in 469 ms
3819 [main] INFO org.apache.hudi.metadata.BaseTableMetadata - Listed partitions from metadata: #partitions=1
3819 [main] ERROR org.apache.hudi.utilities.HoodieMetadataTableValidator - Compare Partitions Failed! AllPartitionPathsFromFS : [20210623/0/20210623, 20210623/0/20210825] and allPartitionPathsMeta : [20210623/0/20210825]
3820 [main] ERROR org.apache.hudi.utilities.HoodieMetadataTableValidator - Metadata table validation failed to HoodieValidationException
org.apache.hudi.exception.HoodieValidationException: Compare Partitions Failed! AllPartitionPathsFromFS : [20210623/0/20210623, 20210623/0/20210825] and allPartitionPathsMeta : [20210623/0/20210825]
at org.apache.hudi.utilities.HoodieMetadataTableValidator.validatePartitions(HoodieMetadataTableValidator.java:435)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doMetadataTableValidation(HoodieMetadataTableValidator.java:388)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doHoodieMetadataTableValidationOnce(HoodieMetadataTableValidator.java:336)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.run(HoodieMetadataTableValidator.java:322)
at MetaDataTable.HoodieMetadataTableValidatorTest.run(HoodieMetadataTableValidatorTest.java:41)
at MetaDataTable.HoodieMetadataTableValidatorTest.main(HoodieMetadataTableValidatorTest.java:11)
Exception in thread "main" org.apache.hudi.exception.HoodieException: Unable to do hoodie metadata table validation in file:///Users/yuezhang/tmp/hudiAfTable/forecast_agg
at org.apache.hudi.utilities.HoodieMetadataTableValidator.run(HoodieMetadataTableValidator.java:325)
at MetaDataTable.HoodieMetadataTableValidatorTest.run(HoodieMetadataTableValidatorTest.java:41)
at MetaDataTable.HoodieMetadataTableValidatorTest.main(HoodieMetadataTableValidatorTest.java:11)
Caused by: org.apache.hudi.exception.HoodieValidationException: Compare Partitions Failed! AllPartitionPathsFromFS : [20210623/0/20210623, 20210623/0/20210825] and allPartitionPathsMeta : [20210623/0/20210825]
at org.apache.hudi.utilities.HoodieMetadataTableValidator.validatePartitions(HoodieMetadataTableValidator.java:435)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doMetadataTableValidation(HoodieMetadataTableValidator.java:388)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.doHoodieMetadataTableValidationOnce(HoodieMetadataTableValidator.java:336)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.run(HoodieMetadataTableValidator.java:322)
... 2 more
3824 [Thread-1] INFO org.apache.spark.SparkContext - Invoking stop() from shutdown hook
3832 [Thread-1] INFO org.spark_project.jetty.server.AbstractConnector - Stopped Spark@7e70bd39{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
3834 [Thread-1] INFO org.apache.spark.ui.SparkUI - Stopped Spark web UI at http://localhost:4040
3844 [dispatcher-event-loop-11] INFO org.apache.spark.MapOutputTrackerMasterEndpoint - MapOutputTrackerMasterEndpoint stopped!
3856 [Thread-1] INFO org.apache.spark.storage.memory.MemoryStore - MemoryStore cleared
3856 [Thread-1] INFO org.apache.spark.storage.BlockManager - BlockManager stopped
3858 [Thread-1] INFO org.apache.spark.storage.BlockManagerMaster - BlockManagerMaster stopped
3860 [dispatcher-event-loop-3] INFO org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint - OutputCommitCoordinator stopped!
3869 [Thread-1] INFO org.apache.spark.SparkContext - Successfully stopped SparkContext
3869 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Shutdown hook called
3870 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Deleting directory /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-2b6036a2-c3d4-461b-8ec0-334aa6ea68e8
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org