You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/09 06:04:03 UTC
[GitHub] [hudi] zhangyue19921010 opened a new pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
zhangyue19921010 opened a new pull request #4994:
URL: https://github.com/apache/hudi/pull/4994
https://issues.apache.org/jira/browse/HUDI-3376
## What is the purpose of the pull request
add an option named `--skip-under-deletion-data-files` which will skip to compare the data files which are under deletion by cleaner
## Brief change log
When enabled, HoodieMetadataTableValidator will read the clean plan for pending cleaning action and get under deletion data files. Then hoodie will skip to compare these files.
This patch is tested on our local env.
There 're pending cleaning actions during performing HoodieMetadataTableValidator.
Before this patch validator failed with
```
4257 [Executor task launch worker for task 1] INFO org.apache.hudi.utilities.HoodieMetadataTableValidator - Validation of getLatestBaseFiles succeeded for partition 20210623
4258 [Executor task launch worker for task 1] ERROR org.apache.hudi.utilities.HoodieMetadataTableValidator - Validation of file slices for partition 20210623 failed.
file slices from metadata: [FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309114505047, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_0-11-0_20220309114505047.parquet, fileLen=14203377, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309124131821, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_1-25-1611_20220309124131821.parquet, fileLen=13973255, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309125005608, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/
hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_1-25-1611_20220309125005608.parquet, fileLen=13975017, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309114505047, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_1-12-0_20220309114505047.parquet, fileLen=14121559, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309124131821, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_0-20-1610_20220309124131821.parquet, fileLen=13895052, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fil
eId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309125005608, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_0-20-1610_20220309125005608.parquet, fileLen=13896710, BootstrapBaseFile=null}', logFiles='[]'}]
file slices from file system and base files: [FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309114505047, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_0-11-0_20220309114505047.parquet, fileLen=14203377, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309124131821, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_1-25-1611_20220309124131821.parquet, fileLen=13973255, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309125005608, baseFile='HoodieBaseFile{fullPath=file:/U
sers/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_1-25-1611_20220309125005608.parquet, fileLen=13975017, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309124131821, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_0-20-1610_20220309124131821.parquet, fileLen=13895052, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309125005608, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_0-20-1610_20220309125005608.parquet, fileLen=13896710, BootstrapBaseFile=null}', logFiles='[]'}]
4258 [Executor task launch worker for task 1] ERROR org.apache.hudi.utilities.HoodieMetadataTableValidator - Metadata table validation failed for 20210623 due to HoodieValidationException
org.apache.hudi.exception.HoodieValidationException: Validation of file slices for partition 20210623 failed.
file slices from metadata: [FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309114505047, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_0-11-0_20220309114505047.parquet, fileLen=14203377, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309124131821, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_1-25-1611_20220309124131821.parquet, fileLen=13973255, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309125005608, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/
hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_1-25-1611_20220309125005608.parquet, fileLen=13975017, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309114505047, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_1-12-0_20220309114505047.parquet, fileLen=14121559, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309124131821, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_0-20-1610_20220309124131821.parquet, fileLen=13895052, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fil
eId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309125005608, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_0-20-1610_20220309125005608.parquet, fileLen=13896710, BootstrapBaseFile=null}', logFiles='[]'}]
file slices from file system and base files: [FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309114505047, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_0-11-0_20220309114505047.parquet, fileLen=14203377, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309124131821, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_1-25-1611_20220309124131821.parquet, fileLen=13973255, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='347bff6c-26bc-4577-ad3d-482ba73453ea-0'}, baseCommitTime=20220309125005608, baseFile='HoodieBaseFile{fullPath=file:/U
sers/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/347bff6c-26bc-4577-ad3d-482ba73453ea-0_1-25-1611_20220309125005608.parquet, fileLen=13975017, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309124131821, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_0-20-1610_20220309124131821.parquet, fileLen=13895052, BootstrapBaseFile=null}', logFiles='[]'}, FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='20210623', fileId='a149d7a0-1377-4694-9f95-c98486d6947c-0'}, baseCommitTime=20220309125005608, baseFile='HoodieBaseFile{fullPath=file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/20210623/a149d7a0-1377-4694-9f95-c98486d6947c-0_0-20-1610_20220309125005608.parquet, fileLen=13896710, BootstrapBaseFile=null}', logFiles='[]'}]
at org.apache.hudi.utilities.HoodieMetadataTableValidator.validate(HoodieMetadataTableValidator.java:660)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.validateAllFileGroups(HoodieMetadataTableValidator.java:522)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.validateFilesInPartition(HoodieMetadataTableValidator.java:476)
at org.apache.hudi.utilities.HoodieMetadataTableValidator.lambda$doMetadataTableValidation$6f568e7e$1(HoodieMetadataTableValidator.java:405)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:308)
at scala.collection.AbstractIterator.to(Iterator.scala:1194)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:300)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1194)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:287)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1194)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4265 [Executor task launch worker for task 1] INFO org.apache.spark.executor.Executor - Finished task 0.0 in stage 1.0 (TID 1). 712 bytes result sent to driver
4271 [task-result-getter-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished task 0.0 in stage 1.0 (TID 1) in 132 ms on localhost (executor driver) (1/1)
4271 [task-result-getter-1] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Removed TaskSet 1.0, whose tasks have all completed, from pool
4272 [dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - ResultStage 1 (collect at HoodieJavaRDD.java:140) finished in 0.216 s
4272 [main] INFO org.apache.spark.scheduler.DAGScheduler - Job 1 finished: collect at HoodieJavaRDD.java:140, took 0.218763 s
4272 [main] WARN org.apache.hudi.utilities.HoodieMetadataTableValidator - Metadata table validation failed.
4274 [Thread-1] INFO org.apache.spark.SparkContext - Invoking stop() from shutdown hook
4281 [Thread-1] INFO org.spark_project.jetty.server.AbstractConnector - Stopped Spark@43ed0ff3{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
4282 [Thread-1] INFO org.apache.spark.ui.SparkUI - Stopped Spark web UI at http://localhost:4040
4288 [dispatcher-event-loop-1] INFO org.apache.spark.MapOutputTrackerMasterEndpoint - MapOutputTrackerMasterEndpoint stopped!
4298 [Thread-1] INFO org.apache.spark.storage.memory.MemoryStore - MemoryStore cleared
4298 [Thread-1] INFO org.apache.spark.storage.BlockManager - BlockManager stopped
4299 [Thread-1] INFO org.apache.spark.storage.BlockManagerMaster - BlockManagerMaster stopped
4300 [dispatcher-event-loop-6] INFO org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint - OutputCommitCoordinator stopped!
4307 [Thread-1] INFO org.apache.spark.SparkContext - Successfully stopped SparkContext
4307 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Shutdown hook called
4308 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Deleting directory /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-97af2b76-9039-45e2-98b4-52400a3a1ecc
```
After this patch
```
12543 [Executor task launch worker for task 1] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz]
12548 [Executor task launch worker for task 1] INFO org.apache.hudi.metadata.BaseTableMetadata - Listed file in partition from metadata: partition=20210623, #files=6
12549 [Executor task launch worker for task 1] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=6, NumFileGroups=2, FileGroupsCreationTime=1, StoreTimeTaken=0
18533 [Executor task launch worker for task 1] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file system view for partition (20210623)
18536 [Executor task launch worker for task 1] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=6, NumFileGroups=2, FileGroupsCreationTime=1, StoreTimeTaken=0
18536 [Executor task launch worker for task 1] INFO org.apache.hudi.utilities.HoodieMetadataTableValidator - Validation of file slices succeeded for partition 20210623
18536 [Executor task launch worker for task 1] INFO org.apache.hudi.utilities.HoodieMetadataTableValidator - Validation of getLatestFileSlices succeeded for partition 20210623
18542 [Executor task launch worker for task 1] INFO org.apache.hudi.utilities.HoodieMetadataTableValidator - Validation of getLatestBaseFiles succeeded for partition 20210623
21672 [Executor task launch worker for task 1] INFO org.apache.hudi.utilities.HoodieMetadataTableValidator - Validation of file slices succeeded for partition 20210623
21672 [Executor task launch worker for task 1] INFO org.apache.hudi.utilities.HoodieMetadataTableValidator - Validation of all file groups succeeded for partition 20210623
21672 [Executor task launch worker for task 1] INFO org.apache.hudi.utilities.HoodieMetadataTableValidator - Metadata table validation succeeded for 20210623
21684 [Executor task launch worker for task 1] INFO org.apache.spark.executor.Executor - Finished task 0.0 in stage 1.0 (TID 1). 669 bytes result sent to driver
21696 [task-result-getter-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished task 0.0 in stage 1.0 (TID 1) in 16418 ms on localhost (executor driver) (1/1)
21696 [task-result-getter-1] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Removed TaskSet 1.0, whose tasks have all completed, from pool
21697 [dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - ResultStage 1 (collect at HoodieJavaRDD.java:140) finished in 16.501 s
21697 [main] INFO org.apache.spark.scheduler.DAGScheduler - Job 1 finished: collect at HoodieJavaRDD.java:140, took 16.503945 s
21698 [main] INFO org.apache.hudi.utilities.HoodieMetadataTableValidator - Metadata table validation succeeded.
```
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063670973
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759)
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063160537
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#discussion_r825337535
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -173,6 +181,9 @@ private TypedProperties readConfigFromFileSystem(JavaSparkContext jsc, Config cf
+ "Can use --min-validate-interval-seconds to control validation frequency", required = false)
public boolean continuous = false;
+ @Parameter(names = {"--skip-under-deletion-data-files"}, description = "Skip to compare the data files which are under deletion by cleaner", required = false)
Review comment:
nit: `--skip-under-deletion-data-files` -> `--skip-data-files-for-cleaning`
`skipUnderDeletionDataFiles` -> `skipDataFilesForCleaning`
using `deletion` here may confuse the user with `DELETE` operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#discussion_r825658841
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+ List<String> baseFilesUnderDeletion = Collections.emptyList();
+
+ if (cfg.skipUnderDeletionDataFiles) {
+ HoodieTimeline pendingCleaningTimeline = metaClient.getActiveTimeline()
+ .getCleanerTimeline()
+ .filter(instant -> instant.getState() != HoodieInstant.State.COMPLETED);
Review comment:
Nice call. Yeap we only need to take care of inflight cleaning.
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+ List<String> baseFilesUnderDeletion = Collections.emptyList();
+
+ if (cfg.skipUnderDeletionDataFiles) {
+ HoodieTimeline pendingCleaningTimeline = metaClient.getActiveTimeline()
+ .getCleanerTimeline()
+ .filter(instant -> instant.getState() != HoodieInstant.State.COMPLETED);
+
+ baseFilesUnderDeletion = pendingCleaningTimeline.getInstants().flatMap(instant -> {
+ try {
+ if (instant.isInflight()) {
+ // convert inflight instant to requested and get clean plan
+ instant = new HoodieInstant(HoodieInstant.State.REQUESTED, instant.getAction(), instant.getTimestamp());
+ }
+ HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(metaClient, instant);
+
+ return cleanerPlan.getFilePathsToBeDeletedPerPartition().values().stream().flatMap(cleanerFIleInfoList -> {
Review comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062584363
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 922548761a98dfd95cfc47b25f915d8a5506f99c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063713204
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063160537
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063177489
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063714607
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062585804
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 922548761a98dfd95cfc47b25f915d8a5506f99c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721)
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063713675
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
yihua merged pull request #4994:
URL: https://github.com/apache/hudi/pull/4994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#discussion_r825337535
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -173,6 +181,9 @@ private TypedProperties readConfigFromFileSystem(JavaSparkContext jsc, Config cf
+ "Can use --min-validate-interval-seconds to control validation frequency", required = false)
public boolean continuous = false;
+ @Parameter(names = {"--skip-under-deletion-data-files"}, description = "Skip to compare the data files which are under deletion by cleaner", required = false)
Review comment:
nit: `--skip-under-deletion-data-files` -> `--skip-data-files-under-deletion`
`skipUnderDeletionDataFiles` -> `skipDataFilesUnderDeletion`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063177489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063609257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1067505322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916",
"triggerID" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"triggerType" : "PUSH"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6947",
"triggerID" : "1067472160",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 56940f0a0ccd7723120611927f24e6022e86cd28 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6947)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063571155
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063714607
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063571155
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#discussion_r825338047
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+ List<String> baseFilesUnderDeletion = Collections.emptyList();
+
+ if (cfg.skipUnderDeletionDataFiles) {
+ HoodieTimeline pendingCleaningTimeline = metaClient.getActiveTimeline()
+ .getCleanerTimeline()
+ .filter(instant -> instant.getState() != HoodieInstant.State.COMPLETED);
Review comment:
`filterInflights()` or `filterInflightsAndRequested()` can be used here. Also, should inflight cleaning instants be considered only? Requested cleaning instants should not affect the validation.
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -495,11 +592,31 @@ private void validateLatestFileSlices(
LOG.info("Validation of getLatestFileSlices succeeded for partition " + partitionPath);
}
+ private List<FileSlice> filterFileSliceBasedOnUnderDeletionFiles(List<FileSlice> sortedLatestFileSliceList, List<String> baseFilesUnderDeletion) {
Review comment:
Could `baseFilesUnderDeletion` be a `Set<String>` to speed up lookup?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -410,42 +466,64 @@ public void doMetadataTableValidation() {
* @param metadataTableBasedContext Validation context containing information based on metadata table
* @param fsBasedContext Validation context containing information based on the file system
* @param partitionPath Partition path String
+ * @param baseFilesUnderDeletion Base files under un-complete cleaner action
*/
private void validateFilesInPartition(
HoodieMetadataValidationContext metadataTableBasedContext,
- HoodieMetadataValidationContext fsBasedContext, String partitionPath) {
+ HoodieMetadataValidationContext fsBasedContext, String partitionPath,
+ List<String> baseFilesUnderDeletion) {
if (cfg.validateLatestFileSlices) {
- validateLatestFileSlices(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateLatestFileSlices(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
if (cfg.validateLatestBaseFiles) {
- validateLatestBaseFiles(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateLatestBaseFiles(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
if (cfg.validateAllFileGroups) {
- validateAllFileGroups(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateAllFileGroups(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
if (cfg.validateAllColumnStats) {
- validateAllColumnStats(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateAllColumnStats(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
if (cfg.validateBloomFilters) {
- validateBloomFilters(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateBloomFilters(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
}
private void validateAllFileGroups(
HoodieMetadataValidationContext metadataTableBasedContext,
- HoodieMetadataValidationContext fsBasedContext, String partitionPath) {
- List<FileSlice> allFileSlicesFromMeta = metadataTableBasedContext
- .getSortedAllFileGroupList(partitionPath).stream()
- .flatMap(HoodieFileGroup::getAllFileSlices).sorted(new FileSliceComparator())
- .collect(Collectors.toList());
- List<FileSlice> allFileSlicesFromFS = fsBasedContext
- .getSortedAllFileGroupList(partitionPath).stream()
- .flatMap(HoodieFileGroup::getAllFileSlices).sorted(new FileSliceComparator())
- .collect(Collectors.toList());
+ HoodieMetadataValidationContext fsBasedContext,
+ String partitionPath,
+ List<String> baseFilesUnderDeletion) {
Review comment:
similar here for variable naming.
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+ List<String> baseFilesUnderDeletion = Collections.emptyList();
Review comment:
nit: similar here for variable naming
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+ List<String> baseFilesUnderDeletion = Collections.emptyList();
+
+ if (cfg.skipUnderDeletionDataFiles) {
+ HoodieTimeline pendingCleaningTimeline = metaClient.getActiveTimeline()
+ .getCleanerTimeline()
+ .filter(instant -> instant.getState() != HoodieInstant.State.COMPLETED);
+
+ baseFilesUnderDeletion = pendingCleaningTimeline.getInstants().flatMap(instant -> {
+ try {
+ if (instant.isInflight()) {
+ // convert inflight instant to requested and get clean plan
+ instant = new HoodieInstant(HoodieInstant.State.REQUESTED, instant.getAction(), instant.getTimestamp());
+ }
+ HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(metaClient, instant);
+
+ return cleanerPlan.getFilePathsToBeDeletedPerPartition().values().stream().flatMap(cleanerFIleInfoList -> {
+ return cleanerFIleInfoList.stream().map(fileInfo -> {
+ return new Path(fileInfo.getFilePath()).getName();
+ });
+ });
+
+ } catch (HoodieIOException ex) {
+
+ if (ex.getIOException() instanceof FileNotFoundException) {
+ // cleaner instant could be deleted by archive and FileNotFoundException could be threw during getInstantDetails function
Review comment:
Inflight cleaning instants shouldn't be archived. Is this considering the inflight cleaning instant deleted due to failed cleaning?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+ List<String> baseFilesUnderDeletion = Collections.emptyList();
+
+ if (cfg.skipUnderDeletionDataFiles) {
+ HoodieTimeline pendingCleaningTimeline = metaClient.getActiveTimeline()
+ .getCleanerTimeline()
+ .filter(instant -> instant.getState() != HoodieInstant.State.COMPLETED);
+
+ baseFilesUnderDeletion = pendingCleaningTimeline.getInstants().flatMap(instant -> {
+ try {
+ if (instant.isInflight()) {
+ // convert inflight instant to requested and get clean plan
+ instant = new HoodieInstant(HoodieInstant.State.REQUESTED, instant.getAction(), instant.getTimestamp());
+ }
+ HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(metaClient, instant);
+
+ return cleanerPlan.getFilePathsToBeDeletedPerPartition().values().stream().flatMap(cleanerFIleInfoList -> {
Review comment:
nit: `cleanerFIleInfoList` -> `cleanerFileInfoList`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#discussion_r825659273
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+ List<String> baseFilesUnderDeletion = Collections.emptyList();
+
+ if (cfg.skipUnderDeletionDataFiles) {
+ HoodieTimeline pendingCleaningTimeline = metaClient.getActiveTimeline()
+ .getCleanerTimeline()
+ .filter(instant -> instant.getState() != HoodieInstant.State.COMPLETED);
+
+ baseFilesUnderDeletion = pendingCleaningTimeline.getInstants().flatMap(instant -> {
+ try {
+ if (instant.isInflight()) {
+ // convert inflight instant to requested and get clean plan
+ instant = new HoodieInstant(HoodieInstant.State.REQUESTED, instant.getAction(), instant.getTimestamp());
+ }
+ HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(metaClient, instant);
+
+ return cleanerPlan.getFilePathsToBeDeletedPerPartition().values().stream().flatMap(cleanerFIleInfoList -> {
+ return cleanerFIleInfoList.stream().map(fileInfo -> {
+ return new Path(fileInfo.getFilePath()).getName();
+ });
+ });
+
+ } catch (HoodieIOException ex) {
+
+ if (ex.getIOException() instanceof FileNotFoundException) {
+ // cleaner instant could be deleted by archive and FileNotFoundException could be threw during getInstantDetails function
Review comment:
Actually no need to check FileNotFoundException here. removed.
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -410,42 +466,64 @@ public void doMetadataTableValidation() {
* @param metadataTableBasedContext Validation context containing information based on metadata table
* @param fsBasedContext Validation context containing information based on the file system
* @param partitionPath Partition path String
+ * @param baseFilesUnderDeletion Base files under un-complete cleaner action
*/
private void validateFilesInPartition(
HoodieMetadataValidationContext metadataTableBasedContext,
- HoodieMetadataValidationContext fsBasedContext, String partitionPath) {
+ HoodieMetadataValidationContext fsBasedContext, String partitionPath,
+ List<String> baseFilesUnderDeletion) {
if (cfg.validateLatestFileSlices) {
- validateLatestFileSlices(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateLatestFileSlices(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
if (cfg.validateLatestBaseFiles) {
- validateLatestBaseFiles(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateLatestBaseFiles(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
if (cfg.validateAllFileGroups) {
- validateAllFileGroups(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateAllFileGroups(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
if (cfg.validateAllColumnStats) {
- validateAllColumnStats(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateAllColumnStats(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
if (cfg.validateBloomFilters) {
- validateBloomFilters(metadataTableBasedContext, fsBasedContext, partitionPath);
+ validateBloomFilters(metadataTableBasedContext, fsBasedContext, partitionPath, baseFilesUnderDeletion);
}
}
private void validateAllFileGroups(
HoodieMetadataValidationContext metadataTableBasedContext,
- HoodieMetadataValidationContext fsBasedContext, String partitionPath) {
- List<FileSlice> allFileSlicesFromMeta = metadataTableBasedContext
- .getSortedAllFileGroupList(partitionPath).stream()
- .flatMap(HoodieFileGroup::getAllFileSlices).sorted(new FileSliceComparator())
- .collect(Collectors.toList());
- List<FileSlice> allFileSlicesFromFS = fsBasedContext
- .getSortedAllFileGroupList(partitionPath).stream()
- .flatMap(HoodieFileGroup::getAllFileSlices).sorted(new FileSliceComparator())
- .collect(Collectors.toList());
+ HoodieMetadataValidationContext fsBasedContext,
+ String partitionPath,
+ List<String> baseFilesUnderDeletion) {
Review comment:
chanegd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063570511
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063242221
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063713204
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1066472374
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916",
"triggerID" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773)
* 56940f0a0ccd7723120611927f24e6022e86cd28 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#discussion_r828610748
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+ List<String> baseFilesUnderDeletion = Collections.emptyList();
+
+ if (cfg.skipUnderDeletionDataFiles) {
+ HoodieTimeline pendingCleaningTimeline = metaClient.getActiveTimeline()
+ .getCleanerTimeline()
+ .filter(instant -> instant.getState() != HoodieInstant.State.COMPLETED);
Review comment:
Sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062585804
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 922548761a98dfd95cfc47b25f915d8a5506f99c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721)
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062617258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 922548761a98dfd95cfc47b25f915d8a5506f99c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721)
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062666486
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062666486
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063670973
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759)
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063800528
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1067472160
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063669704
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759)
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063180154
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062587258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 922548761a98dfd95cfc47b25f915d8a5506f99c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721)
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1066470660
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773)
* 56940f0a0ccd7723120611927f24e6022e86cd28 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1067472693
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916",
"triggerID" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"triggerType" : "PUSH"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6947",
"triggerID" : "1067472160",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 56940f0a0ccd7723120611927f24e6022e86cd28 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6947)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1066583489
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916",
"triggerID" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 56940f0a0ccd7723120611927f24e6022e86cd28 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063030013
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063242221
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063669704
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759)
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063800528
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063030013
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1066470660
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773)
* 56940f0a0ccd7723120611927f24e6022e86cd28 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1066472374
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916",
"triggerID" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c6b5940da7efcb86bba53117a70b3a87ab90fa6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773)
* 56940f0a0ccd7723120611927f24e6022e86cd28 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#discussion_r825658558
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -173,6 +181,9 @@ private TypedProperties readConfigFromFileSystem(JavaSparkContext jsc, Config cf
+ "Can use --min-validate-interval-seconds to control validation frequency", required = false)
public boolean continuous = false;
+ @Parameter(names = {"--skip-under-deletion-data-files"}, description = "Skip to compare the data files which are under deletion by cleaner", required = false)
Review comment:
Changed.
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+ List<String> baseFilesUnderDeletion = Collections.emptyList();
Review comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063028512
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062587258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 922548761a98dfd95cfc47b25f915d8a5506f99c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721)
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062584363
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 922548761a98dfd95cfc47b25f915d8a5506f99c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1062617258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 922548761a98dfd95cfc47b25f915d8a5506f99c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721)
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063180154
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063609257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 95d3fa063aa4911efde9d99272d2ea7ca96aaa0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1063900982
Azure Passed. Hi @yihua really appreciate it if you could help me review this patch :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1067472693
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916",
"triggerID" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"triggerType" : "PUSH"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6947",
"triggerID" : "1067472160",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 56940f0a0ccd7723120611927f24e6022e86cd28 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6947)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#discussion_r825659699
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##########
@@ -495,11 +592,31 @@ private void validateLatestFileSlices(
LOG.info("Validation of getLatestFileSlices succeeded for partition " + partitionPath);
}
+ private List<FileSlice> filterFileSliceBasedOnUnderDeletionFiles(List<FileSlice> sortedLatestFileSliceList, List<String> baseFilesUnderDeletion) {
Review comment:
Make sense! changed. Thanks a lot for your reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#issuecomment-1066583489
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6721",
"triggerID" : "922548761a98dfd95cfc47b25f915d8a5506f99c",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6726",
"triggerID" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"triggerType" : "PUSH"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6749",
"triggerID" : "1063028512",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6752",
"triggerID" : "1063177489",
"triggerType" : "MANUAL"
}, {
"hash" : "95d3fa063aa4911efde9d99272d2ea7ca96aaa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6759",
"triggerID" : "1063570511",
"triggerType" : "MANUAL"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6768",
"triggerID" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"triggerType" : "PUSH"
}, {
"hash" : "9c6b5940da7efcb86bba53117a70b3a87ab90fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6773",
"triggerID" : "1063713675",
"triggerType" : "MANUAL"
}, {
"hash" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916",
"triggerID" : "56940f0a0ccd7723120611927f24e6022e86cd28",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 56940f0a0ccd7723120611927f24e6022e86cd28 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6916)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org