You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "adityaverma1997 (via GitHub)" <gi...@apache.org> on 2023/07/21 11:46:42 UTC
[GitHub] [hudi] adityaverma1997 opened a new issue, #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
adityaverma1997 opened a new issue, #9257:
URL: https://github.com/apache/hudi/issues/9257
**Describe the problem you faced**
Hi team, we are facing an issue where failed clean operation is cleaning up the data files.
We are writing a COW table with insert operation but after running successfully for a day or so it is failing on clustering operation(stacktrace given it separate section below). When debugging we found out that actual issue was with the failed clean operation which was performed 4-5 commits before clustering.
Clean operation was not able to complete and failed but there are two issues here, which are:
1. cleaning failed but data files it was supposed to clean are removed
2. although cleaning failed but spark job succeeded.
Because files were cleaned up(which was not expected) and same files were used in clustering for stitching, job runs into an error state.
Post cleaning there is a successful rollback and failed cleaning operation as well.
Attaching screenshots from hoodie timeline as well where S represents Successful operation while F represents Fail operation:

Cleaner operation screenshot showing failed cleaning:

Screenshot for successful rollback:

**To Reproduce**
Steps to reproduce the behavior:
1. Create a new COW Hudi table with given set of configurations (refer additional context section for the same).
2. Run it on hourly frequency and monitor the same for couple of days.
3. Cleaner will give warning and ultimately fail but spark job will succeed with following error:
```
Failed to perform previous clean operation, instant: [==>20230716130816302__clean__INFLIGHT]
```
The subsequent clustering operation will also fail post that because it will try to stitch the parquet file which was cleaned by the failed clean operation leading to inconsistency.
Please refer to stacktrace section below for both cleaner and clustering stacktrace.
**Expected behavior**
As per the logic, we expected that it shouldn't have cleaned up the data files when cleaner operation have failed, which lead to subsequent clustering failure.
**Environment Description**
* Hudi version : 0.13.0 / 0.13.1
* Spark version : 3.3.2
* EMR version: 6.11.0
* Hive version : N/A
* Hadoop version : N/A
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Hudi configurations to be used:
```
{
"hoodie.database.name": <db_name>,
"hoodie.table.name": <table_name>,
"hoodie.datasource.write.recordkey.field": "id",
"hoodie.datasource.write.precombine.field": "created_dt",
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.hive_sync.enable": "true",
"hoodie.datasource.hive_sync.database": <db_name>,
"hoodie.datasource.hive_sync.table": <table_name>,
"hoodie.datasource.hive_sync.support_timestamp": "true",
"hoodie.datasource.hive_sync.use_jdbc": "false",
"hoodie.datasource.hive_sync.mode": "hms",
"hoodie.datasource.meta_sync.condition.sync": "true",
"hoodie.schema.on.read.enable": "true",
"hoodie.datasource.write.reconcile.schema": "true",
"hoodie.copyonwrite.record.size.estimate": 100,
"hoodie.merge.allow.duplicate.on.inserts": "false",
"hoodie.metadata.enable": "true",
"hoodie.metadata.index.column.stats.enable": "true",
"hoodie.enable.data.skipping": "true",
"hoodie.datasource.write.operation": "insert",
"hoodie.insert.shuffle.parallelism": 0,
"hoodie.datasource.write.partitionpath.field": "created_dt_yyyy_mm_dd_ist",
"hoodie.datasource.hive_sync.partition_fields": "created_dt_yyyy_mm_dd_ist",
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.clean.async": "true",
"hoodie.clean.max.commits": 10,
"hoodie.cleaner.policy": "KEEP_LATEST_COMMITS",
"hoodie.cleaner.commits.retained": 2,
"hoodie.clustering.inline": "true",
"hoodie.clustering.inline.max.commits": 4,
"hoodie.clustering.plan.strategy.sort.columns": "battery_event_type_id",
"hoodie.clustering.plan.partition.filter.mode": "RECENT_DAYS",
"hoodie.clustering.execution.strategy.class": "org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy",
"hoodie.clustering.plan.strategy.target.file.max.bytes": 134217728,
"hoodie.clustering.plan.strategy.small.file.limit": 104857600,
"hoodie.parquet.small.file.limit": 0,
"hoodie.parquet.max.file.size": 134217728,
"hoodie.archive.async": "true",
"hoodie.keep.min.commits": 10,
"hoodie.keep.max.commits": 15,
"hoodie.commits.archival.batch": 15,
"hoodie.archive.merge.enable": "true",
"hoodie.archive.merge.files.batch.size": 15,
"hoodie.archive.merge.small.file.limit.bytes": 20971520,
}
```
**Stacktrace**
Cleaning error stacktrace:
```
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - 23/07/16 14:07:21 WARN CleanActionExecutor: Failed to perform previous clean operation, instant: [==>20230716130816302__clean__INFLIGHT]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - java.lang.IllegalArgumentException: null
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:31) ~[hudi-aws-bundle-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:633) ~[hudi-aws-bundle-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:614) ~[hudi-aws-bundle-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.saveAsComplete(HoodieActiveTimeline.java:223) ~[hudi-aws-bundle-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.commit(BaseSparkCommitActionExecutor.java:305) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.commit(BaseSparkCommitActionExecutor.java:291) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.autoCommit(BaseCommitActionExecutor.java:190) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.commitOnAutoCommit(BaseCommitActionExecutor.java:174) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.updateIndexAndCommitIfNeeded(BaseSparkCommitActionExecutor.java:273) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:178) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.deltacommit.SparkUpsertPreppedDeltaCommitActionExecutor.execute(SparkUpsertPreppedDeltaCommitActionExecutor.java:44) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsertPrepped(HoodieSparkMergeOnReadTable.java:111) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsertPrepped(HoodieSparkMergeOnReadTable.java:80) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.client.SparkRDDWriteClient.upsertPreppedRecords(SparkRDDWriteClient.java:154) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.commit(SparkHoodieBackedTableMetadataWriter.java:186) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:830) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:910) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.BaseActionExecutor.lambda$writeTableMetadata$1(BaseActionExecutor.java:68) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.common.util.Option.ifPresent(Option.java:97) ~[hudi-aws-bundle-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:68) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.clean.CleanActionExecutor.runClean(CleanActionExecutor.java:221) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.clean.CleanActionExecutor.runPendingClean(CleanActionExecutor.java:187) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.clean.CleanActionExecutor.lambda$execute$8(CleanActionExecutor.java:256) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_372]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.action.clean.CleanActionExecutor.execute(CleanActionExecutor.java:250) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.clean(HoodieSparkCopyOnWriteTable.java:263) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.client.BaseHoodieTableServiceClient.clean(BaseHoodieTableServiceClient.java:554) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:765) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:737) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at org.apache.hudi.async.AsyncCleanerService.lambda$startService$0(AsyncCleanerService.java:55) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_372]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_372]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_372]
[2023-07-16, 14:07:21 UTC] {subprocess.py:89} INFO - at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_372]
```
Clustering error stacktrace:
```
[2023-07-16 20:07:27,119] {subprocess.py:89} INFO - : java.util.concurrent.CompletionException: org.apache.spark.sql.AnalysisException: Path does not exist: s3://<bucket_name>/<folder_name>/<table_name>/created_dt_yyyy_mm_dd_ist=2023-07-16/44824999-78d5-4ea7-b5aa-0f8b5df80671-0_0-28-85_20230716040832186.parquet
[2023-07-16 20:07:27,119] {subprocess.py:89} INFO - at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
[2023-07-16 20:07:27,119] {subprocess.py:89} INFO - at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
[2023-07-16 20:07:27,119] {subprocess.py:89} INFO - at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
[2023-07-16 20:07:27,119] {subprocess.py:89} INFO - at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
[2023-07-16 20:07:27,119] {subprocess.py:89} INFO - at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[2023-07-16 20:07:27,119] {subprocess.py:89} INFO - at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[2023-07-16 20:07:27,120] {subprocess.py:89} INFO - at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[2023-07-16 20:07:27,120] {subprocess.py:89} INFO - at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
[2023-07-16 20:07:27,120] {subprocess.py:89} INFO - Caused by: org.apache.spark.sql.AnalysisException: Path does not exist: s3://<bucket_name>/<folder_name>/<table_name>/created_dt_yyyy_mm_dd_ist=2023-07-16/44824999-78d5-4ea7-b5aa-0f8b5df80671-0_0-28-85_20230716040832186.parquet
[2023-07-16 20:07:27,120] {subprocess.py:89} INFO - at org.apache.spark.sql.errors.QueryCompilationErrors$.dataPathNotExistError(QueryCompilationErrors.scala:1016)
[2023-07-16 20:07:27,120] {subprocess.py:89} INFO - at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4(DataSource.scala:785)
[2023-07-16 20:07:27,120] {subprocess.py:89} INFO - at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4$adapted(DataSource.scala:782)
[2023-07-16 20:07:27,120] {subprocess.py:89} INFO - at org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:372)
[2023-07-16 20:07:27,120] {subprocess.py:89} INFO - at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[2023-07-16 20:07:27,120] {subprocess.py:89} INFO - at scala.util.Success.$anonfun$map$1(Try.scala:255)
[2023-07-16 20:07:27,121] {subprocess.py:89} INFO - at scala.util.Success.map(Try.scala:213)
[2023-07-16 20:07:27,121] {subprocess.py:89} INFO - at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[2023-07-16 20:07:27,121] {subprocess.py:89} INFO - at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[2023-07-16 20:07:27,121] {subprocess.py:89} INFO - at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[2023-07-16 20:07:27,121] {subprocess.py:89} INFO - at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[2023-07-16 20:07:27,121] {subprocess.py:89} INFO - at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
[2023-07-16 20:07:27,121] {subprocess.py:89} INFO - ... 4 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-2348820397
I don't quite understand the issue. cleaner will touch the latest file slice. and clustering can only take in latest file slices. So, I am not very sure how this is happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1647721853
cc @ad1happy2go , it is greate if you can help,
@adityaverma1997 Did you try other cleaning strategies instead, like the default cleaning strategy, does it work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] adityaverma1997 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "adityaverma1997 (via GitHub)" <gi...@apache.org>.
adityaverma1997 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1647925326
@danny0405 I haven't tried the other cleaning strategies, not even default one because I don't want to run cleaner after every commit. Also, in my case I want only last 2-3 commits to retain so thatswhy I changed following properties.
```
"hoodie.clean.max.commits": 10,
"hoodie.cleaner.commits.retained": 2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1649244855
> I don't want to run cleaner after every commit.
The clean servoce would be triggered for fixed time interval for default async cleaning, for each cleaning, it would see whether a cleaning plan is generated, if it is, the plan would then be executed.
The less retained commits you configured, the more frequency cleaning action would actually take effect and got executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1655353186
@adityaverma1997 I will try to reproduce this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-4845421636
Closing this out — it was never reproduced (multiple attempts by maintainers), it was on a 0.13.0 vendor build, and the relevant code paths have since been substantially reworked.
On the substance: the failure was a clean that deleted files and then failed in HoodieActiveTimeline.transitionState during its metadata-table update, leaving the clean inflight with files already removed — and since it ran async, the failure didn't fail the job, so a later clustering broke on the missing file.
In 1.x:
- the timeline was rewritten (completion-time / LSM timeline in 1.0), so the transitionState path that threw IllegalArgumentException: null here no longer exists;
- clean and archival are now wrapped in try-catch (HUDI-8804, [#12562](https://github.com/apache/hudi/pull/12562), 1.0.1+), so a clean failure is surfaced rather than silently succeeding;
- clean robustness was further hardened (cap commits per clean, cleanup/rollback policy before writes, incremental clean-planning fixes), with the clean/archival interplay simplification ongoing ([#17279]).
Given it wasn't reproducible and the clean + timeline code has been reworked, I'd recommend retrying on a recent open-source 1.x release. If you can reproduce a failed clean that deletes files (with the job still succeeding) on 1.x, please reopen with the .hoodie/ timeline around the failure and the clean plan, and we'll dig in. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1652978783
You are right, and to supplement, the each file group, we keep at least the latest file slice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
URL: https://github.com/apache/hudi/issues/9257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1778596706
@adityaverma1997 Sorry for all the delay's here. I did try to reproduce this couple of times but never got any error. Tried to mock up some failures too when cleaning happens. Actually it depends on when the cleaning exactly fails. Are you able to reproduce this consistently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] adityaverma1997 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "adityaverma1997 (via GitHub)" <gi...@apache.org>.
adityaverma1997 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1647320952
Thanks for your reply Danny.
I was also considering the same logic for cleaning and clustering but something is going wrong it seems, hope so my configurations are well in place and I'm not messing up with them.
Attaching the sheet containing plans for both cleaning and clustering instances which got failed.
[HudiPlanRequestedDetails.xlsx](https://github.com/apache/hudi/files/12142635/HudiPlanRequestedDetails.xlsx)
Hope it works, let me know if something else required from my end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] adityaverma1997 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "adityaverma1997 (via GitHub)" <gi...@apache.org>.
adityaverma1997 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1649440527
Correct me if I am wrong here, though I am running async cleaning but cleaning frequency is controlled by the following hudi configuration, which is:
```
hoodie.clean.max.commits
```
which is set as 10 in my case, so cleaner will get scheduled and executed after every 10th commit.
On the other hand, we can retain no of commits when cleaning is executed based on below configuration:
```
hoodie.cleaner.commits.retained
```
I have set it as 2, so it will retain latest 2 commits and clean remaining commits on every cleaning execution.
Looking forward for your reply @danny0405 and @ad1happy2go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1708759312
@adityaverma1997 Sorry for the delay, I couldn't got time to work on this. Will be prioritising this and get back to you in couple of days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning [hudi]
Posted by "adityaverma1997 (via GitHub)" <gi...@apache.org>.
adityaverma1997 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1752778848
@ad1happy2go Any update on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] adityaverma1997 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "adityaverma1997 (via GitHub)" <gi...@apache.org>.
adityaverma1997 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1653270340
Thanks for confirming the same Danny.
Have we found any reason behind this cleaner failure issue, would be really helpful if we can resolve this as many of our tables will be running on same set of configurations. Any help would be really appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1653544205
I might have no enough time for this, wondering whether @ad1happy2go could have some time to reproduce and dig into the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1647115694
I see the cleaning retain commits is configured as 2: `"hoodie.cleaner.commits.retained": 2`
Theoretically, the cleaner would always keep at least 1 valid file silce for each file group, the clustering should always replace the latest file slice with new files.
Maybe you can also give more details about the clustering and cleaning plan, they are stored in the `xxx.requested` file which is in Avro bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] adityaverma1997 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning
Posted by "adityaverma1997 (via GitHub)" <gi...@apache.org>.
adityaverma1997 commented on issue #9257:
URL: https://github.com/apache/hudi/issues/9257#issuecomment-1702471850
Hi @ad1happy2go , do you get chance to check out on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org