You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/16 05:53:05 UTC
[GitHub] [hudi] BalaMahesh opened a new issue, #5591: [SUPPORT] Hudi Error rolling back using marker files
BalaMahesh opened a new issue, #5591:
URL: https://github.com/apache/hudi/issues/5591
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
We have recently upgraded hudi to 0.11.0 and application has been running good for last few days. Then all of sudden due to some reason, marker files in .hoodie/.temp directory are not created with .marker extension and the application while trying to roll back this commit, validation of marker files is failing and application is getting stopped. I have attached the screenshot of marker files created inside the folder.
Steps to reproduce the behavior:
1. Start the application and keep it running.
2. We don't know exactly what has stopped the application, but it started trying to rollback the failed commit and it is failing to do that.
**Expected behavior**
Run and rollback the failed commits if application has failed.
**Environment Description**
* Hudi version : 0.11.0
* Spark version : 3.2.1
* Hive version : 2.1.3
* Hadoop version : 3.x.x
* Storage (HDFS/S3/GCS..) : GCS
* Running on Docker? (yes/no) : yes(k8's).
**Additional context**
**Stacktrace**
```
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback gs://xxx/hudi/xxxx/ commits 20220514180424825
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:103)
at org.apache.hudi.async.AsyncCleanerService.waitForCompletion(AsyncCleanerService.java:75)
... 11 more
Caused by: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback gs://xxx/hudi/xxx/ commits 20220514180424825
at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:783)
at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1193)
at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1176)
at org.apache.hudi.client.BaseHoodieWriteClient.lambda$clean$33796fd2$1(BaseHoodieWriteClient.java:856)
at org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:142)
at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:855)
at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:825)
at org.apache.hudi.async.AsyncCleanerService.lambda$startService$0(AsyncCleanerService.java:55)
... 4 more
Caused by: org.apache.hudi.exception.HoodieRollbackException: Error rolling back using marker files written for [==>20220514180424825__commit__INFLIGHT]
at org.apache.hudi.table.action.rollback.MarkerBasedRollbackStrategy.getRollbackRequests(MarkerBasedRollbackStrategy.java:103)
at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.requestRollback(BaseRollbackPlanActionExecutor.java:109)
at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.execute(BaseRollbackPlanActionExecutor.java:132)
at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.scheduleRollback(HoodieSparkCopyOnWriteTable.java:212)
at org.apache.hudi.client.BaseHoodieWriteClient.lambda$rollback$6(BaseHoodieWriteClient.java:757)
at org.apache.hudi.common.util.Option.orElseGet(Option.java:142)
at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:757)
... 11 more
Caused by: java.lang.IllegalArgumentException
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:31)
at org.apache.hudi.common.util.MarkerUtils.stripMarkerFolderPrefix(MarkerUtils.java:67)
at org.apache.hudi.table.marker.DirectWriteMarkers.lambda$allMarkerFilePaths$0(DirectWriteMarkers.java:136)
at org.apache.hudi.common.fs.FSUtils.processFiles(FSUtils.java:277)
at org.apache.hudi.table.marker.DirectWriteMarkers.allMarkerFilePaths(DirectWriteMarkers.java:135)
at org.apache.hudi.table.marker.MarkerBasedRollbackUtils.getAllMarkerPaths(MarkerBasedRollbackUtils.java:62)
at org.apache.hudi.table.action.rollback.MarkerBasedRollbackStrategy.getRollbackRequests(MarkerBasedRollbackStrategy.java:76)
... 17 more
```
<img width="1360" alt="Screenshot 2022-05-16 at 11 01 13 AM" src="https://user-images.githubusercontent.com/25053668/168527450-2acf3a5e-a34b-49ca-a6be-d5fa78d69d09.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5591: [SUPPORT] Hudi Error rolling back using marker files
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5591:
URL: https://github.com/apache/hudi/issues/5591#issuecomment-1169711765
@BalaMahesh sorry for getting to this issue late. Have you resolved this issue by any chance? If `MARKERS.type` is not present, the logic assumes that the direct markers are stored, which causes the read failure in your case. I'll check if we can improve the error handling in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5591: [SUPPORT] Hudi Error rolling back using marker files
Posted by GitBox <gi...@apache.org>.
codope commented on issue #5591:
URL: https://github.com/apache/hudi/issues/5591#issuecomment-1251044254
Closing as patch is available and should be released in 0.12.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BalaMahesh commented on issue #5591: [SUPPORT] Hudi Error rolling back using marker files
Posted by GitBox <gi...@apache.org>.
BalaMahesh commented on issue #5591:
URL: https://github.com/apache/hudi/issues/5591#issuecomment-1127321049
On checking I have found that `MARKER_TYPE_FILENAME` file is not created but the marker files are being created using timeline server. Since MARKER_TYPE_FILENAME is not found, hudi is assuming it as a direct marker file creating and trying to validate the MARKERSX to have .marker extension. Instead it should read the content of MARKERSX file and then try to validate the content of it as filenames. I am not certain, where this mismatch is happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BalaMahesh commented on issue #5591: [SUPPORT] Hudi Error rolling back using marker files
Posted by GitBox <gi...@apache.org>.
BalaMahesh commented on issue #5591:
URL: https://github.com/apache/hudi/issues/5591#issuecomment-1127252282
@yihua - please do take a look at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5591: [SUPPORT] Hudi Error rolling back using marker files
Posted by GitBox <gi...@apache.org>.
codope commented on issue #5591:
URL: https://github.com/apache/hudi/issues/5591#issuecomment-1208026085
Also, you could try out the following patch #6266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5591: [SUPPORT] Hudi Error rolling back using marker files
Posted by GitBox <gi...@apache.org>.
codope commented on issue #5591:
URL: https://github.com/apache/hudi/issues/5591#issuecomment-1207229300
@BalaMahesh Do you have application logs? Do you see any of the following lines in the log?
`Failed to create marker type file`
`Failed to write marker type file`
Hudi will always throw an exception if marker-type file is not created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #5591: [SUPPORT] Hudi Error rolling back using marker files
Posted by GitBox <gi...@apache.org>.
codope closed issue #5591: [SUPPORT] Hudi Error rolling back using marker files
URL: https://github.com/apache/hudi/issues/5591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5591: [SUPPORT] Hudi Error rolling back using marker files
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5591:
URL: https://github.com/apache/hudi/issues/5591#issuecomment-1243948062
@BalaMahesh : any updates please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org