You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/02/04 15:14:30 UTC
[GitHub] [hudi] abhijeetkushe opened a new issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
abhijeetkushe opened a new issue #2533:
URL: https://github.com/apache/hudi/issues/2533
I am no longer able to write to a hudi table because of the error mentioned in the subject.This happened probably because 2 simultaneous EMRs were running at the sametime.I did terminate one EMR but I keep getting the above error.I also tried to running hudi-cli clean and rollback operations based on https://github.com/apache/hudi/issues/1785 but those are failing as well,I have added the stacktrace and commit details.Let me know if anything more is needed
**Environment Description**
* Hudi version : 0.6.0
* Spark version : 2.4.4
* Hive version : 2.3.7
* Hadoop version : 2.10.0
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
Stacktrace:
2021-02-01T19:13:38.849+0000: 84.629: Application time: 0.0001674 seconds
2021-02-01T19:13:38.849+0000: 84.630: Total time for which application threads were stopped: 0.0001441 seconds, Stopping threads took: 0.0000495 seconds
2021-02-01 19:13:38 ERROR AbstractAsyncService:149 - Monitor noticed one or more threads failed. Requesting graceful shutdown of other threads
java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Failed to rollback s3://akctct-transient-dev-us-east-1-eigi-datalake/hudi_deltastreamer_large_100/engmt_open_cow_1 commits 20210129225129
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.hudi.async.AbstractAsyncService.lambda$monitorThreads$1(AbstractAsyncService.java:147)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: Failed to rollback s3://akctct-transient-dev-us-east-1-eigi-datalake/hudi_deltastreamer_large_100/engmt_open_cow_1 commits 20210129225129
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:595)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
... 3 more
Caused by: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback s3://akctct-transient-dev-us-east-1-eigi-datalake/hudi_deltastreamer_large_100/engmt_open_cow_1 commits 20210129225129
at org.apache.hudi.client.HoodieWriteClient.rollback(HoodieWriteClient.java:505)
at org.apache.hudi.client.HoodieWriteClient.rollbackPendingCommits(HoodieWriteClient.java:708)
at org.apache.hudi.client.HoodieWriteClient.startCommit(HoodieWriteClient.java:576)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.startCommit(DeltaSync.java:465)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:381)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:245)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:579)
... 4 more
Caused by: org.apache.hudi.exception.HoodieRollbackException: Found commits after time :20210129225129, please rollback greater commits first
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.validateRollbackCommitSequence(BaseRollbackActionExecutor.java:145)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.doRollbackAndGetStats(BaseRollbackActionExecutor.java:171)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:108)
at org.apache.hudi.table.HoodieCopyOnWriteTable.rollback(HoodieCopyOnWriteTable.java:193)
at org.apache.hudi.client.HoodieWriteClient.rollback(HoodieWriteClient.java:494)
... 10 more
Commit
Loaded instants [[20210129211107__commit__COMPLETED], [20210129211434__clean__COMPLETED], [20210129211434__commit__COMPLETED], [20210129211810__clean__COMPLETED], [20210129211810__commit__COMPLETED], [20210129212234__clean__COMPLETED], [20210129212234__commit__COMPLETED], [20210129212622__clean__COMPLETED], [20210129212622__commit__COMPLETED], [20210129212913__clean__COMPLETED], [20210129212913__commit__COMPLETED], [20210129213326__clean__COMPLETED], [20210129213326__commit__COMPLETED], [20210129213638__clean__COMPLETED], [20210129213638__commit__COMPLETED], [20210129214016__clean__COMPLETED], [20210129214016__commit__COMPLETED], [20210129214432__clean__COMPLETED], [20210129214432__commit__COMPLETED], [20210129214813__clean__COMPLETED], [20210129214813__commit__COMPLETED], [20210129215121__clean__COMPLETED], [20210129215121__commit__COMPLETED], [20210129215538__clean__COMPLETED], [20210129215538__commit__COMPLETED], [20210129215816__clean__COMPLETED], [20210129215816__commit__COMP
LETED], [20210129220159__clean__COMPLETED], [20210129220159__commit__COMPLETED], [20210129220532__clean__COMPLETED], [20210129220532__commit__COMPLETED], [20210129220839__clean__COMPLETED], [20210129220839__commit__COMPLETED], [20210129221301__clean__COMPLETED], [20210129221301__commit__COMPLETED], [20210129221623__clean__COMPLETED], [20210129221623__commit__COMPLETED], [20210129221930__clean__COMPLETED], [20210129221930__commit__COMPLETED], [20210129222350__clean__COMPLETED], [20210129222350__commit__COMPLETED], [20210129222736__clean__COMPLETED], [20210129222736__commit__COMPLETED], [20210129223034__clean__COMPLETED], [20210129223034__commit__COMPLETED], [20210129223422__clean__COMPLETED], [20210129223422__commit__COMPLETED], [20210129223818__clean__COMPLETED], [20210129223818__commit__COMPLETED], [20210129224110__clean__COMPLETED], [20210129224110__commit__COMPLETED], [20210129224446__clean__COMPLETED], [20210129224446__commit__COMPLETED], [20210129224849__clean__COMPLETED], [202
10129224849__commit__COMPLETED], [20210129225129__clean__COMPLETED], [==>20210129225129__commit__INFLIGHT], [20210129225133__commit__COMPLETED], [20210201183500__clean__COMPLETED]]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] abhijeetkushe commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
abhijeetkushe commented on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-777570192
So I did not find hudi-cli restore option in the available options
```
bootstrap index showmapping - Show bootstrap index mapping
* bootstrap index showpartitions - Show bootstrap indexed partitions
* bootstrap run - Run a bootstrap action for current Hudi table
* clean showpartitions - Show partition level details of a clean
* cleans refresh - Refresh table metadata
* cleans run - run clean
* cleans show - Show the cleans
* clear - Clears the console
* cls - Clears the console
* commit rollback - Rollback a commit
* commits compare - Compare commits with another Hoodie table
* commit showfiles - Show file level details of a commit
* commit showpartitions - Show partition level details of a commit
* commits refresh - Refresh table metadata
* commits show - Show the commits
* commits showarchived - Show the archived commits
* commits sync - Compare commits with another Hoodie table
* compaction repair - Renames the files to make them consistent with the timeline as dictated by Hoodie metadata. Use when compaction unschedule fails partially.
* compaction run - Run Compaction for given instant time
* compaction schedule - Schedule Compaction
* compaction show - Shows compaction details for a specific compaction instant
* compaction showarchived - Shows compaction details for a specific compaction instant
* compactions show all - Shows all compactions that are in active timeline
* compactions showarchived - Shows compaction details for specified time window
* compaction unschedule - Unschedule Compaction
* compaction unscheduleFileId - UnSchedule Compaction for a fileId
* compaction validate - Validate Compaction
* connect - Connect to a hoodie table
* create - Create a hoodie table if not present
* date - Displays the local date and time
* desc - Describe Hoodie Table properties
* exit - Exits the shell
* export instants - Export Instants and their metadata from the Timeline
* hdfsparquetimport - Imports Parquet table to a hoodie table
* help - List all commands usage
* metadata refresh - Refresh table metadata
* quit - Exits the shell
* refresh - Refresh table metadata
* repair addpartitionmeta - Add partition metadata to a table, if not present
* repair corrupted clean files - repair corrupted clean files
* repair deduplicate - De-duplicate a partition path contains duplicates & produce repaired files to replace with
* repair overwrite-hoodie-props - Overwrite hoodie.properties with provided file. Risky operation. Proceed with caution!
* savepoint create - Savepoint a commit
* savepoint delete - Delete the savepoint
* savepoint rollback - Savepoint a commit
* savepoints refresh - Refresh table metadata
* savepoints show - Show the savepoints
* script - Parses the specified resource file and executes its commands
* set - Set spark launcher env to cli
* show archived commits - Read commits from archived files and show details
* show archived commit stats - Read commits from archived files and show details
* show env - Show spark launcher env by key
* show envs all - Show spark launcher envs
* show fsview all - Show entire file-system view
* show fsview latest - Show latest file-system view
* show logfile metadata - Read commit metadata from log files
* show logfile records - Read records from log files
* show rollback - Show details of a rollback instant
* show rollbacks - List all rollback instants
* stats filesizes - File Sizes. Display summary stats on sizes of files
* stats wa - Write Amplification. Ratio of how many records were upserted to how many records were actually written
* sync validate - Validate the sync by counting the number of records
* system properties - Shows the shell's properties
* temp_delete - Delete view name
* temp_query - query against created temp view
* temp delete - Delete view name
* temp query - query against created temp view
* temps_show - Show all views name
* temps show - Show all views name
* utils loadClass - Load a class
* version - Displays shell version
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-774418186
>Caused by: org.apache.hudi.exception.HoodieRollbackException: Found commits after time :20210129225129, please rollback greater commits first
if you try rolling back `20210129225133` and then retrying or even that fails? If you don't want to lose `[20210129225133__commit__COMPLETED]`. you can do some sort of easy manual repair, given it s copy on write table.
a) Stop the pipeline
b) delete all files that have `20210129225129` in its name
c) delete `20210129225129__commit__INFLIGHT`
next run should start flowing again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] abhijeetkushe commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
abhijeetkushe commented on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-775967717
@vinothchandar I was able to proceed after deleting the 20210129225129 files from .hoodie folder in s3.But I wanted to know whether anything else needs to be cleaned up ? For example there might have been some versions created for commit time 20210129225129 which i can get from 20210129225129.commit file.Does that need to be cleaned up and if not why ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] abhijeetkushe closed issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
abhijeetkushe closed issue #2533:
URL: https://github.com/apache/hudi/issues/2533
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-781035419
yeah, unfortunately we don't expose this as of today. We have a tracking ticket https://issues.apache.org/jira/browse/HUDI-1593. We will try to get it in next release.
@abhijeetkushe : can we close this support ticket if you are good w/ clarifications.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-774126655
@n3nash @bvaradar : any suggestions on cleaning /repairing Hudi if 2 concurrent jobs have messed up things.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] abhijeetkushe commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
abhijeetkushe commented on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-774488959
@vinothchandar yes I tried rolling back 20210129225133 ..I get the same exception.I had to stop the pipeline as I kept getting the above exception.I will try manually deleting and see if I am able to proceed Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar edited a comment on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
vinothchandar edited a comment on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-776025099
>b) delete all files that have 20210129225129 in its name
actually they need to be deleted as well.
the hudi cli restore should have been able to do this for you actually, as long as you were okay to lose the later commit. I was planning to investigate why that did not work for e,g
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-776025099
>b) delete all files that have 20210129225129 in its name
actually they need to be cleaned as well.
the hudi cli restore should have been able to do this for you actually, as long as you were okay to lose the later commit. I was planning to investigate why that did not work for e,g
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-781391464
got it, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] abhijeetkushe commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
abhijeetkushe commented on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-781352392
Yeah this ticket can be closed.We are using EMR built in version of hudi so will be hard for us to upgrade.So for now we have build checks to ensure this does not happen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] abhijeetkushe edited a comment on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first
Posted by GitBox <gi...@apache.org>.
abhijeetkushe edited a comment on issue #2533:
URL: https://github.com/apache/hudi/issues/2533#issuecomment-777570192
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org