You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/30 01:38:08 UTC
[GitHub] [hudi] zhangyue19921010 opened a new pull request #5174: [Minor]Make cli 'commit rollback' using rollbackUsingMarkers false as default
zhangyue19921010 opened a new pull request #5174:
URL: https://github.com/apache/hudi/pull/5174
When users trigger 'commit rollback' for a committed instant, it will throw followed exception when set rollbackUsingMarkers
true as default.
We may need a common strategy for rollback action either rolling back an uncompleted instant or a finished commit.
Also users could set rollbackUsingMarkers on demands for better performance when rolling back an uncompleted instant
```
1643654 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:03 INFO Executor: Adding file:/private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-97304f5c-e7f7-41b3-a535-c646b6ed02c6/userFiles-1d3d3fa9-f552-4939-b630-f0bc3bad7ed0/jetty-http-9.4.15.v20190215.jar to class loader
1643851 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 710 bytes result sent to driver
1643863 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4441 ms on localhost (executor driver) (1/1)
1643868 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
1643874 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO DAGScheduler: ResultStage 0 (collect at HoodieSparkEngineContext.java:100) finished in 5.984 s
1643881 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO DAGScheduler: Job 0 finished: collect at HoodieSparkEngineContext.java:100, took 6.062257 s
1643962 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220329175957497__rollback__REQUESTED]}
1643962 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO BaseRollbackPlanActionExecutor: Requesting Rollback with instant time [==>20220329175957497__rollback__REQUESTED]
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 ERROR SparkMain: Fail to execute commandString
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - org.apache.hudi.exception.HoodieRollbackException: Failed to rollback /Users/yuezhang/tmp/hudiAfTable/forecast_agg commits 20220329175657735
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:706)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:652)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.cli.commands.SparkMain.rollback(SparkMain.java:447)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.cli.commands.SparkMain.main(SparkMain.java:98)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at java.lang.reflect.Method.invoke(Method.java:498)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant:[20220329175657735__commit__COMPLETED]
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.<init>(BaseRollbackActionExecutor.java:93)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.<init>(BaseRollbackActionExecutor.java:73)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.table.action.rollback.CopyOnWriteRollbackActionExecutor.<init>(CopyOnWriteRollbackActionExecutor.java:48)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.rollback(HoodieSparkCopyOnWriteTable.java:345)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:689)
1643965 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - ... 15 more
1643974 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO SparkUI: Stopped Spark web UI at http://172.24.10.19:4041
1643984 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
1643993 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO MemoryStore: MemoryStore cleared
1643993 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO BlockManager: BlockManager stopped
1644000 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO BlockManagerMaster: BlockManagerMaster stopped
1644002 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
1644164 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO SparkContext: Successfully stopped SparkContext
1644168 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO ShutdownHookManager: Shutdown hook called
1644168 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO ShutdownHookManager: Deleting directory /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-97304f5c-e7f7-41b3-a535-c646b6ed02c6
1644172 [Thread-55] INFO org.apache.hudi.cli.utils.InputStreamConsumer - 22/03/29 18:00:04 INFO ShutdownHookManager: Deleting directory /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/spark-61dd4ecc-a2b7-45a1-831b-468ce36d3143
1644562 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Loading HoodieTableMetaClient from /Users/yuezhang/tmp/hudiAfTable/forecast_agg
1644572 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/hoodie.properties
1644573 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from /Users/yuezhang/tmp/hudiAfTable/forecast_agg
1644573 [Spring Shell] INFO org.springframework.shell.core.JLineShellComponent - Commit 20220329175657735 failed to roll back
```
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5174: [Minor]Make cli 'commit rollback' using rollbackUsingMarkers false as default
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5174:
URL: https://github.com/apache/hudi/pull/5174#issuecomment-1082531603
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7535",
"triggerID" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d09991ecdecbf15c37352e713a3add4213755e0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7535)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5174: [Minor]Make cli 'commit rollback' using rollbackUsingMarkers false as default
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5174:
URL: https://github.com/apache/hudi/pull/5174#issuecomment-1082531603
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7535",
"triggerID" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d09991ecdecbf15c37352e713a3add4213755e0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7535)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5174: [Minor]Make cli 'commit rollback' using rollbackUsingMarkers false as default
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5174:
URL: https://github.com/apache/hudi/pull/5174#issuecomment-1082530218
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d09991ecdecbf15c37352e713a3add4213755e0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5174: [Minor]Make cli 'commit rollback' using rollbackUsingMarkers false as default
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5174:
URL: https://github.com/apache/hudi/pull/5174#issuecomment-1082530218
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d09991ecdecbf15c37352e713a3add4213755e0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5174: [Minor]Make cli 'commit rollback' using rollbackUsingMarkers false as default
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5174:
URL: https://github.com/apache/hudi/pull/5174#issuecomment-1082627856
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7535",
"triggerID" : "6d09991ecdecbf15c37352e713a3add4213755e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d09991ecdecbf15c37352e713a3add4213755e0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7535)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org