You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/22 03:38:21 UTC
[GitHub] [hudi] watermelon12138 opened a new pull request, #5391: [HUDI-3945] After the async compaction operation is complete, the tas…
watermelon12138 opened a new pull request, #5391:
URL: https://github.com/apache/hudi/pull/5391
…k should exit.
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5391: [HUDI-3945] After the async compaction operation is complete, the tas…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5391:
URL: https://github.com/apache/hudi/pull/5391#issuecomment-1105973489
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "71792295fe72e9a2c988bb4a2989e24c44cd85d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8225",
"triggerID" : "71792295fe72e9a2c988bb4a2989e24c44cd85d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71792295fe72e9a2c988bb4a2989e24c44cd85d0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #5391: [HUDI-3945] After the async compaction operation is complete, the task should exit
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on code in PR #5391:
URL: https://github.com/apache/hudi/pull/5391#discussion_r859716348
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java:
##########
@@ -171,13 +171,15 @@ public static void main(String[] args) {
System.exit(1);
}
final JavaSparkContext jsc = UtilHelpers.buildSparkContext("compactor-" + cfg.tableName, cfg.sparkMaster, cfg.sparkMemory);
+ int ret = 0;
try {
HoodieCompactor compactor = new HoodieCompactor(jsc, cfg);
- compactor.compact(cfg.retry);
+ ret = compactor.compact(cfg.retry);
} catch (Throwable throwable) {
LOG.error("Fail to run compaction for " + cfg.tableName, throwable);
} finally {
jsc.stop();
+ System.exit(ret);
Review Comment:
why we need call System.exit here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5391: [HUDI-3945] After the async compaction operation is complete, the tas…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5391:
URL: https://github.com/apache/hudi/pull/5391#issuecomment-1106025404
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "71792295fe72e9a2c988bb4a2989e24c44cd85d0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8225",
"triggerID" : "71792295fe72e9a2c988bb4a2989e24c44cd85d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71792295fe72e9a2c988bb4a2989e24c44cd85d0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #5391: [HUDI-3945] After the async compaction operation is complete, the task should exit
Posted by GitBox <gi...@apache.org>.
TengHuo commented on PR #5391:
URL: https://github.com/apache/hudi/pull/5391#issuecomment-1362587097
And this is an simple example to produce this error `exitCode: 16` for proving `System.exit` is not appropriate way
```scala
object SparkExitTest {
val spark: SparkSession = SparkSession
.builder
.appName("Test_HuoTeng_Spark_Exit")
.config("spark.driver.memory", "6g")
.config("spark.executor.memory", "4g")
.config("spark.executor.instances", 1)
.config("spark.executor.cores", 1)
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.getOrCreate()
def main(args: Array[String]): Unit = {
var ret = 0
try {
ret = 1
} catch {
case throwable: Throwable =>
println("Fail to run because " + throwable)
} finally {
spark.close()
System.exit(ret)
}
}
}
```
Submit this Spark application in Yarn cluster mode will get this error
```log
22/12/22 17:05:36 INFO SparkContext: Successfully stopped SparkContext
22/12/22 17:05:36 INFO ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)
22/12/22 17:05:36 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Shutdown hook called before final status was reported.)
22/12/22 17:05:36 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rahil-c commented on pull request #5391: [HUDI-3945] After the async compaction operation is complete, the task should exit
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #5391:
URL: https://github.com/apache/hudi/pull/5391#issuecomment-1495174849
@watermelon12138 What spark version did you encounter the issue which prompted you to create the pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5391: [HUDI-3945] After the async compaction operation is complete, the tas…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5391:
URL: https://github.com/apache/hudi/pull/5391#issuecomment-1105972203
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "71792295fe72e9a2c988bb4a2989e24c44cd85d0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "71792295fe72e9a2c988bb4a2989e24c44cd85d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71792295fe72e9a2c988bb4a2989e24c44cd85d0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 commented on a diff in pull request #5391: [HUDI-3945] After the async compaction operation is complete, the task should exit
Posted by GitBox <gi...@apache.org>.
watermelon12138 commented on code in PR #5391:
URL: https://github.com/apache/hudi/pull/5391#discussion_r859753035
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java:
##########
@@ -171,13 +171,15 @@ public static void main(String[] args) {
System.exit(1);
}
final JavaSparkContext jsc = UtilHelpers.buildSparkContext("compactor-" + cfg.tableName, cfg.sparkMaster, cfg.sparkMemory);
+ int ret = 0;
try {
HoodieCompactor compactor = new HoodieCompactor(jsc, cfg);
- compactor.compact(cfg.retry);
+ ret = compactor.compact(cfg.retry);
} catch (Throwable throwable) {
LOG.error("Fail to run compaction for " + cfg.tableName, throwable);
} finally {
jsc.stop();
+ System.exit(ret);
Review Comment:
Thank you for your review. Here you can find out why. https://issues.apache.org/jira/browse/HUDI-3945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rahil-c commented on pull request #5391: [HUDI-3945] After the async compaction operation is complete, the task should exit
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #5391:
URL: https://github.com/apache/hudi/pull/5391#issuecomment-1495127618
@yihua @xiarixiaoyao Wanted to get commmunity thoughts if this is safe to revert, I also tried the steps mentioned in the JIRA to see if this `sys.exit` is required https://issues.apache.org/jira/browse/HUDI-3945 but from my own repro without the sys exit call things are working fine similar to what @TengHuo mentioned
The concern with this `sys.exit` call can be seen here mentioned in spark code https://github.com/apache/spark/blob/v3.3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L258
```
// If user application is exited ahead of time by calling System.exit(N), here mark
// this application as failed with EXIT_EARLY. For a good shutdown, user shouldn't call
// System.exit(0) to terminate the application.
```
This is where the `ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao merged pull request #5391: [HUDI-3945] After the async compaction operation is complete, the task should exit
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao merged PR #5391:
URL: https://github.com/apache/hudi/pull/5391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rahil-c commented on a diff in pull request #5391: [HUDI-3945] After the async compaction operation is complete, the task should exit
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #5391:
URL: https://github.com/apache/hudi/pull/5391#discussion_r1156363206
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java:
##########
@@ -171,13 +171,15 @@ public static void main(String[] args) {
System.exit(1);
}
final JavaSparkContext jsc = UtilHelpers.buildSparkContext("compactor-" + cfg.tableName, cfg.sparkMaster, cfg.sparkMemory);
+ int ret = 0;
try {
HoodieCompactor compactor = new HoodieCompactor(jsc, cfg);
- compactor.compact(cfg.retry);
+ ret = compactor.compact(cfg.retry);
} catch (Throwable throwable) {
LOG.error("Fail to run compaction for " + cfg.tableName, throwable);
} finally {
jsc.stop();
+ System.exit(ret);
Review Comment:
Seeing a similar case to above exception as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on a diff in pull request #5391: [HUDI-3945] After the async compaction operation is complete, the task should exit
Posted by GitBox <gi...@apache.org>.
TengHuo commented on code in PR #5391:
URL: https://github.com/apache/hudi/pull/5391#discussion_r1055233175
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java:
##########
@@ -171,13 +171,15 @@ public static void main(String[] args) {
System.exit(1);
}
final JavaSparkContext jsc = UtilHelpers.buildSparkContext("compactor-" + cfg.tableName, cfg.sparkMaster, cfg.sparkMemory);
+ int ret = 0;
try {
HoodieCompactor compactor = new HoodieCompactor(jsc, cfg);
- compactor.compact(cfg.retry);
+ ret = compactor.compact(cfg.retry);
} catch (Throwable throwable) {
LOG.error("Fail to run compaction for " + cfg.tableName, throwable);
} finally {
jsc.stop();
+ System.exit(ret);
Review Comment:
Hi
We got an issue in `HoodieCompactor` recently, which might be related with this PR.
This is the exception log we got in our application, it showed `ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)`
```
...
22/12/19 02:29:49 INFO EmbeddedTimelineService: Closed Timeline server
22/12/19 02:29:49 INFO AbstractConnector: Stopped Spark@24872375{HTTP/1.1, (http/1.1)}{0.0.0.0:8090}
22/12/19 02:29:49 INFO SparkUI: Stopped Spark web UI at http://xxxxx:8090
22/12/19 02:29:49 INFO YarnClusterSchedulerBackend: Shutting down all executors
22/12/19 02:29:49 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
22/12/19 02:29:49 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/12/19 02:29:49 INFO BlockManager: BlockManager stopped
22/12/19 02:29:49 INFO BlockManagerMaster: BlockManagerMaster stopped
22/12/19 02:29:49 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/12/19 02:29:49 INFO SparkContext: Successfully stopped SparkContext
22/12/19 02:29:49 INFO ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)
22/12/19 02:29:49 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Shutdown hook called before final status was reported.)
22/12/19 02:29:49 INFO Metrics: Stopping the metrics reporter...
22/12/19 02:29:50 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.
22/12/19 02:29:50 INFO ApplicationMaster: Deleting staging directory hdfs://xxxxx
22/12/19 02:29:50 INFO ShutdownHookManager: Shutdown hook called
22/12/19 02:29:50 INFO ShutdownHookManager: Deleting directory /xxxxx
```
After trouble shooting, I think this error is caused by this line here, which is an inappropriate way to shutdown Spark application in yarn cluster mode. (reference in other repo: https://github.com/broadinstitute/gatk/pull/3400, https://github.com/sparklyr/sparklyr/issues/1903).
And I have tried reproduce this issue https://issues.apache.org/jira/browse/HUDI-3945 by using approach mentioned in the ticket description, it said `HoodieCompactor` can't exit after compaction. But I can't reproduce it in my environment.
May I ask if there is a clear way that we can reproduce the issue HUDI-3945? If it can't be reproduced, think we can remove this line here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org