You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/04/15 03:40:00 UTC
[jira] [Assigned] (SPARK-38910) Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION is false too
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-38910:
------------------------------------
Assignee: Apache Spark
> Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION is false too
> ----------------------------------------------------------------
>
> Key: SPARK-38910
> URL: https://issues.apache.org/jira/browse/SPARK-38910
> Project: Spark
> Issue Type: Task
> Components: YARN
> Affects Versions: 3.2.1, 3.3.0
> Reporter: angerszhu
> Assignee: Apache Spark
> Priority: Major
>
> {code:java}
> def run(): Unit = {
> submitApplication()
> if (!launcherBackend.isConnected() && fireAndForget) {
> val report = getApplicationReport(appId)
> val state = report.getYarnApplicationState
> logInfo(s"Application report for $appId (state: $state)")
> logInfo(formatReportDetails(report, getDriverLogsLink(report)))
> if (state == YarnApplicationState.FAILED || state == YarnApplicationState.KILLED) {
> throw new SparkException(s"Application $appId finished with status: $state")
> }
> } else {
> val YarnAppReport(appState, finalState, diags) = monitorApplication(appId)
> if (appState == YarnApplicationState.FAILED || finalState == FinalApplicationStatus.FAILED) {
> var amContainerSucceed = false
> val amContainerExitMsg = s"AM Container for " +
> s"${yarnClient.getApplicationReport(appId).getCurrentApplicationAttemptId} " +
> s"exited with exitCode: 0"
> diags.foreach { err =>
> logError(s"Application diagnostics message: $err")
> if (err.contains(amContainerExitMsg)) {
> amContainerSucceed = true
>
> {code}
> Not clean the staging dir when match case
> {code:jave}
> !launcherBackend.isConnected() && fireAndForget
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org