You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/12 03:13:26 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request, #37162: [SPARK-38910][YARN][FOLLOWUP] Clean spark staging before unregister
AngersZhuuuu opened a new pull request, #37162:
URL: https://github.com/apache/spark/pull/37162
### What changes were proposed in this pull request?
After discussing about https://github.com/apache/spark/pull/36207 and re-check the whole logic, we should revert https://github.com/apache/spark/pull/36207 and do some change
1. If it's the last attempt, anyway yarn won't rerun the job, we can clean staging dir first then we can avoid remaining staging dir if unregister failed
2. If it's not the last attempt and the final status is SUCCESS, if unregister failed, yarn can rerun the job again, we can't clean the staging dir before unregistering success.
### Why are the changes needed?
Revert change and make it more accurate
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a diff in pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on code in PR #37162:
URL: https://github.com/apache/spark/pull/37162#discussion_r926354860
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:
##########
@@ -327,8 +333,8 @@ private[spark] class ApplicationMaster(
ApplicationMaster.EXIT_UNCAUGHT_EXCEPTION,
"Uncaught exception: " + StringUtils.stringifyException(e))
if (!unregistered) {
- unregister(finalStatus, finalMsg)
cleanupStagingDir(stagingDir)
+ unregister(finalStatus, finalMsg)
Review Comment:
DOne
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a diff in pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
tgravescs commented on code in PR #37162:
URL: https://github.com/apache/spark/pull/37162#discussion_r924799847
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:
##########
@@ -327,8 +333,8 @@ private[spark] class ApplicationMaster(
ApplicationMaster.EXIT_UNCAUGHT_EXCEPTION,
"Uncaught exception: " + StringUtils.stringifyException(e))
if (!unregistered) {
- unregister(finalStatus, finalMsg)
cleanupStagingDir(stagingDir)
+ unregister(finalStatus, finalMsg)
Review Comment:
add comment saying ok to clean staging dir first because unmanaged AM can't be retried.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #37162:
URL: https://github.com/apache/spark/pull/37162#issuecomment-1209381775
> can you try kicking the tests again?
Yea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
tgravescs commented on PR #37162:
URL: https://github.com/apache/spark/pull/37162#issuecomment-1189371711
items 2 and 3 in the description, is one of those supposed to be client mode? Otherwise they are the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #37162:
URL: https://github.com/apache/spark/pull/37162#issuecomment-1192245494
@tgravescs Seems latest twice GA failed not caused by this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #37162: [SPARK-38910][YARN][FOLLOWUP] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #37162:
URL: https://github.com/apache/spark/pull/37162#issuecomment-1181266134
waiting for @tgravescs back and review this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
URL: https://github.com/apache/spark/pull/37162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
tgravescs commented on PR #37162:
URL: https://github.com/apache/spark/pull/37162#issuecomment-1209359619
can you try kicking the tests again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
tgravescs commented on PR #37162:
URL: https://github.com/apache/spark/pull/37162#issuecomment-1210758506
merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #37162:
URL: https://github.com/apache/spark/pull/37162#issuecomment-1210072864
@tgravescs All GA passed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #37162:
URL: https://github.com/apache/spark/pull/37162#issuecomment-1193608014
Test the failed UT in local, it can success.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #37162: [SPARK-38910][YARN] Clean spark staging before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #37162:
URL: https://github.com/apache/spark/pull/37162#issuecomment-1197950750
ping @dongjoon-hyun The latest GA failed caused by
```
* DONE (miniUI)
ERROR: dependency ‘pkgdown’ is not available for package ‘devtools’
* removing ‘/usr/local/lib/R/site-library/devtools’
The downloaded source packages are in
‘/tmp/RtmpTvMfJ6/downloaded_packages’
Warning messages:
1: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘systemfonts’ had non-zero exit status
2: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘textshaping’ had non-zero exit status
3: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘ragg’ had non-zero exit status
4: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘pkgdown’ had non-zero exit status
5: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘devtools’ had non-zero exit status
Error in loadNamespace(x) : there is no package called ‘devtools’
Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted
Error: Process completed with exit code 1.
```
Any advise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org