You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/07 14:28:57 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request, #37119: [SPARK-38910][YARN][FOLLOWUP] Unmanaged AM should clean staging dir before unregister
AngersZhuuuu opened a new pull request, #37119:
URL: https://github.com/apache/spark/pull/37119
### What changes were proposed in this pull request?
Unmanaged AM should clean staging dir before unregister, since unregister may throw timeout exception caused by yarn's bad performance
### Why are the changes needed?
Clan staging dir
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Not need
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #37119: [SPARK-38910][YARN][FOLLOWUP] Unmanaged AM should clean staging dir before unregister
Posted by GitBox <gi...@apache.org>.
tgravescs commented on PR #37119:
URL: https://github.com/apache/spark/pull/37119#issuecomment-1177716755
I still want to understand this issue before making more change as discussing on:
https://github.com/apache/spark/pull/36207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #37119: [SPARK-38910][YARN][FOLLOWUP] Unmanaged AM should clean staging dir before unregister
Posted by GitBox <gi...@apache.org>.
tgravescs commented on PR #37119:
URL: https://github.com/apache/spark/pull/37119#issuecomment-1179080739
-1 because other pr should be reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #37119: [SPARK-38910][YARN][FOLLOWUP] Unmanaged AM should clean staging dir before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #37119:
URL: https://github.com/apache/spark/pull/37119#issuecomment-1177723143
> I still want to understand this issue before making more change as discussing on: #36207
The process is:
1. Application success
2. Call AM's shutdown hook
3. Unregister with AM throw timeout exception.
4. Shutdown hook throw Timeout exception but won't impact the process since it have try catch
5. staging dir no cleaned
Your concern for this case is yarn should rerun the JOB?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu closed pull request #37119: [SPARK-38910][YARN][FOLLOWUP] Unmanaged AM should clean staging dir before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu closed pull request #37119: [SPARK-38910][YARN][FOLLOWUP] Unmanaged AM should clean staging dir before unregister
URL: https://github.com/apache/spark/pull/37119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #37119: [SPARK-38910][YARN][FOLLOWUP] Unmanaged AM should clean staging dir before unregister
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #37119:
URL: https://github.com/apache/spark/pull/37119#issuecomment-1177705796
@srowen @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org