You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liujianhuiouc <gi...@git.apache.org> on 2017/01/03 04:23:07 UTC
[GitHub] spark pull request #16456: [SPARK-18994] clean up the local directories for ...
GitHub user liujianhuiouc opened a pull request:
https://github.com/apache/spark/pull/16456
[SPARK-18994] clean up the local directories for application in future by annother thread
## What changes were proposed in this pull request?
clean up the directories of the app by asynchronous method in future block
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liujianhuiouc/spark-1 spark-18994
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16456.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16456
----
commit 0351f2c5b875ed2da6a17cdff4ac690cf145bb6b
Author: liujianhui <li...@didichuxing>
Date: 2017-01-03T04:21:25Z
[spark-18994] asyn to delete the app directories
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16456: [SPARK-18994] clean up the local directories for applica...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16456
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16456: [SPARK-18994] clean up the local directories for applica...
Posted by liujianhuiouc <gi...@git.apache.org>.
Github user liujianhuiouc commented on the issue:
https://github.com/apache/spark/pull/16456
yes, the root cause is the heartbeat message blocked\uff0cthe Heartbeat could be send in parallel, but in any way, the deleting of the app's directories should be async
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16456: [SPARK-18994] clean up the local directories for applica...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16456
My point is that this is an ad-hoc band-aid for one type of message, and not really a solution to other messages blocking a heartbeat. IT's probably best to fix this for real if at all.
If a band-aid is important, deleting N directories in parallel might solve most practical problems.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16456: [SPARK-18994] clean up the local directories for applica...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16456
I don't know that this is the right way to fix it. The fundamental problem is that the heartbeat message shouldn't block on this.
Also, it might be simpler to do this in parallel rather than asynchronously. T
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16456: [SPARK-18994] clean up the local directories for applica...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16456
Should we close this? @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16456: [SPARK-18994] clean up the local directories for applica...
Posted by liujianhuiouc <gi...@git.apache.org>.
Github user liujianhuiouc commented on the issue:
https://github.com/apache/spark/pull/16456
in actual scene, it's only one executor's director for an app, does you mean that delete the child directories in parallel? in my opinion, it's unnecessary to delete that in parallel, could be deleted in future, to avoid other message to block the heartbeat, does it right to send heartbeat in another thread?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16456: [SPARK-18994] clean up the local directories for ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16456
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16456: [SPARK-18994] clean up the local directories for applica...
Posted by liujianhuiouc <gi...@git.apache.org>.
Github user liujianhuiouc commented on the issue:
https://github.com/apache/spark/pull/16456
add another thread to send heartbeat in parallel, and also remove the part of deleting directories in async
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org