You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by puravaggarwal <gi...@git.apache.org> on 2014/03/06 04:55:03 UTC
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
GitHub user puravaggarwal opened a pull request:
https://github.com/apache/spark/pull/89
MetadataCleaner - fine control cleanup documentation
Documentation with reference to commit b5025d90bbf3e81701a3c3f5e22efbb2e2164eea by Mridul
Date: Fri Oct 4 21:57:57 2013 +0530
- Allow for finer control of cleaner
- Address review comments, move to incubator spark
- Also includes a change to speculation - including preventing exceptions in rare cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/puravaggarwal/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/89.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #89
----
commit 8af169fdbb5b9b4c7de170068699a9209d48f416
Author: Purav Aggarwal <pu...@guavus.com>
Date: 2014-03-06T03:43:39Z
MetadataCleaner - fine control cleanup documentation
commit adc1d84e46aad6960e7d85246e86f39eb905ee12
Author: Purav Aggarwal <pu...@guavus.com>
Date: 2014-03-06T03:50:05Z
MetadataCleaner - fine control cleanup documentation, putting back the unintended removal of a property.
commit 58eab7dc4d48ce37227775318f0835c60c1567a0
Author: Purav Aggarwal <pu...@guavus.com>
Date: 2014-03-06T03:53:05Z
MetadataCleaner - fine control cleanup documentation, rectifying the indentation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-53973053
I agree, we should not expose these to the user given the recent changes. Would it be okay to close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/89#discussion_r10559136
--- Diff: docs/configuration.md ---
@@ -487,6 +477,88 @@ Apart from these, the following properties are also available, and may be useful
</tr>
</table>
+
+The following are the properties that can be used to schedule cleanup jobs at different levels.
+The below mentioned metadata tuning parameters should be set with a lot of consideration and only where required.
+Scheduling metadata cleaning in the middle of job can result in a lot of unnecessary re-computations.
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr>
+ <td>spark.cleaner.ttl</td>
+ <td>(infinite)</td>
+ <td>
+ Duration (seconds) of how long Spark will remember any metadata (stages generated, tasks generated, etc.).
+ Periodic cleanups will ensure that metadata older than this duration will be forgetten. This is
+ useful for running Spark for many hours / days (for example, running 24/7 in case of Spark Streaming
+ applications). Note that any RDD that persists in memory for more than this duration will be cleared as well.
+ </td>
+</tr>
+<tr>
+ <td>spark.cleaner.ttl.MAP_OUTPUT_TRACKER</td>
+ <td>spark.cleaner.ttl, with a min. value of 10 secs</td>
+ <td>
+ Cleans up the map containing the information of the mapper (the input block manager Id and the output result size) corresponding to a shuffle Id.
+ </td>
--- End diff --
you might want to add that this takes precedence over spark.cleaner.ttl
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by puravaggarwal <gi...@git.apache.org>.
Github user puravaggarwal closed the pull request at:
https://github.com/apache/spark/pull/89
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-36824083
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by puravaggarwal <gi...@git.apache.org>.
Github user puravaggarwal commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-54717488
Closed the PR#89.
On Sat, Sep 6, 2014 at 6:20 AM, andrewor14 <no...@github.com> wrote:
> @puravaggarwal <https://github.com/puravaggarwal> Would you mind closing
> this issue?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/89#issuecomment-54697420>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-38881911
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-37554932
@mridulm and @puravaggarwal Thanks for doing this. As a general direction, however, we are actually moving away from MetadataCleaner in the long run - PR#126 is the first step, which cleans up state explicitly once it is out of scope of the application. Periodically cleaning up is really a somewhat hacky mechanism because surprisingly often we end up removing what the application actually still needs, especially in streaming.
Therefore, I am not sure if we actually want to expose these fields to the user, since doing so encourages them to use this feature that is not technically correct and intended to be deprecated in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/89#discussion_r10559142
--- Diff: docs/configuration.md ---
@@ -487,6 +477,88 @@ Apart from these, the following properties are also available, and may be useful
</tr>
</table>
+
+The following are the properties that can be used to schedule cleanup jobs at different levels.
+The below mentioned metadata tuning parameters should be set with a lot of consideration and only where required.
+Scheduling metadata cleaning in the middle of job can result in a lot of unnecessary re-computations.
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr>
+ <td>spark.cleaner.ttl</td>
+ <td>(infinite)</td>
+ <td>
+ Duration (seconds) of how long Spark will remember any metadata (stages generated, tasks generated, etc.).
+ Periodic cleanups will ensure that metadata older than this duration will be forgetten. This is
+ useful for running Spark for many hours / days (for example, running 24/7 in case of Spark Streaming
+ applications). Note that any RDD that persists in memory for more than this duration will be cleared as well.
+ </td>
+</tr>
+<tr>
+ <td>spark.cleaner.ttl.MAP_OUTPUT_TRACKER</td>
+ <td>spark.cleaner.ttl, with a min. value of 10 secs</td>
+ <td>
+ Cleans up the map containing the information of the mapper (the input block manager Id and the output result size) corresponding to a shuffle Id.
+ </td>
--- End diff --
same for rest ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-37518471
Looks good to me, sorry for not updating the documentation when I added this !
/CC @ pwendell
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-54694763
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by puravaggarwal <gi...@git.apache.org>.
Github user puravaggarwal commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-54703862
As per discussions above - closing the pull request
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-54697420
@puravaggarwal Would you mind closing this issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org