You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tankkyo <gi...@git.apache.org> on 2016/05/08 13:30:05 UTC
[GitHub] spark pull request: Limit total tasks displayed on Web UI to avoid...
GitHub user tankkyo opened a pull request:
https://github.com/apache/spark/pull/12990
Limit total tasks displayed on Web UI to avoid History Server OOM
## What changes were proposed in this pull request?
History Server will load all task ui data, which would cause OOM if there were too many tasks.
We should allow user to trim them by spark.ui.retainedTasks(default:1000)
## How was this patch tested?
bin/run-example SparkPi 100000, and open stages page on Web UI
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tankkyo/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12990.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12990
----
commit f131cd871d9b4202cad65d5b6d0e967d51713a01
Author: Zheng Tan <zh...@hulu.com>
Date: 2016-05-03T01:10:56Z
Fix SPARK-15059: remove fine-grained lock in ChildFirstURLClassLoader
commit f91966e8998256371666be221fa8f7a6247ff750
Author: Zheng Tan <zh...@hulu.com>
Date: 2016-05-08T12:10:12Z
Merge remote-tracking branch 'upstream/master'
commit 593b294ce36c5e99e6a4c719e4e4f4d64d6b976b
Author: Zheng Tan <zh...@hulu.com>
Date: 2016-05-08T12:48:57Z
Trim tasks in a stage to avoid history server OOM
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/12990
Some of our customers are hitting this issue, any chance you could take a look at this old pr with the fix? @srowen @tgravescs @JoshRosen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/12990
@vanzin maybe you have time to check this out? Also @tankkyo seems to have disappeared, should I open a new pr based on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12990
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/12990
@tankkyo, it looks like this issue has been fixed by #14673, so do you mind closing this PR? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/12990
I'll create a new pr based on this one with a higher default max tasks (maybe 10,000?) and an on/off config when I have time this week
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/12990
@tgravescs I will look into a different solution for this issue (seems @tankkyo has vanished) If I'm understanding you correctly you're suggesting modifying the way the History Server caches applications so task data is not loaded until the application ui for a specific application is opened?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/12990
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12990
**[Test build #63390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63390/consoleFull)** for PR 12990 at commit [`593b294`](https://github.com/apache/spark/commit/593b294ce36c5e99e6a4c719e4e4f4d64d6b976b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/12990
Yeah. That might be a much larger change though too. Ideally the history server is smart and can load just meta data quickly to list application, then if application actually clicked it would load everything else. If this is currently an issue and we need to limit # of tasks shown I'm fine with config I just think it should be off or very large by default. If there are a bunch of others that disagree because their users hit it and 1000 for default is reasonable I guess its ok but would like to hear from others.
I think you should open new PR is he hasn't responded on this one so we aren't blocked waiting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/12990
Sorry I haven't had time to look at this in great detail but have some concerns.
So the way I read this, it will only ever show 1000 tasks on the task table (by default at least)? I don't see that there is a way for it to dynamically load this data or that the user is informed that they hit this limit so I'm concerned it will be confusing. It looks like it applies to both active application and once it was re-read by the history server and I'm assuming history server is using the application setting for this? It seems like it would make more sense for the history server to have its own setting that would override the applications if that is what you are trying to protect running out of memory. I guess the other settings work that way so perhaps that is a separate jira.
We have a ton of jobs that have stages with over 1000 tasks and if I can't see the data needed this page is useless. 1000 seems way low to me. I know the same thing exists for stages but that limit is much less likely to be hit and when you do you either have all data from that stage or none of it. Here you have partial stage data which could be confusing if we happen to remove the "interesting" task.
If I read the screen shot for heap dump properly 65000 tasks were taking about 250MB. On a running application that doesn't seem to bad. On the history server where you can have thousands of applications obviously that can add up, but personally I would prefer us do something smart like not load it until someone clicks on the button and needs the data.
I'm ok with adding a config for this but would prefer to see it default to all or very high number and those that want it smaller can decrease.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12990
**[Test build #63390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63390/consoleFull)** for PR 12990 at commit [`593b294`](https://github.com/apache/spark/commit/593b294ce36c5e99e6a4c719e4e4f4d64d6b976b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12990: [SPARK-15083] [Web UI] Limit total tasks displaye...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12990
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/12990
Also @tankkyo are you still around, your github has been silent since this pr. If necessary I can reopen a pr based on this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12990: [SPARK-15083] [Web UI] Limit total tasks displaye...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/12990#discussion_r74151754
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -137,6 +138,17 @@ class JobProgressListener(conf: SparkConf) extends SparkListener with Logging {
)
}
+ /** If Tasks is too large, remove and garbage collect old tasks */
+ private def trimTasksIfNecessary(taskData: HashMap[Long, TaskUIData]) = synchronized {
+ if (taskData.size > retainedTasks) {
+ val toRemove = math.max(retainedTasks / 10, 1)
+ val oldIds = taskData.map(_._2.taskInfo.taskId).toList.sorted.take(toRemove)
--- End diff --
Also @tgravescs before I make the new pr, what's your opinion on this bit of code? Should I leave this as is or change it to match how jobs and stages are trimmed? I don't see a probe;em with this code, but I don't know why he decided to implement it differently than the precedent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Limit total tasks displayed on Web UI to avoid...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12990#issuecomment-217718804
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12990: [SPARK-15083] [Web UI] Limit total tasks displayed on We...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12990
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63390/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12990: [SPARK-15083] [Web UI] Limit total tasks displaye...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/12990#discussion_r73247048
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -137,6 +138,17 @@ class JobProgressListener(conf: SparkConf) extends SparkListener with Logging {
)
}
+ /** If Tasks is too large, remove and garbage collect old tasks */
+ private def trimTasksIfNecessary(taskData: HashMap[Long, TaskUIData]) = synchronized {
+ if (taskData.size > retainedTasks) {
+ val toRemove = math.max(retainedTasks / 10, 1)
+ val oldIds = taskData.map(_._2.taskInfo.taskId).toList.sorted.take(toRemove)
--- End diff --
Could you explain your reasoning behind doing this different than the equivalent stages and jobs functions below? This just seems a bit redundant comparatively (doing all this to get oldIds then going through a loop, rather than using trimStart) or I may just be missing some key scala understanding
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org