You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mridul Muralidharan (JIRA)" <ji...@apache.org> on 2018/10/02 07:59:00 UTC
[jira] [Commented] (SPARK-25594) OOM in long running applications
even with UI disabled
[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635122#comment-16635122 ]
Mridul Muralidharan commented on SPARK-25594:
---------------------------------------------
Task level information is required only when spark UI is enabled, and tends to be expensive to maintain (TaskDataWrapper was occupying bulk of the heap at OOM).
We can avoid maintaining task information when it is a live UI (not history server) and spark UI has been disabled (spark.ui.enabled = false)
> OOM in long running applications even with UI disabled
> ------------------------------------------------------
>
> Key: SPARK-25594
> URL: https://issues.apache.org/jira/browse/SPARK-25594
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.0, 2.4.0
> Reporter: Mridul Muralidharan
> Assignee: Mridul Muralidharan
> Priority: Major
>
> Typically for long running applications with large number of tasks it is common to disable UI to minimize overhead at driver.
> Earlier, with spark ui disabled, only stage/job information was kept as part of JobProgressListener.
> As part of history server scalability fixes, particularly SPARK-20643, inspite of disabling UI - task information continues to be maintained in memory.
> In our long running tests against spark thrift server, this eventually results in OOM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org