[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

["Stanislav Los (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646723#comment-16646723 ] 

Stanislav Los commented on SPARK-23904:
---------------------------------------

[~igreenfi] [~RBerenguel] we had the same issue and I found easier solution to it (without need of altering Spark code). See below. I also updated stackoverflow.

We faced the same issue, and solution is to set parameter "spark.sql.ui.retainedExecutions" to lower value, for example --conf "spark.sql.ui.retainedExecutions=10" 
By default it's 1000.


It keeps instances count of org.apache.spark.sql.execution.ui.SQLExecutionUIData low enough.
SQLExecutionUIData have a reference to physicalPlanDescription, which can get very big.
In our case we had to read huge avro messages from Kafka with lot's of fields, and plan description was in the area of 8mg each.

> Big execution plan cause OOM
> ----------------------------
>
>                 Key: SPARK-23904
>                 URL: https://issues.apache.org/jira/browse/SPARK-23904
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.1
>            Reporter: Izek Greenfield
>            Priority: Major
>              Labels: SQL, query
>
> I create a question in [StackOverflow|https://stackoverflow.com/questions/49508683/spark-physicalplandescription-string-is-to-big] 
> Spark create the text representation of query in any case even if I don't need it.
> That causes many garbage object and unneeded GC... 
>  [Gist with code to reproduce|https://gist.github.com/igreenfield/584c3336f03ba7d63e9026774eaf5e23]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org