[jira] [Updated] (SPARK-10155) Memory leak in SQL parsers

["Yin Huai (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yin Huai updated SPARK-10155:
-----------------------------
    Target Version/s: 1.6.0, 1.5.1  (was: 1.6.0)

> Memory leak in SQL parsers
> --------------------------
>
>                 Key: SPARK-10155
>                 URL: https://issues.apache.org/jira/browse/SPARK-10155
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Shixiong Zhu
>            Priority: Critical
>         Attachments: Screen Shot 2015-08-21 at 5.45.24 PM.png
>
>
> I saw a lot of `ThreadLocal` objects in the following app:
> {code}
> import org.apache.spark._
> import org.apache.spark.sql._
> object SparkApp {
>   def foo(sqlContext: SQLContext): Unit = {
>     import sqlContext.implicits._
>     sqlContext.sparkContext.parallelize(Seq("aaa", "bbb", "ccc")).toDF().filter("length(_1) > 0").count()
>   }
>   def main(args: Array[String]): Unit = {
>     val conf = new SparkConf().setAppName("sql-memory-leak")
>     val sc = new SparkContext(conf)
>     val sqlContext = new SQLContext(sc)
>     while (true) {
>       foo(sqlContext)
>     }
>   }
> }
> {code}
> Running the above codes in a long time and finally it will OOM.
> These "ThreadLocal"s are from "scala.util.parsing.combinator.Parsers.lastNoSuccessVar", which stores `Failure("end of input", ...)`.
> There is an issue in Scala here: https://issues.scala-lang.org/browse/SI-9010
> and some discussions here: https://issues.scala-lang.org/browse/SI-4929



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org