You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "kejiqing (JIRA)" <ji...@apache.org> on 2018/03/02 02:03:00 UTC

[jira] [Commented] (SPARK-23563) make the size fo cache in CodeGenerator configable

    [ https://issues.apache.org/jira/browse/SPARK-23563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383005#comment-16383005 ] 

kejiqing commented on SPARK-23563:
----------------------------------

a long term spark sql task, the meta space in driver will increase endless, and will lead full gc frequently.

a suitable cache size can slow down the speed of new class generate in jvm and rise the ratio of cache hit 

> make the size fo cache in CodeGenerator configable
> --------------------------------------------------
>
>                 Key: SPARK-23563
>                 URL: https://issues.apache.org/jira/browse/SPARK-23563
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: kejiqing
>            Priority: Minor
>
> the cache in class org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator has a hard cod maxmunSize 100, current code is:
>  
> {code:java}
> // scala
> private val cache = CacheBuilder.newBuilder()
>   .maximumSize(100)
>   .build(
>     new CacheLoader[CodeAndComment, (GeneratedClass, Int)]() {
>       override def load(code: CodeAndComment): (GeneratedClass, Int) = {
>         val startTime = System.nanoTime()
>         val result = doCompile(code)
>         val endTime = System.nanoTime()
>         def timeMs: Double = (endTime - startTime).toDouble / 1000000
>         CodegenMetrics.METRIC_SOURCE_CODE_SIZE.update(code.body.length)
>         CodegenMetrics.METRIC_COMPILATION_TIME.update(timeMs.toLong)
>         logInfo(s"Code generated in $timeMs ms")
>         result
>       }
>     })
> {code}
>  In some specific situation, for example: a long term and spark tasks are unchanged,  the size of cache maximumSize configuration is a better idea.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org