You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "kejiqing (JIRA)" <ji...@apache.org> on 2018/03/02 02:03:00 UTC
[jira] [Commented] (SPARK-23563) make the size fo cache in
CodeGenerator configable
[ https://issues.apache.org/jira/browse/SPARK-23563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383005#comment-16383005 ]
kejiqing commented on SPARK-23563:
----------------------------------
a long term spark sql task, the meta space in driver will increase endless, and will lead full gc frequently.
a suitable cache size can slow down the speed of new class generate in jvm and rise the ratio of cache hit
> make the size fo cache in CodeGenerator configable
> --------------------------------------------------
>
> Key: SPARK-23563
> URL: https://issues.apache.org/jira/browse/SPARK-23563
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: kejiqing
> Priority: Minor
>
> the cache in class org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator has a hard cod maxmunSize 100, current code is:
>
> {code:java}
> // scala
> private val cache = CacheBuilder.newBuilder()
> .maximumSize(100)
> .build(
> new CacheLoader[CodeAndComment, (GeneratedClass, Int)]() {
> override def load(code: CodeAndComment): (GeneratedClass, Int) = {
> val startTime = System.nanoTime()
> val result = doCompile(code)
> val endTime = System.nanoTime()
> def timeMs: Double = (endTime - startTime).toDouble / 1000000
> CodegenMetrics.METRIC_SOURCE_CODE_SIZE.update(code.body.length)
> CodegenMetrics.METRIC_COMPILATION_TIME.update(timeMs.toLong)
> logInfo(s"Code generated in $timeMs ms")
> result
> }
> })
> {code}
> In some specific situation, for example: a long term and spark tasks are unchanged, the size of cache maximumSize configuration is a better idea.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org