You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (JIRA)" <ji...@apache.org> on 2018/07/03 11:30:00 UTC
[jira] [Comment Edited] (SPARK-24727) The cache 100 in
CodeGenerator is too small for streaming
[ https://issues.apache.org/jira/browse/SPARK-24727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531225#comment-16531225 ]
Wenchen Fan edited comment on SPARK-24727 at 7/3/18 11:29 AM:
--------------------------------------------------------------
BTW this needs to be a static conf. the CodeGenerator object is per JVM.
was (Author: cloud_fan):
BTW this needs to be a static SQL. the CodeGenerator object is per JVM.
> The cache 100 in CodeGenerator is too small for streaming
> ---------------------------------------------------------
>
> Key: SPARK-24727
> URL: https://issues.apache.org/jira/browse/SPARK-24727
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: ant_nebula
> Priority: Major
>
> {code:java}
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator
> private val cache = CacheBuilder.newBuilder().maximumSize(100).build{code}
> The cache 100 in CodeGenerator is too small for realtime streaming calculation, although is ok for offline calculation. Because realtime streaming calculation is mostly more complex in one driver, and performance sensitive.
> I suggest spark support configging for user with default 100, such as spark.codegen.cache=1000
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org