You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Boyang Jerry Peng (Jira)" <ji...@apache.org> on 2024/11/27 23:41:00 UTC
[jira] [Commented] (SPARK-50437) Reduce overhead of creating deserializers in TransformWithStateExec
[ https://issues.apache.org/jira/browse/SPARK-50437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17901601#comment-17901601 ]
Boyang Jerry Peng commented on SPARK-50437:
-------------------------------------------
We can further optimize this to be done once per query.
Currently, the deserializers are created once per partition per batch.
We can improve this by creating the deserializers once per query run.
To do this, we need adjust certain APIs in CodeGenerator to allow
us to generate the code and compile it separately. We can
generate the java code on the driver side and compile it on the executor side.
Note that we only need to compile it once on the executor side since there is a cache
that allows us to reuse the compiled code.
> Reduce overhead of creating deserializers in TransformWithStateExec
> -------------------------------------------------------------------
>
> Key: SPARK-50437
> URL: https://issues.apache.org/jira/browse/SPARK-50437
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Boyang Jerry Peng
> Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org