You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2024/03/11 00:22:00 UTC

[jira] [Resolved] (SPARK-47331) Serialization using case classes/primitives/POJO based on SQL encoder for Arbitrary State API v2.

     [ https://issues.apache.org/jira/browse/SPARK-47331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim resolved SPARK-47331.
----------------------------------
    Fix Version/s: 4.0.0
       Resolution: Fixed

Issue resolved by pull request 45447
[https://github.com/apache/spark/pull/45447]

> Serialization using case classes/primitives/POJO based on SQL encoder for Arbitrary State API v2. 
> --------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-47331
>                 URL: https://issues.apache.org/jira/browse/SPARK-47331
>             Project: Spark
>          Issue Type: Task
>          Components: Structured Streaming
>    Affects Versions: 4.0.0
>            Reporter: Jing Zhan
>            Assignee: Jing Zhan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>
> In the new operator for arbitrary state-v2, we cannot rely on the session/encoder being available since the initialization for the various state instances happens on the executors. Also, we can only support limited state types with the available encoders. Hence, for the state serialization, we propose to serialize primitives/case classes/POJO with SQL encoder. Leveraging SQL encoder can speed up the serialization.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org