You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/09/04 04:11:00 UTC

[jira] [Updated] (SPARK-49307) Support Kryo Serialization with AgnosticEncoders

     [ https://issues.apache.org/jira/browse/SPARK-49307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-49307:
-----------------------------------
    Labels: pull-request-available  (was: )

> Support Kryo Serialization with AgnosticEncoders
> ------------------------------------------------
>
>                 Key: SPARK-49307
>                 URL: https://issues.apache.org/jira/browse/SPARK-49307
>             Project: Spark
>          Issue Type: New Feature
>          Components: Connect
>    Affects Versions: 4.0.0
>            Reporter: Herman van Hövell
>            Priority: Major
>              Labels: pull-request-available
>
> Add support for Kryo based serialization to Agnostic encoders. This will allow us to port the entire Encoders class from sql/core to sql/api.
> Unfortunately supporting connect is not really possible at this moment. We cannot share the configuration of the Kryo objects between the server and connect. This is not possible due to the fact that connect - by design - does not have a all classes needed on its classpath. This makes constructing the same configuration (with the same class ids) almost impossible. On top of this, backwards compatibility will be a problem.
> For connect the only way forward is to have a separately configured version of the kryo serializer that leverages hard coded class ids. That is probably going to need some form of configurability.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org