You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Herman van Hövell (Jira)" <ji...@apache.org> on 2024/08/19 20:28:00 UTC
[jira] [Created] (SPARK-49307) Support Kryo Serialization with AgnosticEncoders
Herman van Hövell created SPARK-49307:
-----------------------------------------
Summary: Support Kryo Serialization with AgnosticEncoders
Key: SPARK-49307
URL: https://issues.apache.org/jira/browse/SPARK-49307
Project: Spark
Issue Type: New Feature
Components: Connect
Affects Versions: 4.0.0
Reporter: Herman van Hövell
Add support for Kryo based serialization to Agnostic encoders. This will allow us to port the entire Encoders class from sql/core to sql/api.
Unfortunately supporting connect is not really possible at this moment. We cannot share the configuration of the Kryo objects between the server and connect. This is not possible due to the fact that connect - by design - does not have a all classes needed on its classpath. This makes constructing the same configuration (with the same class ids) almost impossible. On top of this, backwards compatibility will be a problem.
For connect the only way forward is to have a separately configured version of the kryo serializer that leverages hard coded class ids. That is probably going to need some form of configurability.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org