[jira] [Created] (SPARK-41690) Introduce AgnosticEncoders

["Herman van Hövell (Jira)" <ji...@apache.org>]

Herman van Hövell created SPARK-41690:
-----------------------------------------

             Summary: Introduce AgnosticEncoders
                 Key: SPARK-41690
                 URL: https://issues.apache.org/jira/browse/SPARK-41690
             Project: Spark
          Issue Type: Task
          Components: Connect, SQL
    Affects Versions: 3.4.0
            Reporter: Herman van Hövell
            Assignee: Herman van Hövell


For the Spark Connect Scala Client we need Encoders. We cannot use the current ExpressionEncoder infrastructure because of the following reasons:
 * Connect uses Arrow instead of internal rows
 * We don't want to have a dependency on Catalyst in the client.
 * We don't want to be sending the expressions that expression encoders use over the wire.

An additional challenge is that we cannot send TypeTags over the wire.

The proposal is to create an intermediate encoder. This encoder describes exactly how which external type maps to an internal type, but it does not prescribe which format is used.

An upside of this approach is that we can reduce the cost of encoder creation by quite a margin.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org