[jira] [Commented] (SPARK-41690) Introduce AgnosticEncoders

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-41690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651389#comment-17651389 ] 

Apache Spark commented on SPARK-41690:
--------------------------------------

User 'hvanhovell' has created a pull request for this issue:
https://github.com/apache/spark/pull/39186

> Introduce AgnosticEncoders
> --------------------------
>
>                 Key: SPARK-41690
>                 URL: https://issues.apache.org/jira/browse/SPARK-41690
>             Project: Spark
>          Issue Type: Task
>          Components: Connect, SQL
>    Affects Versions: 3.4.0
>            Reporter: Herman van Hövell
>            Assignee: Herman van Hövell
>            Priority: Major
>
> For the Spark Connect Scala Client we need Encoders. We cannot use the current ExpressionEncoder infrastructure because of the following reasons:
>  * Connect uses Arrow instead of internal rows
>  * We don't want to have a dependency on Catalyst in the client.
>  * We don't want to be sending the expressions that expression encoders use over the wire.
> An additional challenge is that we cannot send TypeTags over the wire.
> The proposal is to create an intermediate encoder. This encoder describes exactly how which external type maps to an internal type, but it does not prescribe which format is used.
> An upside of this approach is that we can reduce the cost of encoder creation by quite a margin.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org