You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Herman van Hövell (Jira)" <ji...@apache.org> on 2024/12/03 15:46:00 UTC

[jira] [Assigned] (SPARK-49570) Use agnostic encoders instead of expression encoders in user facing API

     [ https://issues.apache.org/jira/browse/SPARK-49570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Herman van Hövell reassigned SPARK-49570:
-----------------------------------------

    Assignee: Herman van Hövell

> Use agnostic encoders instead of expression encoders in user facing API
> -----------------------------------------------------------------------
>
>                 Key: SPARK-49570
>                 URL: https://issues.apache.org/jira/browse/SPARK-49570
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: Herman van Hövell
>            Assignee: Herman van Hövell
>            Priority: Major
>
> ExpressionEncoders are quite heavily used in the classic Dataset and KeyValueGroupedDataset. Because of the move to sql/api most of the user provided are in fact AgnosticEncoders, this is mostly fine but in some cases this leads to situations where we try to combine the two which does not work.
> Actual ExpressionEncoders are generally only needed inside the engine, or for collecting results. We should try to limit them to these use cases.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org