You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Herman van Hövell (Jira)" <ji...@apache.org> on 2025/03/01 01:04:00 UTC
[jira] [Resolved] (SPARK-49960) Support end DeveloperAPI provided AgnosticEncoders based on ExpressionEncoders to allow migration
[ https://issues.apache.org/jira/browse/SPARK-49960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Herman van Hövell resolved SPARK-49960.
---------------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 50023
[https://github.com/apache/spark/pull/50023]
> Support end DeveloperAPI provided AgnosticEncoders based on ExpressionEncoders to allow migration
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-49960
> URL: https://issues.apache.org/jira/browse/SPARK-49960
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Chris Twiner
> Assignee: Chris Twiner
> Priority: Major
> Labels: Encoding, pull-request-available
> Fix For: 4.0.0
>
>
> In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used throughout the original api. SPARK-49025 removed re-use of ExpressionEncoders for derivation of tuples rendering user provided ExpressionEncoders such as Frameless incompatible with joinWith (which only uses the AgnosticEncoder hierarchy). There are other likely combinations where the encoder is not passed into the function.
> This patch aims to add an extension AgnosticExpressionPathEncoder trait with added:
> {code:java}
> def toCatalyst(input: Expression): Expression
> def fromCatalyst(inputPath: Expression): Expression {code}
> functions mirroring the underlying Des/SerializerBuildHelper approach (also used by Frameless). In turn the BuildHelper's delegate to these functions when the AgnosticEncoder is an AgnosticExpressionPathEncoder.
> Moreover it aims to fix AgnosticEncoder issues identified with moving Frameless to AgnosticEncoders instead of ExpressionEncoders allowing users to migrate via both the existing Expression codebase on 4.0 and future use of AgnosticEncoders (limited in the short term to Classic as serialization of derived codecs requires closures)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org