[jira] [Updated] (SPARK-59151) Implement the CanonicalizeVariant Catalyst expression

["Stefan Savic (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stefan Savic updated SPARK-59151:
---------------------------------
    Summary: Implement the CanonicalizeVariant Catalyst expression  (was: Variant aggregation- canonicalization)

> Implement the CanonicalizeVariant Catalyst expression
> -----------------------------------------------------
>
>                 Key: SPARK-59151
>                 URL: https://issues.apache.org/jira/browse/SPARK-59151
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Aleksandr Chernousov
>            Priority: Major
>              Labels: pull-request-available
>
> Implement canonicalization (+ fastpath aka checking if Variant is already canonicalized)
> This part contains only canonicalization, without wiring to expressions
> After canonicalization Variant should have
> - All dict keys sorted
> - No unused keys
> - All numeric types normalized to smallest fitting datatype (e.g.: [10 as int64] and [10.0 as decimal] to 10 as int8
> - float/double -0.0 converted to 0.0
> - Have all NaNs in normalised form



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org