[jira] [Updated] (SPARK-48275) Improve array_sort documentation for default comparator

["Matt Braymer-Hayes (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-48275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Braymer-Hayes updated SPARK-48275:
---------------------------------------
    Summary: Improve array_sort documentation for default comparator  (was: array_sort and sort_array fail for structs containing any unorderable fields)

> Improve array_sort documentation for default comparator
> -------------------------------------------------------
>
>                 Key: SPARK-48275
>                 URL: https://issues.apache.org/jira/browse/SPARK-48275
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.5.0
>         Environment: Running in Databricks on [Databricks Runtime 15.1|https://docs.databricks.com/en/release-notes/runtime/15.1.html].
>            Reporter: Matt Braymer-Hayes
>            Priority: Critical
>
> {{When array_sort()}} and {{sort_array()}} are used on arrays of structs, the first field is used to determine ordering. Unfortunately, it appears that the current implementation requires _all_ fields to be orderable. Here's a minimal example:
>  
> {code:java}
> import pyspark.sql.functions as F
> import pyspark.sql.types as T
> schema = T.StructType([
>     T.StructField(
>         'value',
>         T.ArrayType(
>             T.StructType([
>                 T.StructField('order', T.IntegerType(), True),
>                 T.StructField('values', T.MapType(T.StringType(), T.StringType(), True), True), # remove this field and both commands below succeed
>             ]),
>             False
>         ),
>         False
>     )
> ])
> df = spark.createDataFrame([], schema=schema)
> df.select(F.array_sort(df['value'])).collect()
> df.select(F.sort_array(df['value'])).collect(){code}
>  
> {{array_sort()}} output:
>  
> {code:java}
> [DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] Cannot resolve "(namedlambdavariable() < namedlambdavariable())" due to data type mismatch: The `<` does not support ordering on type "STRUCT<comparable: INT, uncomparable: MAP<STRING, STRING>>". SQLSTATE: 42K09 {code}
>  
>  
> {{sort_array()}} output:
>  
> {code:java}
> [DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] Cannot resolve "sort_array(value, true)" due to data type mismatch: The `sort_array` does not support ordering on type "ARRAY<STRUCT<comparable: INT, uncomparable: MAP<STRING, STRING>>>". SQLSTATE: 42K09 {code}
>  
>  
> I would expect the behavior to be different: this error should be raised if the _first field_ is not orderable, otherwise it should succeed.
>  
> With the implementation as-is, I can't sort arrays of structs that contain maps or other unorderable fields. The only workaround is painful: you'd have to store the maps as a separate array, store the map array index on the struct, and after sorting add the map to the struct using {{transform()}} and {{{}element_at(){}}}.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org