[jira] [Created] (SPARK-60095) Interpreted MAX/MIN over collated strings return a wrong value

["Yaoxuan Wu (Jira)" <ji...@apache.org>]

Yaoxuan Wu created SPARK-60095:
----------------------------------

             Summary: Interpreted MAX/MIN over collated strings return a wrong value
                 Key: SPARK-60095
                 URL: https://issues.apache.org/jira/browse/SPARK-60095
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.1.1, 4.2.0, 4.0.0, 5.0.0
            Reporter: Yaoxuan Wu


{code:java}
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[1]").getOrCreate()   # all rows in one partition
df = spark.createDataFrame([("abcd",), ("xy",), ("ab",)], "s STRING COLLATE UTF8_LCASE")print(df.agg(F.max("s"), F.min("s")).collect())
# [Row(max(s)='xy', min(s)='ab')]: correct with codegenspark.conf.set("spark.sql.codegen.wholeStage", "false")
spark.conf.set("spark.sql.codegen.factoryMode", "NO_CODEGEN")
print(df.agg(F.max("s"), F.min("s")).collect())
# expected:    [Row(max(s)='xy', min(s)='ab')]
# Spark 4.2.0: [Row(max(s)='ab', min(s)='ab')] {code}
Same result on 4.1.1, 4.2.0 and master (4.0.0 fails the same way; use a CAST to UTF8_BINARY in the output to see it, since collecting a collated column returns no rows on 4.0.0). It happens for UTF8_LCASE, UTF8_LCASE_RTRIM and UNICODE_CI, not for the default UTF8_BINARY. With more rows the result can also be corrupted bytes (we saw 'Ssv\x00'), i.e. the buffer points into memory that was overwritten. The rows must be in one partition.

*Cause:* InternalRow.getWriter copies strings with case StringType => ... v.asInstanceOf[UTF8String].copy(). That pattern only matches the default StringType object; a collated StringType(id) falls through to case _ => input.update(ordinal, v) and is stored without a copy. Fix: case _: StringType =>.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org