[jira] [Created] (SPARK-48868) Incorrect AnalysisException thrown using when() and mixed data types

["Tabrez Mohammed (Jira)" <ji...@apache.org>]

Tabrez Mohammed created SPARK-48868:
---------------------------------------

             Summary: Incorrect AnalysisException thrown using when() and mixed data types
                 Key: SPARK-48868
                 URL: https://issues.apache.org/jira/browse/SPARK-48868
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 3.4.0, 3.3.0
            Reporter: Tabrez Mohammed


Observe the following sample code, where I'm using when() based on typeof():
{code:java}
from pyspark.sql.types import (
    ArrayType,
    IntegerType,
    StringType,
    StructField,
    StructType,
)

schema = StructType(
    [
        StructField("ID", IntegerType(), nullable=False),
        StructField("name", StringType(), nullable=False),
        StructField("colors", ArrayType(StringType()), nullable=False),
    ]
)
data = [
    (1, "John", ["red", "blue", "green"]),
    (2, "Jane", ["yellow", "orange", "purple"]),
    (3, "Bob", ["black", "white"]),
    (4, "Alice", ["pink"]),
    (5, "Tom", ["brown", "gray"]),
]
df = spark.createDataFrame(data, schema)
col = "name"
df = df.withColumn(
    col,
    F.when(F.expr(f"typeof({col}) == 'string'"), F.trim(col))
    .when(
        F.expr(f"typeof({col}) LIKE 'array%'"),
        F.array_join(col, ","),
    )
    .otherwise(F.lit(None)),
)
{code}
 
Here's the exception I'm seeing:
{noformat}
pyspark.sql.utils.AnalysisException: cannot resolve 'array_join(name, ',')' due to data type mismatch: argument 1 requires array<string> type, however, 'name' is of string type.;
'Project [ID#0, CASE WHEN (typeof(name#1) = string) THEN trim(name#1, None) WHEN typeof(name#1) LIKE array% THEN array_join(name#1, ,, None) ELSE null END AS name#6, colors#2]
+- LogicalRDD [ID#0, name#1, colors#2], false
{noformat}
 

If I change col to "colors", I get this similar exception:
{noformat}
pyspark.sql.utils.AnalysisException: cannot resolve 'trim(colors)' due to data type mismatch: argument 1 requires string type, however, 'colors' is of array<string> type.;
'Project [ID#0, name#1, CASE WHEN (typeof(colors#2) = string) THEN trim(colors#2, None) WHEN typeof(colors#2) LIKE array% THEN array_join(colors#2, ,, None) ELSE null END AS colors#6]
+- LogicalRDD [ID#0, name#1, colors#2], false
{noformat}
 

It seems to try to evaluate all possible paths of code for type checking, even if that code path won't be hit for the current query. I was able to repro this on 3.3.0 and 3.4.0.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org