[jira] [Commented] (SPARK-40770) Improved error messages for applyInPandas for schema mismatch

["Bill Schneider (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-40770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012440#comment-18012440 ] 

Bill Schneider commented on SPARK-40770:
----------------------------------------

I'm still seeing a variation of this on Spark 3.5+ and 4.0, when I accidentally had the same field name repeated twice in the StructType.

Example

 
{code:java}
@pandas_udf(returnType=StructType([
    StructField("foo", DoubleType()),
    StructField("foo", DoubleType())]))
def calculate_udf_fail(input: pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame({
        "foo": input['foo']
   })
spark = SparkSession.builder \
    .appName("Test UDF") \
    .getOrCreate()

spark.sql("select 1 as foo").select(calculate_udf_fail(struct("foo")).alias("result")).show() {code}
Repeating the same column twice in the returnType's `StructType` causes this cryptic error

java.lang.IllegalArgumentException: not all nodes, buffers and variadicBufferCounts were consumed. nodes: [ArrowFieldNode [length=1, nullCount=0]] buffers: [ArrowBuf[10], address:140498365661192, capacity:0, ArrowBuf[11], address:140498365661192, capacity:8] variadicBufferCounts: []

Of course if I have a duplicate field in the StructType (probably from copy-paste or Copilot) that's a problem I need to fix - but the error message gives no indication about the underlying issue.

 

> Improved error messages for applyInPandas for schema mismatch
> -------------------------------------------------------------
>
>                 Key: SPARK-40770
>                 URL: https://issues.apache.org/jira/browse/SPARK-40770
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 3.4.0
>            Reporter: Enrico Minack
>            Assignee: Enrico Minack
>            Priority: Minor
>             Fix For: 3.5.0
>
>
> Error messages raised by `applyInPandas` and `mapInPadnas` are very generic or useless when used with complex schemata:
> {code}
> KeyError: 'val'
> {code}
> {code}
> RuntimeError: Number of columns of the returned pandas.DataFrame doesn't match specified schema. Expected: 2 Actual: 3
> {code}
> {code}
> java.lang.IllegalArgumentException: not all nodes and buffers were consumed. nodes: [ArrowFieldNode [length=3, nullCount=0]] buffers: [ArrowBuf[304], address:139860828549160, length:0, ArrowBuf[305], address:139860828549160, length:24]
> {code}
> {code}
> pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64
> {code}
> {code}
> pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double
> {code}
> These should be improved by adding column names or descriptive messages (in the same order as above):
> {code}
> RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema.  Missing: val  Unexpected: v  Schema: id, val
> {code}
> {code}
> RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema.  Missing: val  Unexpected: foo, v  Schema: id, val
> {code}
> {code}
> RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema.  Unexpected: v  Schema: id, id
> {code}
> {code}
> pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64
> The above exception was the direct cause of the following exception:
> TypeError: Exception thrown when converting pandas.Series (int64) with name 'val' to Arrow Array (string).
> {code}
> {code}
> pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double
> The above exception was the direct cause of the following exception:
> ValueError: Exception thrown when converting pandas.Series (object) with name 'val' to Arrow Array (double).
> {code}
> When no column names are given, the following error was returned:
> {code}
> RuntimeError: Number of columns of the returned pandas.DataFrame doesn't match specified schema. Expected: 2 Actual: 3
> {code}
> Where it should contain the output schema:
> {code}
> RuntimeError: Number of columns of the returned pandas.DataFrame doesn't match specified schema.  Expected: 2  Actual: 3  Schema: id, val
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org