[jira] [Updated] (SPARK-55056) toPandas() crashes with SIGSEGV on nested empty arrays

["Yicong Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-55056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yicong Huang updated SPARK-55056:
---------------------------------
    Description: 
{{toPandas()}} crashes with SIGSEGV when a DataFrame contains nested array types (depth >= 3) with an empty outer array.
{code:python}
schema = StructType([
    StructField("data", ArrayType(StructType([
        StructField("arr", ArrayType(StructType([
            StructField("inner", ArrayType(StringType()))
        ])))
    ])))
])
df = spark.createDataFrame([Row(data=[])], schema=schema)
df.toPandas()  # SIGSEGV
{code}
Arrow format requires ListArray offset buffer to have N+1 entries. Even when N=0, the buffer must contain {{{}[0]{}}}. When the outer array is empty, nested {{ArrayWriters are never invoked, so their count}} stays 0. Then {{getBufferSizeFor(0)}} returns 0, and the offset buffer is omitted in IPC serialization — violating Arrow spec.
{code:scala}
// ArrayWriter.scala - current behavior
override def setValue(...): Unit = {
  while (i < array.numElements()) {  // never runs when empty
    elementWriter.write(array, i)    // nested writer never called
  }
}
{code}

  was:
{{toPandas()}} crashes with SIGSEGV when a DataFrame contains nested array types (depth >= 3) with an empty outer array.

{code:python}
schema = StructType([
    StructField("data", ArrayType(StructType([
        StructField("arr", ArrayType(StructType([
            StructField("inner", ArrayType(StringType()))
        ])))
    ])))
])
df = spark.createDataFrame([Row(data=[])], schema=schema)
df.toPandas()  # SIGSEGV
{code}

Arrow format requires ListArray offset buffer to have N+1 entries. Even when N=0, the buffer must contain {{\[0\]}}. When the outer array is empty, nested {{ArrayWriter}}s are never invoked, so their {{count}} stays 0. Then {{getBufferSizeFor(0)}} returns 0, and the offset buffer is omitted in IPC serialization — violating Arrow spec.

{code:scala}
// ArrayWriter.scala - current behavior
override def setValue(...): Unit = {
  while (i < array.numElements()) {  // never runs when empty
    elementWriter.write(array, i)    // nested writer never called
  }
}
{code}


> toPandas() crashes with SIGSEGV on nested empty arrays
> ------------------------------------------------------
>
>                 Key: SPARK-55056
>                 URL: https://issues.apache.org/jira/browse/SPARK-55056
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 4.2.0
>            Reporter: Yicong Huang
>            Priority: Critical
>
> {{toPandas()}} crashes with SIGSEGV when a DataFrame contains nested array types (depth >= 3) with an empty outer array.
> {code:python}
> schema = StructType([
>     StructField("data", ArrayType(StructType([
>         StructField("arr", ArrayType(StructType([
>             StructField("inner", ArrayType(StringType()))
>         ])))
>     ])))
> ])
> df = spark.createDataFrame([Row(data=[])], schema=schema)
> df.toPandas()  # SIGSEGV
> {code}
> Arrow format requires ListArray offset buffer to have N+1 entries. Even when N=0, the buffer must contain {{{}[0]{}}}. When the outer array is empty, nested {{ArrayWriters are never invoked, so their count}} stays 0. Then {{getBufferSizeFor(0)}} returns 0, and the offset buffer is omitted in IPC serialization — violating Arrow spec.
> {code:scala}
> // ArrayWriter.scala - current behavior
> override def setValue(...): Unit = {
>   while (i < array.numElements()) {  // never runs when empty
>     elementWriter.write(array, i)    // nested writer never called
>   }
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org