[jira] [Assigned] (SPARK-55897) ColumnarRow.get() and ColumnarBatchRow.get() throw on UserDefinedType

["Huaxin Gao (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-55897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Huaxin Gao reassigned SPARK-55897:
----------------------------------

    Assignee: James Willis

> ColumnarRow.get() and ColumnarBatchRow.get() throw on UserDefinedType
> ---------------------------------------------------------------------
>
>                 Key: SPARK-55897
>                 URL: https://issues.apache.org/jira/browse/SPARK-55897
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.1.1
>         Environment: I don't think this is hardware-dependent but I discovered this on an M3 Macbook pro.
>            Reporter: James Willis
>            Assignee: James Willis
>            Priority: Major
>              Labels: pull-request-available
>
> {{ColumnarRow.get()}} and {{ColumnarBatchRow.get()}} do not handle {{{}UserDefinedType{}}}, throwing {{SparkUnsupportedOperationException("_LEGACY_ERROR_TEMP_3155")}} when a UDT field is accessed via the interpreted eval path (e.g., {{GetStructField.nullSafeEval}} on a nested struct from the vectorized Parquet reader).
> {code:java}
> org.apache.spark.SparkException: [INTERNAL_ERROR] Undefined error message parameter for error class:
> '_LEGACY_ERROR_TEMP_3155', MessageTemplate: Datatype not supported <dataType>, Parameters: Map()
>     at org.apache.spark.sql.vectorized.ColumnarRow.get(ColumnarRow.java:221)
>     at org.apache.spark.sql.catalyst.expressions.GetStructField.nullSafeEval(complexTypeExtractors.scala:207){code}
> *This happens when:*
>  # The vectorized Parquet reader produces a {{ColumnarBatch}}
>  # {{ColumnarToRowExec}} (in WSCG codegen mode) reads top-level columns via typed accessors ({{{}getArray{}}}, {{{}getBinary{}}}, etc.), which works fine
>  # But for *nested* structures (e.g., {{{}array[0].field{}}}), the top-level {{getArray()}} returns a {{{}ColumnarArray{}}}, and indexing into it returns a {{ColumnarRow}} — these remain as columnar objects, not copied to {{UnsafeRow}}
>  # A downstream expression that *can't be codegenned* (like {{InferredExpression}} in PR #611) falls back to {{{}eval(){}}}, which calls {{GetStructField.nullSafeEval()}} on the {{ColumnarRow}}
>  # {{GetStructField}} passes the UDT type (from the expression's schema) to {{{}ColumnarRow.get(){}}}, which doesn't handle UDT → crash
> *The codegen path never hits this* because {{CodeGenerator.getValue()}} (line 1683) resolves {{UserDefinedType }}to {{sqlType}} before generating code, so it generates {{getBinary()}} instead of {{{}get(ordinal, UDT){}}}.
> h3. Root Cause
> {{ColumnarRow.get()}} and {{ColumnarBatchRow.get()}} dispatch on {{dataType}} via {{instanceof}} checks for all concrete Spark types but have no branch for {{{}UserDefinedType{}}}. When {{{}GetStructField.nullSafeEval(){}}}passes a UDT type to {{{}get(){}}}, it falls through to the default error branch.
> The codegen path is unaffected because {{CodeGenerator.getValue()}} unwraps {{udt.sqlType()}} before generating type-specific accessor calls ({{{}getInt{}}}, {{{}getStruct{}}}, etc.), bypassing {{get()}} entirely. This is why the existing SPARK-39086 tests pass — they run through whole-stage codegen.
> The bug surfaces when the interpreted path is used (codegen disabled, codegen fallback, or exceeding {{{}spark.sql.codegen.maxFields{}}}).
> h3. Affected Code
>  * {{ColumnarRow.java:184-223}} — {{get(int ordinal, DataType dataType)}}
>  * {{ColumnarBatchRow.java:179-222}} — {{get(int ordinal, DataType dataType)}}
>  * {{ColumnarArray.java:215-217}} — {{get(int ordinal, DataType dataType)}}
>  *  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org