You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Matt Zhang (Jira)" <ji...@apache.org> on 2026/05/18 22:31:00 UTC
[jira] [Created] (SPARK-56931) Support ArrayType/MapType/StructType constant metadata in row materialization path
Matt Zhang created SPARK-56931:
----------------------------------
Summary: Support ArrayType/MapType/StructType constant metadata in row materialization path
Key: SPARK-56931
URL: https://issues.apache.org/jira/browse/SPARK-56931
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Matt Zhang
Fix For: 4.2.0, 5.0.0
Follow-up to SPARK-56844, which enabled ArrayType/MapType/StructType in FileSourceMetadataAttribute and added the populate() branches for ConstantColumnVector. That covered the columnar metadata path (ColumnarBatch output).
For file scans that produce row output (Batched=false: text, JSON, CSV, or any reader that does not vectorize), the metadata row is filled via FileFormat.updateMetadataInternalRow -> getFileConstantMetadataColumnValue -> Literal(extractor.apply(file)).
Literal.apply(Any) dispatches on the value class and has no case for ArrayData / MapData / InternalRow, so a complex constant metadata column trips UNSUPPORTED_FEATURE.LITERAL_TYPE before the row is populated.
Separately, SchemaPruning.sortLeftFieldsByRight recurses through the metadata schema and prunes nested struct fields inside an array/map/struct subfield. That is correct for data files (the reader projects the requested columns) but wrong for constant metadata, where each subfield is produced whole by a single extractor; pruning shaves catalyst row positions out from under the extractor.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org