You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2026/05/14 01:17:00 UTC
[jira] [Assigned] (SPARK-56844) Support ArrayType / MapType / StructType in ConstantColumnVector and FileSourceMetadataAttribute
[ https://issues.apache.org/jira/browse/SPARK-56844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-56844:
-----------------------------------
Assignee: Matt Zhang
> Support ArrayType / MapType / StructType in ConstantColumnVector and FileSourceMetadataAttribute
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-56844
> URL: https://issues.apache.org/jira/browse/SPARK-56844
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Matt Zhang
> Assignee: Matt Zhang
> Priority: Minor
> Labels: pull-request-available
>
> `FileSourceMetadataAttribute.isSupportedType` currently rejects ArrayType, MapType, and StructType, citing `ColumnVectorUtils.populate` as the limiting factor. As a result, file source constant metadata columns cannot use complex types, even though all the underlying machinery (`RowToColumnConverter`, `OffHeapColumnVector` array/map layout) supports them.
> This issue tracks broadening that gate, implementing the missing populate and scatter paths in `ColumnVectorUtils.populate` and `ConstantColumnVector.writeToOffHeapColumnVector`, and enabling complex constants end to end.
> Specifically:
> - `FileSourceMetadataAttribute.isSupportedType` allows array/map/struct recursively, contingent on element types also being supported.
> - `ColumnVectorUtils.populate` gains struct/array/map branches. Array and map allocate a one-row off-heap backing vector and reuse the existing `RowToColumnConverter` to write the constant value with full recursive type support.
> - `ConstantColumnVector` gains optional ownership of a backing `WritableColumnVector` (closed by `close()`), and `writeToOffHeapColumnVector` gains array/map branches that copy the M constant elements once into the target child vector and write per-row `(offset=0, length=M)`.
> No user-facing API changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org