[jira] [Commented] (NIFI-16069) PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records)

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-16069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113000#comment-18113000 ] 

ASF subversion and git services commented on NIFI-16069:
--------------------------------------------------------

Commit be626038b3e668c8ba69c2ddce98141e0c2793fa in nifi's branch refs/heads/main from maltesander
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=be626038b3e ]

NIFI-16069 Improved PutIcebergRecord support for complex types and Timestamps (#11391)

Signed-off-by: David Handermann <ex...@apache.org>

> PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records)
> --------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-16069
>                 URL: https://issues.apache.org/jira/browse/NIFI-16069
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 2.10.0
>            Reporter: Sönke Liebau
>            Priority: Minor
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> PutIcebergRecord fails to write FlowFiles whose schema contains complex/nested types like Iceberg list, map, or struct columns. RecordConverter only translates top-level scalar values (java.sql timestamp/date/time -> java.time) and passes complex values through unchanged. 
> As a result, values reach Iceberg's Parquet writer in NiFi's native representation, which is incompatible with what Iceberg expects:
>  * Nested records arrive as org.apache.nifi.serialization.record.MapRecord but Iceberg requires org.apache.iceberg.StructLike.
>  * Array fields arrive as Object[] but Iceberg's writer requires a java.util.Collection.
>  * Maps and elements/values nested inside these types are likewise not converted (e.g. a date inside an array or map value).
> Because conversion is gated on scalar field types only, records consisting solely of complex fields skipp conversion entirely.
> h3. Steps to reproduce
>  # Create an Iceberg table with a complex column, e.g. a struct (nested record), a list<...> (array), or a map<...>.
>  # Configure a PutIcebergRecord processor pointing at that table with a matching record reader schema.
>  # Send a FlowFile containing a record with a value for the complex column (e.g. a nested record for the struct, or an array for the list).
>  # Observed: the FlowFile routes to failure with a ClassCastException (MapRecord -> StructLike for structs, Object[] -> Collection for arrays).
>  # Expected: the record is written to the Iceberg table successfully.
> h3. Root cause
> RecordConverter performs only shallow, scalar-only conversion and has no knowledge of the target Iceberg types, so complex values are passed through in NiFi's native form.
> h3. Proposed Fix
> Make RecordConverter recursive and Iceberg-schema-aware. Convert arrays/collections to List, wrap nested records as Iceberg StructLike (DelegatedRecord), and convert maps, recursing into element/key/value types so scalar conversions still apply at any depth.
> DelegatedRecord now passes the target StructType into the converter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)