[jira] [Commented] (NIFI-16074) PutIcebergRecord missing convertors

["Henri Saksi (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121965#comment-18121965 ] 

Henri Saksi commented on NIFI-16074:
------------------------------------

The timestamp, list and struct parts of this issue were addressed in NIFI-16069 (2.12.0). Byte values are still passed to Iceberg unconverted: writing to a {{binary}} column fails with {{ClassCastException: [Ljava.lang.Object; cannot be cast to java.nio.ByteBuffer}} (seen on 2.12.0 with JsonTreeReader and an Avro {{bytes}} field), and {{fixed}} columns fail similarly. On main (Iceberg 1.12), String values for {{uuid}} columns also fail in the Parquet writer with {{String cannot be cast to UUID}}, unless the reader already provides {{java.util.UUID}} values.

The same conversion is now needed for {{geometry}} and {{geography}} columns: Iceberg 1.12 ([apache/iceberg#16982|https://github.com/apache/iceberg/pull/16982], upgraded in NIFI-16407) writes both from WKB in a {{ByteBuffer}}.

I opened [PR #11749|https://github.com/apache/nifi/pull/11749], which converts {{binary}}, {{fixed}}, {{uuid}}, {{geometry}} and {{geography}} values by the target column type in {{RecordConverter}}, including values nested in structs, lists and maps. Its tests include writing through the Parquet writer to a format-version 3 table with geometry and geography columns. If you'd prefer the geospatial types under a separate issue, I can split them out.


> PutIcebergRecord missing convertors
> -----------------------------------
>
>                 Key: NIFI-16074
>                 URL: https://issues.apache.org/jira/browse/NIFI-16074
>             Project: Apache NiFi
>          Issue Type: Bug
>            Reporter: Nir Yanay
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> The following Iceberg types currently don't appear to be handled correctly: * {{timestampz}}
>  * {{bytes}}
>  * {{list}}
>  * {{struct}}
> None of these have conversion logic implemented, which results in {{ClassCastException}} errors.
> Additionally, the UUID conversion should use {{{}UUID.fromString(){}}}, but it currently doesn't have its own converter, although it does not fail like the others.
>  
> I opened a PR:
> [https://github.com/apache/nifi/pull/11423]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)