[jira] [Updated] (NIFI-15866) Inserting Date values in Iceberg tables results in Exception

["Sönke Liebau (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-15866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sönke Liebau updated NIFI-15866:
--------------------------------
    Description: 
The PutIcebergRecord processor currently does not support columns with type Date. 

When inserting in a table with a Date column, each time a ClassCastException between java.sql.Date (from the Flowfile) and java.time.LocalDate (required from PutIcebergRecord) is thrown.

This behavior occurs with Avro as well as Parquet and their specific readers configured in PutIcebergRecord, with a hardcorded Avro Schema referencing the Date column.

 

This bug is very similar to an already fixed bug regarding Datetimes/Timestamps: https://issues.apache.org/jira/browse/NIFI-15568.

 

*How to reproduce*
 * Create an Iceberg table with one Date column

 * Generate a record flowfile with GenerateFlowFile containing a value for this column, for example as CSV or JSON. Give an explicit Avro Schema as attribute in the GenerateFlowfile:

 

+CSV:+

 
{code:java}
dateCol
234234
{code}
 

 

+Avro Schema:+

 
{code:java}
{
  "type": "record",
  "name": "Document",
  "namespace": "com.example",
  "fields": [
    {
      "name": "dateCol",
      "type": [
        "null",
        {
          "type": "int",
          "logicalType": "date"
        }
      ]
    }
  ]
}{code}
 
 * Use a ConvertRecord to transform the CSV to an Avro flowfile, using a AvroRecordSetWriter with setting „Use ‚Schema Text‘ Property“
 * Write into Iceberg with PutIcebergRecord and a default AvroReader

 

Stacktrace: 
{code:java}
PutIcebergRecord[id=8e5f6f95-9ec0-37d7-ad34-9aa4500355f7] Write Rows to Table [xyz.test] failed FlowFile[filename=b13a8af8-851e-4f00-93dd-69f40a36e7b5]: java.lang.ClassCastException: class java.sql.Date cannot be cast to class java.time.LocalDate (java.sql.Date is in module java.sql of loader 'platform'; java.time.LocalDate is in module java.base of loader 'bootstrap') java.lang.ClassCastException: class java.sql.Date cannot be cast to class java.time.LocalDate (java.sql.Date is in module java.sql of loader 'platform'; java.time.LocalDate is in module java.base of loader 'bootstrap')at org.apache.iceberg.data.parquet.GenericParquetWriter$DateWriter.write(GenericParquetWriter.java:91)at org.apache.iceberg.parquet.ParquetValueWriters$OptionWriter.write(ParquetValueWriters.java:421)at org.apache.iceberg.parquet.ParquetValueWriters$StructWriter.write(ParquetValueWriters.java:665)at org.apache.iceberg.parquet.ParquetWriter.add(ParquetWriter.java:138)at org.apache.iceberg.io.DataWriter.write(DataWriter.java:71)at org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.write(BaseTaskWriter.java:401)at org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.write(BaseTaskWriter.java:384)at org.apache.iceberg.io.BaseTaskWriter$BaseRollingWriter.write(BaseTaskWriter.java:311)at org.apache.iceberg.io.UnpartitionedWriter.write(UnpartitionedWriter.java:42)at org.apache.nifi.services.iceberg.parquet.io.ParquetIcebergRowWriter.write(ParquetIcebergRowWriter.java:39)at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)at java.base/java.lang.reflect.Method.invoke(Unknown Source)at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:251)at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:237)at jdk.proxy515/jdk.proxy515.$Proxy706.write(Unknown Source)at org.apache.nifi.processors.iceberg.PutIcebergRecord.writeRecords(PutIcebergRecord.java:240)at org.apache.nifi.processors.iceberg.PutIcebergRecord.processFlowFiles(PutIcebergRecord.java:176)at org.apache.nifi.processors.iceberg.PutIcebergRecord.onTrigger(PutIcebergRecord.java:156)at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)at org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)at java.base/java.lang.Thread.run(Unknown Source) {code}

  was:
The PutIcebergRecord processor currently does not support columns with type Date. 

When inserting in a table with a Date column, each time a ClassCastException between java.sql.Date (from the Flowfile) and java.time.LocalDate (required from PutIcebergRecord) is thrown.

This behavior occurs with Avro as well as Parquet and their specific readers configured in PutIcebergRecord, with a hardcorded Avro Schema referencing the Date column.

 

This bug is very similar to an already fixed bug regarding Datetimes/Timestamps: https://issues.apache.org/jira/browse/NIFI-15568.

 

For reproducing the bug in NiFi:
 * Create an Iceberg table with one Date column

 * Generate a record flowfile with GenerateFlowFile containing a value for this column, for example as CSV or JSON. Give an explicit Avro Schema as attribute in the GenerateFlowfile:

 

+CSV:+

dateCol

234234

 

+Avro Schema:+

{

  "type": "record",

  "name": "Document",

  "namespace": "com.example",

  "fields": [

    {

      "name": "dateCol",

      "type": ["null", \{"type": "int", "logicalType": "date"} ]

    }

  ]

}
 * Use a ConvertRecord to transform the CSV to an Avro flowfile, using a AvroRecordSetWriter with setting „Use ‚Schema Text‘ Property“
 * Write into Iceberg with PutIcebergRecord and a default AvroReader

 

        Summary: Inserting Date values in Iceberg tables results in Exception  (was: nserting Date values in Iceberg tables )

> Inserting Date values in Iceberg tables results in Exception
> ------------------------------------------------------------
>
>                 Key: NIFI-15866
>                 URL: https://issues.apache.org/jira/browse/NIFI-15866
>             Project: Apache NiFi
>          Issue Type: Bug
>    Affects Versions: 2.9.0
>            Reporter: Sönke Liebau
>            Priority: Minor
>
> The PutIcebergRecord processor currently does not support columns with type Date. 
> When inserting in a table with a Date column, each time a ClassCastException between java.sql.Date (from the Flowfile) and java.time.LocalDate (required from PutIcebergRecord) is thrown.
> This behavior occurs with Avro as well as Parquet and their specific readers configured in PutIcebergRecord, with a hardcorded Avro Schema referencing the Date column.
>  
> This bug is very similar to an already fixed bug regarding Datetimes/Timestamps: https://issues.apache.org/jira/browse/NIFI-15568.
>  
> *How to reproduce*
>  * Create an Iceberg table with one Date column
>  * Generate a record flowfile with GenerateFlowFile containing a value for this column, for example as CSV or JSON. Give an explicit Avro Schema as attribute in the GenerateFlowfile:
>  
> +CSV:+
>  
> {code:java}
> dateCol
> 234234
> {code}
>  
>  
> +Avro Schema:+
>  
> {code:java}
> {
>   "type": "record",
>   "name": "Document",
>   "namespace": "com.example",
>   "fields": [
>     {
>       "name": "dateCol",
>       "type": [
>         "null",
>         {
>           "type": "int",
>           "logicalType": "date"
>         }
>       ]
>     }
>   ]
> }{code}
>  
>  * Use a ConvertRecord to transform the CSV to an Avro flowfile, using a AvroRecordSetWriter with setting „Use ‚Schema Text‘ Property“
>  * Write into Iceberg with PutIcebergRecord and a default AvroReader
>  
> Stacktrace: 
> {code:java}
> PutIcebergRecord[id=8e5f6f95-9ec0-37d7-ad34-9aa4500355f7] Write Rows to Table [xyz.test] failed FlowFile[filename=b13a8af8-851e-4f00-93dd-69f40a36e7b5]: java.lang.ClassCastException: class java.sql.Date cannot be cast to class java.time.LocalDate (java.sql.Date is in module java.sql of loader 'platform'; java.time.LocalDate is in module java.base of loader 'bootstrap') java.lang.ClassCastException: class java.sql.Date cannot be cast to class java.time.LocalDate (java.sql.Date is in module java.sql of loader 'platform'; java.time.LocalDate is in module java.base of loader 'bootstrap')at org.apache.iceberg.data.parquet.GenericParquetWriter$DateWriter.write(GenericParquetWriter.java:91)at org.apache.iceberg.parquet.ParquetValueWriters$OptionWriter.write(ParquetValueWriters.java:421)at org.apache.iceberg.parquet.ParquetValueWriters$StructWriter.write(ParquetValueWriters.java:665)at org.apache.iceberg.parquet.ParquetWriter.add(ParquetWriter.java:138)at org.apache.iceberg.io.DataWriter.write(DataWriter.java:71)at org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.write(BaseTaskWriter.java:401)at org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.write(BaseTaskWriter.java:384)at org.apache.iceberg.io.BaseTaskWriter$BaseRollingWriter.write(BaseTaskWriter.java:311)at org.apache.iceberg.io.UnpartitionedWriter.write(UnpartitionedWriter.java:42)at org.apache.nifi.services.iceberg.parquet.io.ParquetIcebergRowWriter.write(ParquetIcebergRowWriter.java:39)at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)at java.base/java.lang.reflect.Method.invoke(Unknown Source)at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:251)at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:237)at jdk.proxy515/jdk.proxy515.$Proxy706.write(Unknown Source)at org.apache.nifi.processors.iceberg.PutIcebergRecord.writeRecords(PutIcebergRecord.java:240)at org.apache.nifi.processors.iceberg.PutIcebergRecord.processFlowFiles(PutIcebergRecord.java:176)at org.apache.nifi.processors.iceberg.PutIcebergRecord.onTrigger(PutIcebergRecord.java:156)at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)at org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)at java.base/java.lang.Thread.run(Unknown Source) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)