[jira] [Updated] (NIFI-4844) AvroRecordSetWriter should be able to convert a double having less scale than intended target Avro schema instead of throwing an AvroTypeException

["Koji Kawamura (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Koji Kawamura updated NIFI-4844:
--------------------------------
    Status: Patch Available  (was: In Progress)

> AvroRecordSetWriter should be able to convert a double having less scale than intended target Avro schema instead of throwing an AvroTypeException
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-4844
>                 URL: https://issues.apache.org/jira/browse/NIFI-4844
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Extensions
>    Affects Versions: 1.3.0
>            Reporter: Koji Kawamura
>            Assignee: Koji Kawamura
>            Priority: Major
>
> Current AvroTypeUtil conversion logic can throw AvroTypeException when it maps double values into Avro decimal logical type fields if the double value has less scale than the one defined at the target Avro decimal field schema.
> For example, with following schema:
> {code}
> {
>   "type": "record",
>   "name": "logicalDecimalTest",
>   "fields": [
>     {"name": "id", "type": "int"},
>     {"name": "name", "type": "string"},
>     {
>       "name": "price",
>       "type": {
>         "type": "bytes",
>         "logicalType": "decimal",
>         "precision": 18,
>         "scale": 8
>       }}]}
> {code}
> And following CSV records:
> {code}
> id|name|price
> 1|one|1.23
> 2|two|2.34
> {code}
> Would produce this Exception:
> {code}
> 2018-02-06 09:57:27,461 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.ConvertRecord ConvertRecord[id=6897bc30-0161-1000-a8e7-9ce0ce8eb9ae] Failed to process StandardFlowFileRecord[uuid=a97366a0-79bb-42ff-9023-c5d62ecfdbc5,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1517878123416-2, container=default, section=2], offset=5, length=48],offset=0,name=220105646548465,size=48]; will route to failure: org.apache.avro.AvroTypeException: Cannot encode decimal with scale 17 as scale 8
> org.apache.avro.AvroTypeException: Cannot encode decimal with scale 17 as scale 8
>         at org.apache.avro.Conversions$DecimalConversion.toBytes(Conversions.java:86)
>         at org.apache.nifi.avro.AvroTypeUtil.convertToAvroObject(AvroTypeUtil.java:546)
>         at org.apache.nifi.avro.AvroTypeUtil.createAvroRecord(AvroTypeUtil.java:457)
>         at org.apache.nifi.avro.WriteAvroResultWithExternalSchema.writeRecord(WriteAvroResultWithExternalSchema.java:76)
>         at org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
>         at org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:122)
>         at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2827)
>         at org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:109)
>         at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>         at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>         at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> The same issue is reported in the Avro project, [AVRO-1864|https://issues.apache.org/jira/browse/AVRO-1864]. The recommended approach is to adjust the scale at NiFi side. Actually, for BigDecimal input values, NiFi already does this, but not with double values. AvroTypeUtil should do the same scale adjustment for double values, too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)