You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Simon Bence (Jira)" <ji...@apache.org> on 2020/06/03 06:37:00 UTC
[jira] [Closed] (NIFI-7369) Fixing precision of floating point
numbers when reading Avro record
[ https://issues.apache.org/jira/browse/NIFI-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Bence closed NIFI-7369.
-----------------------------
> Fixing precision of floating point numbers when reading Avro record
> -------------------------------------------------------------------
>
> Key: NIFI-7369
> URL: https://issues.apache.org/jira/browse/NIFI-7369
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.9.2
> Reporter: Simon Bence
> Assignee: Simon Bence
> Priority: Major
> Fix For: 1.12.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> h3. Summary:
> When ConvertRecord reads in decimal (logical) data type from Avro record, it converts it into Java Double, which depending on the decimal precision, might cause precision loss.
> h3. In details:
> AbstractRecordProcessor (parent of ConvertRecord) works in a way it reads in record with RecordReader and writes it out with RecordSetWriter. Between the two steps, we have an internal representation uses RecordFieldTypes. In this example, the following transformations are happening: AVRO into internal Record. Internal Record into the format defined by the writer. The specific reader where the issue comes up is the AvroReaderWithEmbeddedSchema. This (based on the embedded schema) tries to figure out what could be the proper internal representation for the given fields, using AvroTypeUtil#determineDataType. This util, with the type definition specifies decimal [ends up|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java#L343-L346] using double (the originally read data with the reader is BigDecimal and it is converted). Double is not capable to correctly represent the incoming data when it is high precision so in further steps, a less accurate number will be used and written into the output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)