You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2020/02/19 21:31:55 UTC
NIFI Bug with Convert Record - 999999.99 changed to 1000000.0
Hi,
I am using below text to convert CSV to JSON using convert record
processor. Value of 999999.99 changed to 1000000.0 , May I know how is this
possible?
OPERATING_UNIT|TEST_34_2|TEST_34_15|TEST_34_5
"141516"|"1.1"|"1.1"|"1.1"
"141517"|"1.11"|"1.11"|"1.11"
"141518"|"1101.11"|"999999.99"|"1101.11"
[image: image.png]
Is this happening with CSV reader or JSON record set writer?
Thanks,
Asmath
Re: NIFI Bug with Convert Record - 999999.99 changed to 1000000.0
Posted by KhajaAsmath Mohammed <md...@gmail.com>.
Thanks Mark, this is helpful. I found out some more issues. Let me circle back with test data.
Btw, this works with Jackson parser
Sent from my iPhone
> On Feb 19, 2020, at 3:52 PM, Mark Payne <ma...@hotmail.com> wrote:
>
> I presume you're using schema inference to determine the column type in the reader?
>
> Interestingly, the following code:
> System.out.println("As a double: " + Double.parseDouble("999999.99"));
> System.out.println("As a float: " + Float.parseFloat("999999.99"));
> Produces the following output:
> As a double: 999999.99
> As a float: 1000000.0
>
> The CSV inference logic has the following code:
> if (value.contains(".")) {
> try {
> final double doubleValue = Double.parseDouble(value);
> if (doubleValue > Float.MAX_VALUE || doubleValue < Float.MIN_VALUE) {
> return RecordFieldType.DOUBLE.getDataType();
> }
>
> return RecordFieldType.FLOAT.getDataType();
> } catch (final NumberFormatException nfe) {
> return RecordFieldType.STRING.getDataType();
> }
> }
>
> So it determines that there's a decimal point. And the value easily fits within a Float, so it considers the value to be a Float. We should probably improve our inference logic to do a better job determining whether a value is a Float or Double - or else always use a Double instead of a Float.
>
> Thanks
> -Mark
>
>
>> On Feb 19, 2020, at 4:42 PM, Shawn Weeks <sw...@weeksconsulting.us> wrote:
>>
>> How are you defining the schema and what data type are setting for that column?
>>
>> Thanks
>> Shawn
>>
>> From: KhajaAsmath Mohammed <md...@gmail.com>
>> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Date: Wednesday, February 19, 2020 at 3:32 PM
>> To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Subject: NIFI Bug with Convert Record - 999999.99 changed to 1000000.0
>>
>> Hi,
>>
>> I am using below text to convert CSV to JSON using convert record processor. Value of 999999.99 changed to 1000000.0 , May I know how is this possible?
>>
>>
>> OPERATING_UNIT|TEST_34_2|TEST_34_15|TEST_34_5
>> "141516"|"1.1"|"1.1"|"1.1"
>> "141517"|"1.11"|"1.11"|"1.11"
>> "141518"|"1101.11"|"999999.99"|"1101.11"
>>
>>
>> <image001.png>
>>
>> Is this happening with CSV reader or JSON record set writer?
>>
>> Thanks,
>> Asmath
>
Re: NIFI Bug with Convert Record - 999999.99 changed to 1000000.0
Posted by KhajaAsmath Mohammed <md...@gmail.com>.
This is what I have in CSVReader.
[image: image.png]
On Wed, Feb 19, 2020 at 3:52 PM Mark Payne <ma...@hotmail.com> wrote:
> I presume you're using schema inference to determine the column type in
> the reader?
>
> Interestingly, the following code:
>
> System.out.println("As a double: " + Double.parseDouble("999999.99"));
> System.out.println("As a float: " + Float.parseFloat("999999.99"));
>
> Produces the following output:
> As a double: 999999.99
> As a float: 1000000.0
>
> The CSV inference logic has the following code:
>
> if (value.contains(".")) {
> try {
> final double doubleValue = Double.parseDouble(value);
> if (doubleValue > Float.MAX_VALUE || doubleValue < Float.MIN_VALUE) {
> return RecordFieldType.DOUBLE.getDataType();
> }
>
> return RecordFieldType.FLOAT.getDataType();
> } catch (final NumberFormatException nfe) {
> return RecordFieldType.STRING.getDataType();
> }
> }
>
>
> So it determines that there's a decimal point. And the value easily fits
> within a Float, so it considers the value to be a Float. We should probably
> improve our inference logic to do a better job determining whether a value
> is a Float or Double - or else always use a Double instead of a Float.
>
> Thanks
> -Mark
>
>
> On Feb 19, 2020, at 4:42 PM, Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> How are you defining the schema and what data type are setting for that
> column?
>
> Thanks
> Shawn
>
> *From: *KhajaAsmath Mohammed <md...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Wednesday, February 19, 2020 at 3:32 PM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *NIFI Bug with Convert Record - 999999.99 changed to 1000000.0
>
> Hi,
>
> I am using below text to convert CSV to JSON using convert record
> processor. Value of 999999.99 changed to 1000000.0 , May I know how is this
> possible?
>
>
> OPERATING_UNIT|TEST_34_2|TEST_34_15|TEST_34_5
> "141516"|"1.1"|"1.1"|"1.1"
> "141517"|"1.11"|"1.11"|"1.11"
> "141518"|"1101.11"|"999999.99"|"1101.11"
>
>
> <image001.png>
>
> Is this happening with CSV reader or JSON record set writer?
>
> Thanks,
> Asmath
>
>
>
Re: NIFI Bug with Convert Record - 999999.99 changed to 1000000.0
Posted by Mark Payne <ma...@hotmail.com>.
I presume you're using schema inference to determine the column type in the reader?
Interestingly, the following code:
System.out.println("As a double: " + Double.parseDouble("999999.99"));
System.out.println("As a float: " + Float.parseFloat("999999.99"));
Produces the following output:
As a double: 999999.99
As a float: 1000000.0
The CSV inference logic has the following code:
if (value.contains(".")) {
try {
final double doubleValue = Double.parseDouble(value);
if (doubleValue > Float.MAX_VALUE || doubleValue < Float.MIN_VALUE) {
return RecordFieldType.DOUBLE.getDataType();
}
return RecordFieldType.FLOAT.getDataType();
} catch (final NumberFormatException nfe) {
return RecordFieldType.STRING.getDataType();
}
}
So it determines that there's a decimal point. And the value easily fits within a Float, so it considers the value to be a Float. We should probably improve our inference logic to do a better job determining whether a value is a Float or Double - or else always use a Double instead of a Float.
Thanks
-Mark
On Feb 19, 2020, at 4:42 PM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
How are you defining the schema and what data type are setting for that column?
Thanks
Shawn
From: KhajaAsmath Mohammed <md...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, February 19, 2020 at 3:32 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: NIFI Bug with Convert Record - 999999.99 changed to 1000000.0
Hi,
I am using below text to convert CSV to JSON using convert record processor. Value of 999999.99 changed to 1000000.0 , May I know how is this possible?
OPERATING_UNIT|TEST_34_2|TEST_34_15|TEST_34_5
"141516"|"1.1"|"1.1"|"1.1"
"141517"|"1.11"|"1.11"|"1.11"
"141518"|"1101.11"|"999999.99"|"1101.11"
<image001.png>
Is this happening with CSV reader or JSON record set writer?
Thanks,
Asmath
Re: NIFI Bug with Convert Record - 999999.99 changed to 1000000.0
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
How are you defining the schema and what data type are setting for that column?
Thanks
Shawn
From: KhajaAsmath Mohammed <md...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, February 19, 2020 at 3:32 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: NIFI Bug with Convert Record - 999999.99 changed to 1000000.0
Hi,
I am using below text to convert CSV to JSON using convert record processor. Value of 999999.99 changed to 1000000.0 , May I know how is this possible?
OPERATING_UNIT|TEST_34_2|TEST_34_15|TEST_34_5
"141516"|"1.1"|"1.1"|"1.1"
"141517"|"1.11"|"1.11"|"1.11"
"141518"|"1101.11"|"999999.99"|"1101.11"
[cid:image001.png@01D5E73B.2E21C730]
Is this happening with CSV reader or JSON record set writer?
Thanks,
Asmath