You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Christopher Manniex (Jira)" <ji...@apache.org> on 2020/02/17 22:38:00 UTC
[jira] [Commented] (NIFI-7159) Mongo processors appear to not
support Decimal 128 data types
[ https://issues.apache.org/jira/browse/NIFI-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038677#comment-17038677 ]
Christopher Manniex commented on NIFI-7159:
-------------------------------------------
Have been having a look at this during the weekend.
Would it be overkill to implement a new RecordFieldType for DECIMAL, distinct from Double, to represent the logical type Decimal defined in an Avro schema?
It would not seem right to force a Double to always be a Decimal128 in Mongo
I've put together an implementation along these lines I'm just testing now
If this is felt to be an appropriate approach to a fix I can roll up into a PR for review and discussion?
Regards
Chris
> Mongo processors appear to not support Decimal 128 data types
> -------------------------------------------------------------
>
> Key: NIFI-7159
> URL: https://issues.apache.org/jira/browse/NIFI-7159
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Mike Thomsen
> Assignee: Mike Thomsen
> Priority: Major
>
> This is the verbatim response from the user posting on nifi-dev:
>
> *_Here’s a really stripped back example._*
>
> *_I have an input as below_*
>
> *_{_*
> *_"value": "123456.00"_*
> *_}_*
>
> *_And I want to write the value into MongoDB as a numeric value, but critically not as a double (these will be currency values)_*
>
> *_In order to try to enforce the type used to write to Mongo we created a simple Avro schema, for the example above this would be as follows;_*
>
> *_{_*
> *_"type": "record",_*
> *_"name": "TransactionEvent",_*
> *_"namespace": "com.example.demo",_*
> *_"fields": [_*
> *_{_*
> *_"name": "value",_*
> *_"type": [_*
> *_"null",_*
> *_{_*
> *_"type": "bytes",_*
> *_"logicalType": "decimal",_*
> *_"precision": 10,_*
> *_"scale": 2_*
> *_}_*
> *_]_*
> *_}_*
> *_]_*
> *_}_*
>
> *_Hoping that this would map to a Decimal128 in Mongo, however we consistently see double as the type in Mongo regardless of any variations of Avro schema we have tried._*
>
> *_On having a quick look into the code I’ve identified 2 possible problem areas._*
>
>
> *_1. The conversion of the Avro schema into the internal representation which seems to treat Avro logical decimal types as double (ref org.apache.nifi.avro.AvroTypeUtil – line 343)_*
> *_2. The Mongo processor which uses this type information to decide what Mongo types to persist data as._*
>
> *_For a quick win, which would hopefully have a smaller impact, I was hoping that I could fork the Mongo processor and keep the changes local to that but since the information about the Avro logical type is lost before the schema information gets to MongoDB i’m not sure that will be possible now._*
>
> *_When we reached this point and the changes we were looking at seemed like they could be a little more complex than hoped we wanted to reach out to see if_*
>
>
> *_1. We’re doing something wrong_*
> *_2. Anybody else had encountered a similar situation_*
> *_3. If we did look to introduce changes either to the Mongo processor or more widely for support of BigDecimal would this be of wider use?_*
>
> It would appear to be a distinctly different type than Double:
>
> https://mongodb.github.io/mongo-java-driver/3.5/javadoc/?org/bson/types/Decimal128.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)