You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Christopher Manniex <cm...@hotmail.com> on 2020/02/15 13:32:02 UTC
BigDecimal support
Hi All,
Hi All, not sure this is the right forum, I've been evaluating Nifi for use in one of our solutions but am hitting issues preserving type information for BigDecimal through to Mongo... am I missing something obvious or is this a possibility for an improvement (happy to contribute if i can be pointed in the right direction)
Regards
Chris
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
RE: BigDecimal support
Posted by Christopher Manniex <cm...@hotmail.com>.
Thanks Mike,
If I can help in any way please let me know.
Regards
Chris
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
________________________________
From: Mike Thomsen <mi...@gmail.com>
Sent: Sunday, February 16, 2020 1:25:37 PM
To: dev@nifi.apache.org <de...@nifi.apache.org>
Subject: Re: BigDecimal support
That could be a legitimate issue. I'll write up a Jira ticket.
On Sun, Feb 16, 2020 at 5:42 AM Christopher Manniex <cm...@hotmail.com>
wrote:
> Thanks Mike,
>
> Here’s a really stripped back example.
>
> I have an input as below
>
> {
> "value": "123456.00"
> }
>
> And I want to write the value into MongoDB as a numeric value, but
> critically not as a double (these will be currency values)
>
> In order to try to enforce the type used to write to Mongo we created a
> simple Avro schema, for the example above this would be as follows;
>
> {
> "type": "record",
> "name": "TransactionEvent",
> "namespace": "com.example.demo",
> "fields": [
> {
> "name": "value",
> "type": [
> "null",
> {
> "type": "bytes",
> "logicalType": "decimal",
> "precision": 10,
> "scale": 2
> }
> ]
> }
> ]
> }
>
> Hoping that this would map to a Decimal128 in Mongo, however we
> consistently see double as the type in Mongo regardless of any variations
> of Avro schema we have tried.
>
> On having a quick look into the code I’ve identified 2 possible problem
> areas.
>
>
> 1. The conversion of the Avro schema into the internal representation
> which seems to treat Avro logical decimal types as double (ref
> org.apache.nifi.avro.AvroTypeUtil – line 343)
> 2. The Mongo processor which uses this type information to decide what
> Mongo types to persist data as.
>
> For a quick win, which would hopefully have a smaller impact, I was hoping
> that I could fork the Mongo processor and keep the changes local to that
> but since the information about the Avro logical type is lost before the
> schema information gets to MongoDB i’m not sure that will be possible now.
>
> When we reached this point and the changes we were looking at seemed like
> they could be a little more complex than hoped we wanted to reach out to
> see if
>
>
> 1. We’re doing something wrong
> 2. Anybody else had encountered a similar situation
> 3. If we did look to introduce changes either to the Mongo processor or
> more widely for support of BigDecimal would this be of wider use?
>
> Many thanks
>
> Chris
>
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
> ________________________________
> From: Mike Thomsen <mi...@gmail.com>
> Sent: Saturday, February 15, 2020 7:48:29 PM
> To: dev@nifi.apache.org <de...@nifi.apache.org>
> Subject: Re: BigDecimal support
>
> Chris,
>
> Drop us some details about the delta between what you're expecting and what
> you see happening so we can investigate and see what we can do to help.
>
> Thanks,
>
> Mike
>
> On Sat, Feb 15, 2020 at 10:09 AM Christopher Manniex <cmanniex@hotmail.com
> >
> wrote:
>
> > Hi All,
> >
> > Hi All, not sure this is the right forum, I've been evaluating Nifi for
> > use in one of our solutions but am hitting issues preserving type
> > information for BigDecimal through to Mongo... am I missing something
> > obvious or is this a possibility for an improvement (happy to contribute
> if
> > i can be pointed in the right direction)
> >
> > Regards
> >
> > Chris
> >
> > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> > Windows 10
> >
> >
>
Re: BigDecimal support
Posted by Mike Thomsen <mi...@gmail.com>.
That could be a legitimate issue. I'll write up a Jira ticket.
On Sun, Feb 16, 2020 at 5:42 AM Christopher Manniex <cm...@hotmail.com>
wrote:
> Thanks Mike,
>
> Here’s a really stripped back example.
>
> I have an input as below
>
> {
> "value": "123456.00"
> }
>
> And I want to write the value into MongoDB as a numeric value, but
> critically not as a double (these will be currency values)
>
> In order to try to enforce the type used to write to Mongo we created a
> simple Avro schema, for the example above this would be as follows;
>
> {
> "type": "record",
> "name": "TransactionEvent",
> "namespace": "com.example.demo",
> "fields": [
> {
> "name": "value",
> "type": [
> "null",
> {
> "type": "bytes",
> "logicalType": "decimal",
> "precision": 10,
> "scale": 2
> }
> ]
> }
> ]
> }
>
> Hoping that this would map to a Decimal128 in Mongo, however we
> consistently see double as the type in Mongo regardless of any variations
> of Avro schema we have tried.
>
> On having a quick look into the code I’ve identified 2 possible problem
> areas.
>
>
> 1. The conversion of the Avro schema into the internal representation
> which seems to treat Avro logical decimal types as double (ref
> org.apache.nifi.avro.AvroTypeUtil – line 343)
> 2. The Mongo processor which uses this type information to decide what
> Mongo types to persist data as.
>
> For a quick win, which would hopefully have a smaller impact, I was hoping
> that I could fork the Mongo processor and keep the changes local to that
> but since the information about the Avro logical type is lost before the
> schema information gets to MongoDB i’m not sure that will be possible now.
>
> When we reached this point and the changes we were looking at seemed like
> they could be a little more complex than hoped we wanted to reach out to
> see if
>
>
> 1. We’re doing something wrong
> 2. Anybody else had encountered a similar situation
> 3. If we did look to introduce changes either to the Mongo processor or
> more widely for support of BigDecimal would this be of wider use?
>
> Many thanks
>
> Chris
>
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
> ________________________________
> From: Mike Thomsen <mi...@gmail.com>
> Sent: Saturday, February 15, 2020 7:48:29 PM
> To: dev@nifi.apache.org <de...@nifi.apache.org>
> Subject: Re: BigDecimal support
>
> Chris,
>
> Drop us some details about the delta between what you're expecting and what
> you see happening so we can investigate and see what we can do to help.
>
> Thanks,
>
> Mike
>
> On Sat, Feb 15, 2020 at 10:09 AM Christopher Manniex <cmanniex@hotmail.com
> >
> wrote:
>
> > Hi All,
> >
> > Hi All, not sure this is the right forum, I've been evaluating Nifi for
> > use in one of our solutions but am hitting issues preserving type
> > information for BigDecimal through to Mongo... am I missing something
> > obvious or is this a possibility for an improvement (happy to contribute
> if
> > i can be pointed in the right direction)
> >
> > Regards
> >
> > Chris
> >
> > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> > Windows 10
> >
> >
>
RE: BigDecimal support
Posted by Christopher Manniex <cm...@hotmail.com>.
Thanks Mike,
Here’s a really stripped back example.
I have an input as below
{
"value": "123456.00"
}
And I want to write the value into MongoDB as a numeric value, but critically not as a double (these will be currency values)
In order to try to enforce the type used to write to Mongo we created a simple Avro schema, for the example above this would be as follows;
{
"type": "record",
"name": "TransactionEvent",
"namespace": "com.example.demo",
"fields": [
{
"name": "value",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 10,
"scale": 2
}
]
}
]
}
Hoping that this would map to a Decimal128 in Mongo, however we consistently see double as the type in Mongo regardless of any variations of Avro schema we have tried.
On having a quick look into the code I’ve identified 2 possible problem areas.
1. The conversion of the Avro schema into the internal representation which seems to treat Avro logical decimal types as double (ref org.apache.nifi.avro.AvroTypeUtil – line 343)
2. The Mongo processor which uses this type information to decide what Mongo types to persist data as.
For a quick win, which would hopefully have a smaller impact, I was hoping that I could fork the Mongo processor and keep the changes local to that but since the information about the Avro logical type is lost before the schema information gets to MongoDB i’m not sure that will be possible now.
When we reached this point and the changes we were looking at seemed like they could be a little more complex than hoped we wanted to reach out to see if
1. We’re doing something wrong
2. Anybody else had encountered a similar situation
3. If we did look to introduce changes either to the Mongo processor or more widely for support of BigDecimal would this be of wider use?
Many thanks
Chris
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
________________________________
From: Mike Thomsen <mi...@gmail.com>
Sent: Saturday, February 15, 2020 7:48:29 PM
To: dev@nifi.apache.org <de...@nifi.apache.org>
Subject: Re: BigDecimal support
Chris,
Drop us some details about the delta between what you're expecting and what
you see happening so we can investigate and see what we can do to help.
Thanks,
Mike
On Sat, Feb 15, 2020 at 10:09 AM Christopher Manniex <cm...@hotmail.com>
wrote:
> Hi All,
>
> Hi All, not sure this is the right forum, I've been evaluating Nifi for
> use in one of our solutions but am hitting issues preserving type
> information for BigDecimal through to Mongo... am I missing something
> obvious or is this a possibility for an improvement (happy to contribute if
> i can be pointed in the right direction)
>
> Regards
>
> Chris
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
Re: BigDecimal support
Posted by Mike Thomsen <mi...@gmail.com>.
Chris,
Drop us some details about the delta between what you're expecting and what
you see happening so we can investigate and see what we can do to help.
Thanks,
Mike
On Sat, Feb 15, 2020 at 10:09 AM Christopher Manniex <cm...@hotmail.com>
wrote:
> Hi All,
>
> Hi All, not sure this is the right forum, I've been evaluating Nifi for
> use in one of our solutions but am hitting issues preserving type
> information for BigDecimal through to Mongo... am I missing something
> obvious or is this a possibility for an improvement (happy to contribute if
> i can be pointed in the right direction)
>
> Regards
>
> Chris
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>