You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Stéphane Maarek <st...@gmail.com> on 2016/08/12 00:16:13 UTC
How to deal with decimals while they're not supported?
Hi,
I have a flow in which I extract an attribute from json using jsonpath.
That attribute happens to be a decimal number (0.123). I wanted to do a
simple operation such as myAttr:lt(0.1) but obviously that won't work. What
also won't work is myAttr:multiply(10):lt(1). I'm kinda stuck and I really
need this logic to be working. What do you advise as a workaround?
Also, I've seen there is a JIRA for this:
https://issues.apache.org/jira/browse/NIFI-1662 but stuff hasn't moved much
since it first appeared. Not sure if it got de-prioritized or something
Congrats on the 1.0.0 beta, it looks great !!
Cheers,
Stephane
Re: How to deal with decimals while they're not supported?
Posted by Conrad Crampton <co...@SecData.com>.
Excellent.
Again, haven’t tried it, but could possibly calculate the scaling factor from doing a length of the string after the decimal point. Haven’t really worked this through and may get very complicated.
Alternatively you could do another substring on the decimal portion and limit it to just two decimal places for example.
Regards
Conrad
From: Stéphane Maarek <st...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Friday, 12 August 2016 at 08:04
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: How to deal with decimals while they're not supported?
That works :) That implies knowing a bit about the scale of numbers, but that's a decent workaround. It'd be so great if decimals were supported natively though. Or at least the casting of decimal to int without having to remove the decimal
i.e (1.2):toNumber() instead of 1.2:toString().substringBefore('.'):toNumber()
On Fri, Aug 12, 2016 at 4:59 PM Conrad Crampton <co...@secdata.com>> wrote:
HI,
Can you coerce to a string, take the numbers after the decimal point and use that? So a similar logic to multiplying by 10 then doing the comparison. Obviously have to revert back to number to do the actual comparison though.
Haven’t tested but something like…
myAttr:toString():substringAfter(‘.’):toNumber():lt(1)
Conrad
From: Stéphane Maarek <st...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Friday, 12 August 2016 at 01:16
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: How to deal with decimals while they're not supported?
Hi,
I have a flow in which I extract an attribute from json using jsonpath. That attribute happens to be a decimal number (0.123). I wanted to do a simple operation such as myAttr:lt(0.1) but obviously that won't work. What also won't work is myAttr:multiply(10):lt(1). I'm kinda stuck and I really need this logic to be working. What do you advise as a workaround?
Also, I've seen there is a JIRA for this: https://issues.apache.org/jira/browse/NIFI-1662 but stuff hasn't moved much since it first appeared. Not sure if it got de-prioritized or something
Congrats on the 1.0.0 beta, it looks great !!
Cheers,
Stephane
***This email originated outside SecureData***
Click here<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
SecureData, combating cyber threats
________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
Re: How to deal with decimals while they're not supported?
Posted by Stéphane Maarek <st...@gmail.com>.
That works :) That implies knowing a bit about the scale of numbers, but
that's a decent workaround. It'd be so great if decimals were supported
natively though. Or at least the casting of decimal to int without having
to remove the decimal
i.e (1.2):toNumber() instead of
1.2:toString().substringBefore('.'):toNumber()
On Fri, Aug 12, 2016 at 4:59 PM Conrad Crampton <co...@secdata.com>
wrote:
> HI,
>
> Can you coerce to a string, take the numbers after the decimal point and
> use that? So a similar logic to multiplying by 10 then doing the
> comparison. Obviously have to revert back to number to do the actual
> comparison though.
>
> Haven’t tested but something like…
>
> myAttr:toString():substringAfter(‘.’):toNumber():lt(1)
>
> Conrad
>
>
>
> *From: *Stéphane Maarek <st...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Friday, 12 August 2016 at 01:16
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *How to deal with decimals while they're not supported?
>
>
>
> Hi,
>
>
>
> I have a flow in which I extract an attribute from json using jsonpath.
> That attribute happens to be a decimal number (0.123). I wanted to do a
> simple operation such as myAttr:lt(0.1) but obviously that won't work. What
> also won't work is myAttr:multiply(10):lt(1). I'm kinda stuck and I really
> need this logic to be working. What do you advise as a workaround?
>
>
> Also, I've seen there is a JIRA for this:
> https://issues.apache.org/jira/browse/NIFI-1662 but stuff hasn't moved
> much since it first appeared. Not sure if it got de-prioritized or something
>
>
>
> Congrats on the 1.0.0 beta, it looks great !!
>
>
>
> Cheers,
>
> Stephane
>
>
>
> ***This email originated outside SecureData***
>
> Click here <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to
> report this email as spam.
>
>
> SecureData, combating cyber threats
>
> ------------------------------
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email
> are those of the individual and not necessarily of SecureData Europe Ltd.
> Any prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT
>
Re: How to deal with decimals while they're not supported?
Posted by Conrad Crampton <co...@SecData.com>.
HI,
Can you coerce to a string, take the numbers after the decimal point and use that? So a similar logic to multiplying by 10 then doing the comparison. Obviously have to revert back to number to do the actual comparison though.
Haven’t tested but something like…
myAttr:toString():substringAfter(‘.’):toNumber():lt(1)
Conrad
From: Stéphane Maarek <st...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Friday, 12 August 2016 at 01:16
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: How to deal with decimals while they're not supported?
Hi,
I have a flow in which I extract an attribute from json using jsonpath. That attribute happens to be a decimal number (0.123). I wanted to do a simple operation such as myAttr:lt(0.1) but obviously that won't work. What also won't work is myAttr:multiply(10):lt(1). I'm kinda stuck and I really need this logic to be working. What do you advise as a workaround?
Also, I've seen there is a JIRA for this: https://issues.apache.org/jira/browse/NIFI-1662 but stuff hasn't moved much since it first appeared. Not sure if it got de-prioritized or something
Congrats on the 1.0.0 beta, it looks great !!
Cheers,
Stephane
***This email originated outside SecureData***
Click here<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
SecureData, combating cyber threats
______________________________________________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
RE: How to deal with decimals while they're not supported?
Posted by "Lee Laim (leelaim)" <le...@micron.com>.
Hi Stephane,
I've been porting to an Execute Stream Command (ESC) processor with a python one-liner to do that math, return to the output as an attribute, and finally pass the original to the next processor. It is sufficiently fast for our use case.
Argument: -c "from __future__ import division; print '%.1f%%' %(${numerator:multiply(100)}/${denominator})"
Command: /usr/bin/python
Output Destination Attribute: metric_percent
Thanks,
Lee
-----Original Message-----
From: Joe Percivall [mailto:joepercivall@yahoo.com]
Sent: Friday, August 12, 2016 8:22 AM
To: users@nifi.apache.org
Subject: Re: How to deal with decimals while they're not supported?
Not sure why my email got delayed so late but Conrad's way is definitely an option too. When you know that all the numbers will have the exact same number of decimals it is probably faster than my way (substring vs. a regex). My way will allow you to just grab the most significant decimal point, regardless of size, and compare on that.
Regardless, there seems to be a couple of different ways to accomplish this and I'm glad you got it working.
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Friday, August 12, 2016 5:11 AM, Joe Percivall <jo...@yahoo.com> wrote:
Hey Stephane,
Currently working with Decimals in NiFi is like putting a square peg in a round hole. I haven't tried to do it much but for a simple use-case like yours I believe there are two options.
1: Use Expression Language and a Regex to move the decimal
2: (Not 100% sure this works) Use the scripting processors to write a script to do the routing
For option 1 I think you can do something like this "${myAttr:replaceAll("0.(\d)(\d*)", "$1"):lt(1)}. This should "move" the decimal over one place so that you can check it against the 1. This assumes there is nothing else in that attribute.
For option 2, Matt Burgess would need to weigh-in.
As for NIFI-1662, it got de-prioritzed in that I got super busy with everything else I was doing for MiNiFi-0.0.1 and NiFi-1.0.0 (on-going) and it went to the back-burner. It's something I really want to get in though, along with a few other changes, because I think they open NiFi up to many new possibilities. I'd like to do it (decimals) in 1.1.0 since it wouldn't be a breaking change.
Hope that helps,
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Thursday, August 11, 2016 8:16 PM, Stéphane Maarek <st...@gmail.com> wrote:
Hi,
I have a flow in which I extract an attribute from json using jsonpath. That attribute happens to be a decimal number (0.123). I wanted to do a simple operation such as myAttr:lt(0.1) but obviously that won't work. What also won't work is myAttr:multiply(10):lt(1). I'm kinda stuck and I really need this logic to be working. What do you advise as a workaround?
Also, I've seen there is a JIRA for this: https://issues.apache.org/jira/browse/NIFI-1662 but stuff hasn't moved much since it first appeared. Not sure if it got de-prioritized or something
Congrats on the 1.0.0 beta, it looks great !!
Cheers,
Stephane
Re: How to deal with decimals while they're not supported?
Posted by Joe Percivall <jo...@yahoo.com>.
Not sure why my email got delayed so late but Conrad's way is definitely an option too. When you know that all the numbers will have the exact same number of decimals it is probably faster than my way (substring vs. a regex). My way will allow you to just grab the most significant decimal point, regardless of size, and compare on that.
Regardless, there seems to be a couple of different ways to accomplish this and I'm glad you got it working.
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Friday, August 12, 2016 5:11 AM, Joe Percivall <jo...@yahoo.com> wrote:
Hey Stephane,
Currently working with Decimals in NiFi is like putting a square peg in a round hole. I haven't tried to do it much but for a simple use-case like yours I believe there are two options.
1: Use Expression Language and a Regex to move the decimal
2: (Not 100% sure this works) Use the scripting processors to write a script to do the routing
For option 1 I think you can do something like this "${myAttr:replaceAll("0.(\d)(\d*)", "$1"):lt(1)}. This should "move" the decimal over one place so that you can check it against the 1. This assumes there is nothing else in that attribute.
For option 2, Matt Burgess would need to weigh-in.
As for NIFI-1662, it got de-prioritzed in that I got super busy with everything else I was doing for MiNiFi-0.0.1 and NiFi-1.0.0 (on-going) and it went to the back-burner. It's something I really want to get in though, along with a few other changes, because I think they open NiFi up to many new possibilities. I'd like to do it (decimals) in 1.1.0 since it wouldn't be a breaking change.
Hope that helps,
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Thursday, August 11, 2016 8:16 PM, Stéphane Maarek <st...@gmail.com> wrote:
Hi,
I have a flow in which I extract an attribute from json using jsonpath. That attribute happens to be a decimal number (0.123). I wanted to do a simple operation such as myAttr:lt(0.1) but obviously that won't work. What also won't work is myAttr:multiply(10):lt(1). I'm kinda stuck and I really need this logic to be working. What do you advise as a workaround?
Also, I've seen there is a JIRA for this: https://issues.apache.org/jira/browse/NIFI-1662 but stuff hasn't moved much since it first appeared. Not sure if it got de-prioritized or something
Congrats on the 1.0.0 beta, it looks great !!
Cheers,
Stephane
Re: How to deal with decimals while they're not supported?
Posted by Joe Percivall <jo...@yahoo.com>.
Hey Stephane,
Currently working with Decimals in NiFi is like putting a square peg in a round hole. I haven't tried to do it much but for a simple use-case like yours I believe there are two options.
1: Use Expression Language and a Regex to move the decimal2: (Not 100% sure this works) Use the scripting processors to write a script to do the routing
For option 1 I think you can do something like this "${myAttr:replaceAll("0.(\d)(\d*)", "$1"):lt(1)}. This should "move" the decimal over one place so that you can check it against the 1. This assumes there is nothing else in that attribute.For option 2, Matt Burgess would need to weigh-in.
As for NIFI-1662, it got de-prioritzed in that I got super busy with everything else I was doing for MiNiFi-0.0.1 and NiFi-1.0.0 (on-going) and it went to the back-burner. It's something I really want to get in though, along with a few other changes, because I think they open NiFi up to many new possibilities. I'd like to do it (decimals) in 1.1.0 since it wouldn't be a breaking change.
Hope that helps,Joe - - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joepercivall@yahoo.com
On Thursday, August 11, 2016 8:16 PM, Stéphane Maarek <st...@gmail.com> wrote:
Hi,
I have a flow in which I extract an attribute from json using jsonpath. That attribute happens to be a decimal number (0.123). I wanted to do a simple operation such as myAttr:lt(0.1) but obviously that won't work. What also won't work is myAttr:multiply(10):lt(1). I'm kinda stuck and I really need this logic to be working. What do you advise as a workaround?
Also, I've seen there is a JIRA for this: https://issues.apache.org/jira/browse/NIFI-1662 but stuff hasn't moved much since it first appeared. Not sure if it got de-prioritized or something
Congrats on the 1.0.0 beta, it looks great !!
Cheers,Stephane