You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Anitha Raju (Created) (JIRA)" <ji...@apache.org> on 2011/12/07 18:34:39 UTC
[jira] [Created] (PIG-2406) Improve double precision
Improve double precision
-------------------------
Key: PIG-2406
URL: https://issues.apache.org/jira/browse/PIG-2406
Project: Pig
Issue Type: New Feature
Reporter: Anitha Raju
Hi,
When 2.51 (type: double) is added 24 times using SUM, it gives you result of 60.23999999999997.
If a user wants to control the precision, that is, in the result he requires 60.24, it wouldnt be possible unless he implements a custom UDF.
So if we can have a way to improve double precision,
for eg: http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html
Here with BigDecimal, one can improve the double precision and with MathContext, the user has control to set the precision.
Can we have a new UDF which can do the same?
Regards,
Anitha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2406) Improve double precision
Posted by "Vivek Padmanabhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167559#comment-13167559 ]
Vivek Padmanabhan commented on PIG-2406:
----------------------------------------
IMHO we should be having a UDF (in piggybank?) for rounding based on precision setting and may be UDF in piggybank for sum using BigDecimal.
Users can have a choice based on their requirements.
Another note is that, users comparing the results of Pig 0.8 and Pig 0.9. For the above case;
Result in 0.8 = 60.23999999999997
Result in 0.9 = 60.24000000000001 (due to invocation of combiners)
> Improve double precision
> -------------------------
>
> Key: PIG-2406
> URL: https://issues.apache.org/jira/browse/PIG-2406
> Project: Pig
> Issue Type: New Feature
> Reporter: Anitha Raju
>
> Hi,
> When 2.51 (type: double) is added 24 times using SUM, it gives you result of 60.23999999999997.
> If a user wants to control the precision, that is, in the result he requires 60.24, it wouldnt be possible unless he implements a custom UDF.
> So if we can have a way to improve double precision,
> for eg: http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html
> Here with BigDecimal, one can improve the double precision and with MathContext, the user has control to set the precision.
> Can we have a new UDF which can do the same?
> Regards,
> Anitha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2406) Improve double precision
Posted by "Mathias Herberts (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184870#comment-13184870 ]
Mathias Herberts commented on PIG-2406:
---------------------------------------
The use of BigDecimal is indeed way slower than using double and/or float, but in some cases (financial apps come to mind), having exact precision decimal numbers is of prime interest.
Adding a BigDecimal type to Pig would make its adoption easier in several verticals.
> Improve double precision
> -------------------------
>
> Key: PIG-2406
> URL: https://issues.apache.org/jira/browse/PIG-2406
> Project: Pig
> Issue Type: New Feature
> Reporter: Anitha Raju
>
> Hi,
> When 2.51 (type: double) is added 24 times using SUM, it gives you result of 60.23999999999997.
> If a user wants to control the precision, that is, in the result he requires 60.24, it wouldnt be possible unless he implements a custom UDF.
> So if we can have a way to improve double precision,
> for eg: http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html
> Here with BigDecimal, one can improve the double precision and with MathContext, the user has control to set the precision.
> Can we have a new UDF which can do the same?
> Regards,
> Anitha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2406) Improve double precision
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164638#comment-13164638 ]
Daniel Dai commented on PIG-2406:
---------------------------------
That is on roadmap ("Fixed Point Type" in https://cwiki.apache.org/confluence/display/PIG/Pig+Journal).
> Improve double precision
> -------------------------
>
> Key: PIG-2406
> URL: https://issues.apache.org/jira/browse/PIG-2406
> Project: Pig
> Issue Type: New Feature
> Reporter: Anitha Raju
>
> Hi,
> When 2.51 (type: double) is added 24 times using SUM, it gives you result of 60.23999999999997.
> If a user wants to control the precision, that is, in the result he requires 60.24, it wouldnt be possible unless he implements a custom UDF.
> So if we can have a way to improve double precision,
> for eg: http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html
> Here with BigDecimal, one can improve the double precision and with MathContext, the user has control to set the precision.
> Can we have a new UDF which can do the same?
> Regards,
> Anitha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2406) Improve double precision
Posted by "Scott Carey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167296#comment-13167296 ]
Scott Carey commented on PIG-2406:
----------------------------------
The above example of adding 2.51 many times and the result being slightly off is due to the nature of floating point numbers, and is no different than what happens in a database. Double precision math does that, there is no 'improvement' to make to it.
Ideally there would be another data type of higher or limitless precision than double ('numeric' in some databases, BigDecimal in Java), but as Alan says, these data types are extremely slow. They should be used judiciously.
For many use cases, a handful of functions for rounding down floating points is all that is needed. Even rounding to one less decimal in the above example gives the expected result.
In a database, I would have called a rounding function on the result. With Postgres it would be something like 'round(sum(val), 6)' using the round function here: http://www.postgresql.org/docs/9.1/static/functions-math.html#FUNCTIONS-MATH-FUNC-TABLE
If Pig had more built-in numeric functions it would help many use cases where exact precision is not required, but one may wish to print results for human interpretation or compare two numbers after computation. To do these things with double precision numbers you need to round numbers down or compare within a tolerance rather than exact match. In Pig I have had to write code manually to round numbers properly. For example, to round a floating point value to the nearest integer you have to add 0.5 then cast to a long, or use a UDF.
> Improve double precision
> -------------------------
>
> Key: PIG-2406
> URL: https://issues.apache.org/jira/browse/PIG-2406
> Project: Pig
> Issue Type: New Feature
> Reporter: Anitha Raju
>
> Hi,
> When 2.51 (type: double) is added 24 times using SUM, it gives you result of 60.23999999999997.
> If a user wants to control the precision, that is, in the result he requires 60.24, it wouldnt be possible unless he implements a custom UDF.
> So if we can have a way to improve double precision,
> for eg: http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html
> Here with BigDecimal, one can improve the double precision and with MathContext, the user has control to set the precision.
> Can we have a new UDF which can do the same?
> Regards,
> Anitha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2406) Improve double precision
Posted by "Alan Gates (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166314#comment-13166314 ]
Alan Gates commented on PIG-2406:
---------------------------------
You could certainly contribute a UDF to Piggybank to do floating point math in BigDecimal. At one point we looked at moving entirely to BigDecimal, but it's about 10x slower than using Double.
> Improve double precision
> -------------------------
>
> Key: PIG-2406
> URL: https://issues.apache.org/jira/browse/PIG-2406
> Project: Pig
> Issue Type: New Feature
> Reporter: Anitha Raju
>
> Hi,
> When 2.51 (type: double) is added 24 times using SUM, it gives you result of 60.23999999999997.
> If a user wants to control the precision, that is, in the result he requires 60.24, it wouldnt be possible unless he implements a custom UDF.
> So if we can have a way to improve double precision,
> for eg: http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html
> Here with BigDecimal, one can improve the double precision and with MathContext, the user has control to set the precision.
> Can we have a new UDF which can do the same?
> Regards,
> Anitha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2406) Improve double precision
Posted by "Alan Gates (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185349#comment-13185349 ]
Alan Gates commented on PIG-2406:
---------------------------------
I have no problem adding a fixed type decimal to Pig that would be appropriate for financial calculations and others that need precision. I think this would be a great feature. My point was just we don't want to move the existing float and double to BigDecimal.
> Improve double precision
> -------------------------
>
> Key: PIG-2406
> URL: https://issues.apache.org/jira/browse/PIG-2406
> Project: Pig
> Issue Type: New Feature
> Reporter: Anitha Raju
>
> Hi,
> When 2.51 (type: double) is added 24 times using SUM, it gives you result of 60.23999999999997.
> If a user wants to control the precision, that is, in the result he requires 60.24, it wouldnt be possible unless he implements a custom UDF.
> So if we can have a way to improve double precision,
> for eg: http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html
> Here with BigDecimal, one can improve the double precision and with MathContext, the user has control to set the precision.
> Can we have a new UDF which can do the same?
> Regards,
> Anitha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira