You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Jonathan Coveney (JIRA)" <ji...@apache.org> on 2012/06/21 05:16:42 UTC
[jira] [Created] (PIG-2764) Add a biginteger and bigdecimal type to
pig
Jonathan Coveney created PIG-2764:
-------------------------------------
Summary: Add a biginteger and bigdecimal type to pig
Key: PIG-2764
URL: https://issues.apache.org/jira/browse/PIG-2764
Project: Pig
Issue Type: Improvement
Reporter: Jonathan Coveney
I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2764) Add a biginteger and bigdecimal type
to pig
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398962#comment-13398962 ]
Jonathan Coveney commented on PIG-2764:
---------------------------------------
Alan: it seemed like something that would be good to support, since I know there are financial users who could potentially leverage something like this.
Agreed that a fast, fixed point type would be excellent as well. Know any good libraries?
This was all done rogue because I wanted to know how hard it is to implement a type (answer: tedious and lots of corner cases, but not hard per se), and I thought it would be useful to have.
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: PIG-2764-0.patch, PIG-2764-1.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2764) Add a biginteger and bigdecimal type
to pig
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420925#comment-13420925 ]
Jeremy Hanna commented on PIG-2764:
-----------------------------------
It would be nice to have an expanded set of basic java types so that when people use alternate data stores, they can more easily convert data types to native pig types. For example, Cassandra has an BigInteger based data type validator and it would streamline conversion to pig types to have something native.
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: PIG-2764-0.patch, PIG-2764-1.patch, fixedpoint.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2764) Add a biginteger and bigdecimal type to
pig
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney updated PIG-2764:
----------------------------------
Attachment: PIG-2764-0.patch
So here is a first pass at this.[1] It works in the command line, but I'm not sure how to really test it. Would love to have eyes on it. I will upload to revieboard later.
There are a couple of choices that are suboptimal, but aren't really related to the core of the patch, so they can be fixed later (mainly just that I serialize it in a way that can be optimized...but given these aren't for performance but for accuracy, I'm not as worried).
[1] I needed a short respite from SchemaTuple work :) And I also wanted to see how difficult adding types is[2]
[2] difficult
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: PIG-2764-0.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2764) Add a biginteger and bigdecimal type to
pig
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney updated PIG-2764:
----------------------------------
Attachment: PIG-2764-1.patch
Fixed a couple of small things. Also, put up a review request:
https://reviews.apache.org/r/5468/
Based on this and the datetime stuff, we should put together a primer on how to add a type (and I have some refactoring in mind that could make it much better).
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: PIG-2764-0.patch, PIG-2764-1.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2764) Add a biginteger and bigdecimal type
to pig
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399813#comment-13399813 ]
Jonathan Coveney commented on PIG-2764:
---------------------------------------
Alan, I'm curious why you think managing our own fixed point implementation is better than just leveraging BigDecimal and BigInteger, and the arbitrary precision they provide?
I mean, I can benchmark things to see how radical the performance difference is, but I'm just curious.
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: PIG-2764-0.patch, PIG-2764-1.patch, fixedpoint.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2764) Add a biginteger and bigdecimal type
to pig
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398937#comment-13398937 ]
Alan Gates commented on PIG-2764:
---------------------------------
Do you really need biginteger and bigdecimal, or are you trying to get fixed point processing cheaply? Pig definitely needs a fixed point type.
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: PIG-2764-0.patch, PIG-2764-1.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2764) Add a biginteger and bigdecimal type to
pig
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-2764:
----------------------------
Attachment: fixedpoint.patch
First pass at a class to implement fixed point types.
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: PIG-2764-0.patch, PIG-2764-1.patch, fixedpoint.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2764) Add a biginteger and bigdecimal type
to pig
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney reassigned PIG-2764:
-------------------------------------
Assignee: Jonathan Coveney
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2764) Add a biginteger and bigdecimal type
to pig
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399822#comment-13399822 ]
Alan Gates commented on PIG-2764:
---------------------------------
I was hoping we could build something faster since BigDecimal is dreadfully slow. But at the moment what I put in there only beats it by about 20%, so not worth it. We can do it with BigDecimal for now, but I may keep playing with tuning this stuff to see if I can get it faster.
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: PIG-2764-0.patch, PIG-2764-1.patch, fixedpoint.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2764) Add a biginteger and bigdecimal type
to pig
Posted by "Patrick Salami (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439126#comment-13439126 ]
Patrick Salami commented on PIG-2764:
-------------------------------------
+1
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: fixedpoint.patch, PIG-2764-0.patch, PIG-2764-1.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2764) Add a biginteger and bigdecimal type
to pig
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399793#comment-13399793 ]
Alan Gates commented on PIG-2764:
---------------------------------
I don't know of any good libraries. And everything Google turned up looked old and unsupported. I took a quick crack at it (just the fixed point part, not integrating with Pig's data type system). I'll attach it as a separate patch.
> Add a biginteger and bigdecimal type to pig
> -------------------------------------------
>
> Key: PIG-2764
> URL: https://issues.apache.org/jira/browse/PIG-2764
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Attachments: PIG-2764-0.patch, PIG-2764-1.patch, fixedpoint.patch
>
>
> I think it would be useful for applications where precision is more important than speed to have the option of using java's bigdecimal and biginteger types natively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira