You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Robert Newson (JIRA)" <ji...@apache.org> on 2010/06/12 00:00:23 UTC
[jira] Created: (COUCHDB-796) Bignum support
Bignum support
--------------
Key: COUCHDB-796
URL: https://issues.apache.org/jira/browse/COUCHDB-796
Project: CouchDB
Issue Type: Improvement
Components: Database Core
Affects Versions: 0.11
Reporter: Robert Newson
Fix For: 1.0
CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (COUCHDB-796) Bignum support
Posted by Jason Smith <jh...@couch.io>.
On Sun, Jun 13, 2010 at 02:11, Robert Newson <ro...@gmail.com>wrote:
> Jan also mentioned your new library on IRC, so that was certainly one
> of the candidates for that approach.
>
> While it's true that the Javascript view server is where the 32-bit
> limitation will be most noticeable, I think we want bignum support
> everywhere that the limitation appears.
>
AFAIK the JSON standard does not mandate IEEE-754 floats. All it says is
numbers, a dot, and more numbers. AFAIK this is valid JSON:
{"num": 123456789112345678921234567893.123456789112345678921234567893}
Certainly the major view server languages support Bignum/BigDecimal
(Javascript is crippled, but hey--story of Javascript's life). If their JSON
parsers choke on the above JSON they have their own bugs.
I hate to sidetrack conversation but I am unclear which Couch component has
a bug.
Hmm, it looks like mochijson2 cannot parse that JSON however it can parse
bigints:
10> {struct, [{_, BigI}]} = mochijson2:decode(<<"{\"foo\":
12345678911234567892123456789312345678941234567895}">>).
{struct,[{<<"foo">>,
12345678911234567892123456789312345678941234567895}]}
11> BigI + 7.
12345678911234567892123456789312345678941234567902
> B.
>
>
> On Sat, Jun 12, 2010 at 8:06 PM, Jason Smith <jh...@couch.io> wrote:
> > On Sun, Jun 13, 2010 at 01:24, Robert Newson (JIRA) <ji...@apache.org>
> wrote:
> >
> >>
> >> [
> >>
> https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878298#action_12878298
> ]
> >>
> >> Robert Newson commented on COUCHDB-796:
> >> ---------------------------------------
> >>
> >> Sorry, I pasted a terse summary from #couchdb and was then too busy to
> >> flesh this out.
> >>
> >> I'm actually not sure I like the idea of tweaking mochijson2.erl, though
> it
> >> was suggested (by you iirc ;). Also the notion that numbers magically
> become
> >> strings at some point feels wrong, but so also does the notion that a
> sum()
> >> overflows or goes wrong when you hit that limit too.
> >>
> >> Rather, I think a guarantee that some bignum.js library is always
> available
> >> and some prominent docs about it. Ideally a BigDecimal and BigInteger
> class
> >> like Java has (yes, kill me).
> >>
> >
> > Weird, I just created that exact project the other day.
> > http://github.com/jhs/bigdecimal.js
> >
> > It is indeed the Java class! It is the Apache Harmony
> BigDecimal/BigInteger
> > implementation, compiled to Javascript thanks to GWT. This list isn't the
> > place to debate whether that's wise but a the very least, CouchDB's
> default
> > view server can support BigDecimal. I even put up a demo expression
> > evaluator:
> >
> >
> http://jhs.couchone.com/bigdecimal/_design/bigdecimal_demo/_show/ui?e=5192834791234712811238476129384671293847611923192386479347.1234876123467129384761234%20*%205123412612351263.1123412365123541234
> >
> > What is my point? That CouchDB perhaps does not need bigdecimal support
> in
> > mochijson2 because it is quite straightforward to get things working
> within
> > the view server.
> >
> > --
> > Jason Smith
> > Couchio Hosting
> >
>
--
Jason Smith
Couchio Hosting
Re: [jira] Commented: (COUCHDB-796) Bignum support
Posted by Robert Newson <ro...@gmail.com>.
Jan also mentioned your new library on IRC, so that was certainly one
of the candidates for that approach.
While it's true that the Javascript view server is where the 32-bit
limitation will be most noticeable, I think we want bignum support
everywhere that the limitation appears.
B.
On Sat, Jun 12, 2010 at 8:06 PM, Jason Smith <jh...@couch.io> wrote:
> On Sun, Jun 13, 2010 at 01:24, Robert Newson (JIRA) <ji...@apache.org> wrote:
>
>>
>> [
>> https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878298#action_12878298]
>>
>> Robert Newson commented on COUCHDB-796:
>> ---------------------------------------
>>
>> Sorry, I pasted a terse summary from #couchdb and was then too busy to
>> flesh this out.
>>
>> I'm actually not sure I like the idea of tweaking mochijson2.erl, though it
>> was suggested (by you iirc ;). Also the notion that numbers magically become
>> strings at some point feels wrong, but so also does the notion that a sum()
>> overflows or goes wrong when you hit that limit too.
>>
>> Rather, I think a guarantee that some bignum.js library is always available
>> and some prominent docs about it. Ideally a BigDecimal and BigInteger class
>> like Java has (yes, kill me).
>>
>
> Weird, I just created that exact project the other day.
> http://github.com/jhs/bigdecimal.js
>
> It is indeed the Java class! It is the Apache Harmony BigDecimal/BigInteger
> implementation, compiled to Javascript thanks to GWT. This list isn't the
> place to debate whether that's wise but a the very least, CouchDB's default
> view server can support BigDecimal. I even put up a demo expression
> evaluator:
>
> http://jhs.couchone.com/bigdecimal/_design/bigdecimal_demo/_show/ui?e=5192834791234712811238476129384671293847611923192386479347.1234876123467129384761234%20*%205123412612351263.1123412365123541234
>
> What is my point? That CouchDB perhaps does not need bigdecimal support in
> mochijson2 because it is quite straightforward to get things working within
> the view server.
>
> --
> Jason Smith
> Couchio Hosting
>
Re: [jira] Commented: (COUCHDB-796) Bignum support
Posted by Jason Smith <jh...@couch.io>.
On Sun, Jun 13, 2010 at 01:24, Robert Newson (JIRA) <ji...@apache.org> wrote:
>
> [
> https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878298#action_12878298]
>
> Robert Newson commented on COUCHDB-796:
> ---------------------------------------
>
> Sorry, I pasted a terse summary from #couchdb and was then too busy to
> flesh this out.
>
> I'm actually not sure I like the idea of tweaking mochijson2.erl, though it
> was suggested (by you iirc ;). Also the notion that numbers magically become
> strings at some point feels wrong, but so also does the notion that a sum()
> overflows or goes wrong when you hit that limit too.
>
> Rather, I think a guarantee that some bignum.js library is always available
> and some prominent docs about it. Ideally a BigDecimal and BigInteger class
> like Java has (yes, kill me).
>
Weird, I just created that exact project the other day.
http://github.com/jhs/bigdecimal.js
It is indeed the Java class! It is the Apache Harmony BigDecimal/BigInteger
implementation, compiled to Javascript thanks to GWT. This list isn't the
place to debate whether that's wise but a the very least, CouchDB's default
view server can support BigDecimal. I even put up a demo expression
evaluator:
http://jhs.couchone.com/bigdecimal/_design/bigdecimal_demo/_show/ui?e=5192834791234712811238476129384671293847611923192386479347.1234876123467129384761234%20*%205123412612351263.1123412365123541234
What is my point? That CouchDB perhaps does not need bigdecimal support in
mochijson2 because it is quite straightforward to get things working within
the view server.
--
Jason Smith
Couchio Hosting
[jira] Commented: (COUCHDB-796) Bignum support
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878321#action_12878321 ]
Adam Kocoloski commented on COUCHDB-796:
----------------------------------------
>From dev@
On Jun 12, 2010, at 3:33 PM, Jason Smith wrote:
> I hate to sidetrack conversation but I am unclear which Couch component has a bug.
No problem, it's good that people understand this. There are two bugs. Bug #1 is that roundtrip encoding/decoding of big numbers in the Erlang VM is not supported:
1> Original = <<"[12345678911234567892123456789312345678941234567895]">>.
<<"[12345678911234567892123456789312345678941234567895]">>
2> RoundTrip = couch_util:json_encode(couch_util:json_decode(Original)).
[91,"1.2345678911234567e+49",93]
3> io:format("~s~n", [RoundTrip]).
[1.2345678911234567e+49]
Anything larger than a 32 bit integer is cast to a float. This is trivial to fix, and in fact the behavior is a regression since 0.10.x, when we were running a forked mochijson2.
Bug #2 is that the Javascript VM represents Numbers as double precision floating-point, so anything with more than 15 digits of precision gets rounded off. I imagine your bigdecimal.js, but it would need the numbers as strings, right? Hence Bob's original suggestion
> Bignum support
> --------------
>
> Key: COUCHDB-796
> URL: https://issues.apache.org/jira/browse/COUCHDB-796
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Reporter: Robert Newson
> Fix For: 1.0
>
>
> CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
> Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-796) Bignum support
Posted by "Paul Joseph Davis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Joseph Davis updated COUCHDB-796:
--------------------------------------
Skill Level: Regular Contributors Level (Easy to Medium)
> Bignum support
> --------------
>
> Key: COUCHDB-796
> URL: https://issues.apache.org/jira/browse/COUCHDB-796
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Reporter: Robert Newson
> Fix For: 1.0.2
>
>
> CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
> Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-796) Bignum support
Posted by "Noah Slater (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noah Slater updated COUCHDB-796:
--------------------------------
Fix Version/s: 1.0.2
(was: 1.0.1)
> Bignum support
> --------------
>
> Key: COUCHDB-796
> URL: https://issues.apache.org/jira/browse/COUCHDB-796
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Reporter: Robert Newson
> Fix For: 1.0.2
>
>
> CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
> Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-796) Bignum support
Posted by "Mike Leddy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897090#action_12897090 ]
Mike Leddy commented on COUCHDB-796:
------------------------------------
Chris,
I use Ruby for all my views and it should work fine too :-)
irb(main):001:0> require 'json'
=> true
irb(main):002:0> JSON.parse(JSON.generate([2**256]))
=> [115792089237316195423570985008687907853269984665640564039457584007913129639936]
> Bignum support
> --------------
>
> Key: COUCHDB-796
> URL: https://issues.apache.org/jira/browse/COUCHDB-796
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Reporter: Robert Newson
> Fix For: 1.0.1
>
>
> CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
> Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-796) Bignum support
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878297#action_12878297 ]
Adam Kocoloski commented on COUCHDB-796:
----------------------------------------
Hi Bob, can you be a little more specific on this proposal? What does it mean to "handle bignums" -- does anything change in the mochijson2 decoder?
Do you want to make the encoder always encode integers larger than N as strings? I kinda dislike that default because it becomes impossible to roundtrip a bignum through mochijson2.
> Bignum support
> --------------
>
> Key: COUCHDB-796
> URL: https://issues.apache.org/jira/browse/COUCHDB-796
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Reporter: Robert Newson
> Fix For: 1.0
>
>
> CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
> Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-796) Bignum support
Posted by "Chris Anderson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878643#action_12878643 ]
Chris Anderson commented on COUCHDB-796:
----------------------------------------
+1 on repatching mochijson2
-1 on magic-string numbers
if you need super-precise bignums, you're gonna have to run a query server that supports them. might I suggest Erlang views?
> Bignum support
> --------------
>
> Key: COUCHDB-796
> URL: https://issues.apache.org/jira/browse/COUCHDB-796
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Reporter: Robert Newson
> Fix For: 1.0
>
>
> CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
> Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-796) Bignum support
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878323#action_12878323 ]
Adam Kocoloski commented on COUCHDB-796:
----------------------------------------
Bob, yes, I suggested tweaking mochijson2, since that's what we did in the past. Supporting the full integer range is a trivial patch. Including an option to encode bignums as strings would be pretty easy, too, but I'm not sure how much I like it.
> Bignum support
> --------------
>
> Key: COUCHDB-796
> URL: https://issues.apache.org/jira/browse/COUCHDB-796
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Reporter: Robert Newson
> Fix For: 1.0
>
>
> CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
> Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-796) Bignum support
Posted by "Noah Slater (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noah Slater updated COUCHDB-796:
--------------------------------
Fix Version/s: 1.0.1
(was: 1.0)
> Bignum support
> --------------
>
> Key: COUCHDB-796
> URL: https://issues.apache.org/jira/browse/COUCHDB-796
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Reporter: Robert Newson
> Fix For: 1.0.1
>
>
> CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
> Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-796) Bignum support
Posted by "Robert Newson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878298#action_12878298 ]
Robert Newson commented on COUCHDB-796:
---------------------------------------
Sorry, I pasted a terse summary from #couchdb and was then too busy to flesh this out.
I'm actually not sure I like the idea of tweaking mochijson2.erl, though it was suggested (by you iirc ;). Also the notion that numbers magically become strings at some point feels wrong, but so also does the notion that a sum() overflows or goes wrong when you hit that limit too.
Rather, I think a guarantee that some bignum.js library is always available and some prominent docs about it. Ideally a BigDecimal and BigInteger class like Java has (yes, kill me).
> Bignum support
> --------------
>
> Key: COUCHDB-796
> URL: https://issues.apache.org/jira/browse/COUCHDB-796
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Reporter: Robert Newson
> Fix For: 1.0
>
>
> CouchDB uses spidermonkey which can only handle 32 bit integer values before overflowing. This might surprise users of map/reduce on large datasets.
> Instead, alter mochijson2.erl to handle bignums and encode numeric values as strings.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.