You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Jarrod Roberson (JIRA)" <ji...@apache.org> on 2010/04/23 19:26:50 UTC
[jira] Created: (COUCHDB-749) CouchDB does not persist large values
of Numbers correctly.
CouchDB does not persist large values of Numbers correctly.
-----------------------------------------------------------
Key: COUCHDB-749
URL: https://issues.apache.org/jira/browse/COUCHDB-749
Project: CouchDB
Issue Type: Bug
Affects Versions: 0.11
Environment: All
Reporter: Jarrod Roberson
All the following operations exhibit the same bug, large numbers don't get persisted correctly. They get something added to them for some reason.
9223372036854775807 == java.lang.Long.MAX_VALUE
1: go into Futon, create a new document and create a new field and enter the number 9223372036854775807, click the green check mark, the number changes to 9223372036854776000 even before you save it.
2.curl -X PUT http://localhost:5984/test/longTest -d '{"value": 9223372036854775807}', the number gets persisted as 9223372036854776000
trying to persist System.currentTimeMilliseconds() from java causes the same thing to happen occasionally.
This seems to be a pretty serious bug if I can't trust that my data is not being corrupted when submitted to the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-749) CouchDB does not persist large
values of Numbers correctly.
Posted by "Thad Guidry (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860657#action_12860657 ]
Thad Guidry commented on COUCHDB-749:
-------------------------------------
Researched at one time on http://www.programmersparadox.com/2008/02/05/erlang-integers/
February 6, 2008 @ 12:58 am
Comment by Nic:
Arbitrary-sized integer is nice but poorly supported in Erlang.
If one only uses additions or multiplications its okay but
integer power arithmetic operation is not supported. So it
is untrue to say that Erlang can do integer arithmetic of
arbitrary length integers. Even worse, it does not warns you
when it produces a wrong answer. Try the following:
1> math:pow(2,55).
36028797018963970.0
Obviously no integral power of 2 can end by 0.
All integral powers of 2 larger than 55 will give you a result
ending by zero. This is because Erlang has not implemented
an integer power function and relies on a float pow(x,y) function.
This is why the result is a float. Actually it is an IEEE-754 float
number in decimal64 format which means it has a maximum
precision of 16 digits.
Many applications dealing with large integers require true
large integer arithmetic (including power), for example
cryptology applications routinely do arithmetic modular
power operations on very large integers. Though Erlang
allows to write arbitrary large integers and do some basic
operations it surely cannot be said to support arlitrary large
integer arithmetic. For that, one can use mathematica (does
arbitrary large float arithmetic as well) or specialized packages
such as simod/modsim.
> CouchDB does not persist large values of Numbers correctly.
> -----------------------------------------------------------
>
> Key: COUCHDB-749
> URL: https://issues.apache.org/jira/browse/COUCHDB-749
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: All
> Reporter: Jarrod Roberson
>
> All the following operations exhibit the same bug, large numbers don't get persisted correctly. They get something added to them for some reason.
> 9223372036854775807 == java.lang.Long.MAX_VALUE
> 1: go into Futon, create a new document and create a new field and enter the number 9223372036854775807, click the green check mark, the number changes to 9223372036854776000 even before you save it.
> 2.curl -X PUT http://localhost:5984/test/longTest -d '{"value": 9223372036854775807}', the number gets persisted as 9223372036854776000
> trying to persist System.currentTimeMilliseconds() from java causes the same thing to happen occasionally.
> This seems to be a pretty serious bug if I can't trust that my data is not being corrupted when submitted to the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-749) CouchDB does not persist large
values of Numbers correctly.
Posted by "Chris Anderson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882256#action_12882256 ]
Chris Anderson commented on COUCHDB-749:
----------------------------------------
I've patched mochijson2.erl again to give it the proper behavior. The patch is tiny. I'll try sending it upstream and see what we get.
diff --git a/src/mochiweb/mochijson2.erl b/src/mochiweb/mochijson2.erl
index 66f68bf..111c37b 100644
--- a/src/mochiweb/mochijson2.erl
+++ b/src/mochiweb/mochijson2.erl
@@ -98,11 +98,8 @@ json_encode(false, _State) ->
<<"false">>;
json_encode(null, _State) ->
<<"null">>;
-json_encode(I, _State) when is_integer(I) andalso I >= -2147483648 andalso I =< 2147483647 ->
- %% Anything outside of 32-bit integers should be encoded as a float
- integer_to_list(I);
json_encode(I, _State) when is_integer(I) ->
- mochinum:digits(float(I));
+ integer_to_list(I);
json_encode(F, _State) when is_float(F) ->
mochinum:digits(F);
json_encode(S, State) when is_binary(S); is_atom(S) ->
> CouchDB does not persist large values of Numbers correctly.
> -----------------------------------------------------------
>
> Key: COUCHDB-749
> URL: https://issues.apache.org/jira/browse/COUCHDB-749
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: All
> Reporter: Jarrod Roberson
>
> All the following operations exhibit the same bug, large numbers don't get persisted correctly. They get something added to them for some reason.
> 9223372036854775807 == java.lang.Long.MAX_VALUE
> 1: go into Futon, create a new document and create a new field and enter the number 9223372036854775807, click the green check mark, the number changes to 9223372036854776000 even before you save it.
> 2.curl -X PUT http://localhost:5984/test/longTest -d '{"value": 9223372036854775807}', the number gets persisted as 9223372036854776000
> trying to persist System.currentTimeMilliseconds() from java causes the same thing to happen occasionally.
> This seems to be a pretty serious bug if I can't trust that my data is not being corrupted when submitted to the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (COUCHDB-749) CouchDB does not persist large values
of Numbers correctly.
Posted by "Chris Anderson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Anderson closed COUCHDB-749.
----------------------------------
Resolution: Fixed
fixed in 957656
> CouchDB does not persist large values of Numbers correctly.
> -----------------------------------------------------------
>
> Key: COUCHDB-749
> URL: https://issues.apache.org/jira/browse/COUCHDB-749
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: All
> Reporter: Jarrod Roberson
>
> All the following operations exhibit the same bug, large numbers don't get persisted correctly. They get something added to them for some reason.
> 9223372036854775807 == java.lang.Long.MAX_VALUE
> 1: go into Futon, create a new document and create a new field and enter the number 9223372036854775807, click the green check mark, the number changes to 9223372036854776000 even before you save it.
> 2.curl -X PUT http://localhost:5984/test/longTest -d '{"value": 9223372036854775807}', the number gets persisted as 9223372036854776000
> trying to persist System.currentTimeMilliseconds() from java causes the same thing to happen occasionally.
> This seems to be a pretty serious bug if I can't trust that my data is not being corrupted when submitted to the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-749) CouchDB does not persist large
values of Numbers correctly.
Posted by "Sebastian Cohnen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860553#action_12860553 ]
Sebastian Cohnen commented on COUCHDB-749:
------------------------------------------
I would vote for (a). If CouchDB (or any parts of it like spidermonkey) cannot handle integers above a certain size, they should be rejected. if developer need to store bigger values, they are still free to encode them differently (like using strings).
> CouchDB does not persist large values of Numbers correctly.
> -----------------------------------------------------------
>
> Key: COUCHDB-749
> URL: https://issues.apache.org/jira/browse/COUCHDB-749
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: All
> Reporter: Jarrod Roberson
>
> All the following operations exhibit the same bug, large numbers don't get persisted correctly. They get something added to them for some reason.
> 9223372036854775807 == java.lang.Long.MAX_VALUE
> 1: go into Futon, create a new document and create a new field and enter the number 9223372036854775807, click the green check mark, the number changes to 9223372036854776000 even before you save it.
> 2.curl -X PUT http://localhost:5984/test/longTest -d '{"value": 9223372036854775807}', the number gets persisted as 9223372036854776000
> trying to persist System.currentTimeMilliseconds() from java causes the same thing to happen occasionally.
> This seems to be a pretty serious bug if I can't trust that my data is not being corrupted when submitted to the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-749) CouchDB does not persist large
values of Numbers correctly.
Posted by "Chris Anderson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860711#action_12860711 ]
Chris Anderson commented on COUCHDB-749:
----------------------------------------
Since we are only concerned with storage and round-trip, it doesn't matter about Erlang arithmetic.
I think we should patch our JSON parser to handle arbitrary ints again.
We should clearly document that the view server doesn't have the same integer guarantees as the storage engine. I don't agree that we should adopt spidermonkey's limitations. The JSON spec is very clear on the fact that a valid JSON implementation may support only very small numbers. (Eg you can round everything to 0 and still be valid JSON.) That's no reason for us to act that way.
> CouchDB does not persist large values of Numbers correctly.
> -----------------------------------------------------------
>
> Key: COUCHDB-749
> URL: https://issues.apache.org/jira/browse/COUCHDB-749
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: All
> Reporter: Jarrod Roberson
>
> All the following operations exhibit the same bug, large numbers don't get persisted correctly. They get something added to them for some reason.
> 9223372036854775807 == java.lang.Long.MAX_VALUE
> 1: go into Futon, create a new document and create a new field and enter the number 9223372036854775807, click the green check mark, the number changes to 9223372036854776000 even before you save it.
> 2.curl -X PUT http://localhost:5984/test/longTest -d '{"value": 9223372036854775807}', the number gets persisted as 9223372036854776000
> trying to persist System.currentTimeMilliseconds() from java causes the same thing to happen occasionally.
> This seems to be a pretty serious bug if I can't trust that my data is not being corrupted when submitted to the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-749) CouchDB does not persist large
values of Numbers correctly.
Posted by "Jarrod Roberson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860547#action_12860547 ]
Jarrod Roberson commented on COUCHDB-749:
-----------------------------------------
no, in test case 2 with curl the number 9223372036854775807 gets persisted as an integer, just not the correct integer it is 9223372036854776000 no decimal places.
I think this is a SERIOUS flaw regardless to WHY it is a flaw. Imagine if Oracle or SQL Server had a flaw that didn't store data correctly. It would be be trusted by many people.
I think whatever solution you pick it has to be consistent and preserve the data that is sent in correctly. If that means all numbers get stored as strings and that is the only way to make sure the data is correct then make them all strings.
> CouchDB does not persist large values of Numbers correctly.
> -----------------------------------------------------------
>
> Key: COUCHDB-749
> URL: https://issues.apache.org/jira/browse/COUCHDB-749
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: All
> Reporter: Jarrod Roberson
>
> All the following operations exhibit the same bug, large numbers don't get persisted correctly. They get something added to them for some reason.
> 9223372036854775807 == java.lang.Long.MAX_VALUE
> 1: go into Futon, create a new document and create a new field and enter the number 9223372036854775807, click the green check mark, the number changes to 9223372036854776000 even before you save it.
> 2.curl -X PUT http://localhost:5984/test/longTest -d '{"value": 9223372036854775807}', the number gets persisted as 9223372036854776000
> trying to persist System.currentTimeMilliseconds() from java causes the same thing to happen occasionally.
> This seems to be a pretty serious bug if I can't trust that my data is not being corrupted when submitted to the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-749) CouchDB does not persist large
values of Numbers correctly.
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860333#action_12860333 ]
Adam Kocoloski commented on COUCHDB-749:
----------------------------------------
Hi Jared, in case 2) the number is actually persisted as 9223372036854776000.0, right?
The current behaviour of CouchDB's JSON decoder (mochijson2) is that numbers larger than the 32 bit integer limit are decoded as floating-point. That's why you observe the rounding error in Case 2. It's an arbitrary restriction; we could certainly store them unaltered using Erlang's BigNums. On the other hand, any time you tried to access the number in a javascript view you'd lose information, due to the fact that ECMAScript uses double precision floating-point for _all_ numbers.
The Futon issue in 1) is also due a limitation of javascript iirc.
I'm not sure what the right solution is here:
a) reject bignums
b) store them as floating-point
c) store them unaltered, but the view engine will still silently change them
d) store them as strings
As an app developer I would choose d). We used to do c) in CouchDB, but at one point we reverted to the default behavior of mochijson2.
Given that the view engine cannot correctly work with numbers that have more than 15 digits of precision, what would you have us do?
> CouchDB does not persist large values of Numbers correctly.
> -----------------------------------------------------------
>
> Key: COUCHDB-749
> URL: https://issues.apache.org/jira/browse/COUCHDB-749
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: All
> Reporter: Jarrod Roberson
>
> All the following operations exhibit the same bug, large numbers don't get persisted correctly. They get something added to them for some reason.
> 9223372036854775807 == java.lang.Long.MAX_VALUE
> 1: go into Futon, create a new document and create a new field and enter the number 9223372036854775807, click the green check mark, the number changes to 9223372036854776000 even before you save it.
> 2.curl -X PUT http://localhost:5984/test/longTest -d '{"value": 9223372036854775807}', the number gets persisted as 9223372036854776000
> trying to persist System.currentTimeMilliseconds() from java causes the same thing to happen occasionally.
> This seems to be a pretty serious bug if I can't trust that my data is not being corrupted when submitted to the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.