You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Marc Portier (JIRA)" <ji...@apache.org> on 2014/08/11 22:50:15 UTC
[jira] [Updated] (SOLR-6364) _version_ value too big for javascript
clients causing reported _version_ never matching internal _version_ ==>
suggested resolution: json should communicate _version_ as string!
[ https://issues.apache.org/jira/browse/SOLR-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marc Portier updated SOLR-6364:
-------------------------------
Description:
There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents. Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic.
Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision.
Here is what one can do to see this in action - all steps below with
1/ using the solr4 admin UI on
http://localhost:8983/solr/#/mycore/documents
2/ the request-handler box set to
/update?commit=true&versions=true
3/ by adding the following into the "documents" section on the page:
[1] for create
Using:
{ "id": "tst-abcd, "version": 1, "type": "test", "title": ["title"], "_version_": -1 }
Response:
{ "responseHeader": { "status": 0, "QTime": 1882 },
"adds": [ "tst-abcd", 1476172747866374100 ]
}
>> see the returned __version__ is a multiple of 100, always!
[2] update
Using:
{ "id": "tst-abcd", "version": 2, "type": "test", "title": ["title update"], "_version_": 1476172747866374100 }
Response Error:
{ "responseHeader": { "status": 409, "QTime": 51 },
"error": { "msg": "version conflict for tst-abcd expected=1476172747866374100 actual=1476172747866374144",
"code": 409 }}
>> notice how the error-message-string correctly mentions the real actual __version__ that is effective (not rounded to 100)
[3] corrected update, using that effective number
{ "id": "tst-abcd", "version": 2, "type": "test", "title": ["title update"], "_version_": 1476172747866374144 }
Response:
{ "responseHeader": { "status": 0, "QTime": 597 },
"adds": [ "tst-abcd", 1476173026894545000 ] }
Odd at first this behaviour is not shown with curl on the command line...
[1] create
$ curl "$solrbase/update?commit=true&versions=true" -H 'Content-type:application/json' -d '[{ "id": "tst-1234", "version": 1, "type": "test", "title": ["title"], "_version_": -1 }]'
response:
{"responseHeader":{"status":0,"QTime":587},"adds":["tst-1234",1476163269470191616]}
>> number is not rounded, looks good!
[2] update
$ curl "$solrbase/update?commit=true&versions=true" -H 'Content-type:application/json' -d '[{ "id": "tst-1234", "version": 2, "type": "test", "title": ["title updated"], "_version_": 1476163269470191616 }]'
response:
{"responseHeader":{"status":0,"QTime":512},"adds":["tst-1234",1476163320472928256]}
All this was pretty much a mistery to me untill I came across this:
http://stackoverflow.com/questions/15689790/parse-json-in-javascript-long-numbers-get-rounded
This looks like passing down the too big numbers in the __version__ as strings should avoid the issue. Or use numbers that aren't that big, since apparently: "The largest number JavaScript can handle without loss of precision is 9007199254740992" -- quoted from that stackoverflow page.
There are more references (below) talking about this being a Javascript limitation rather then a pure json-spec issue, nevertheless... it might be easier to adapt solr to deal with this know Javascript limitation and thus helping out the Javascript clients out there?
- http://stackoverflow.com/questions/307179/what-is-javascripts-max-int-whats-the-highest-integer-value-a-number-can-go-t
- http://stackoverflow.com/questions/13502398/json-integers-limit-on-size
In terms of backwards compatibility I don't see an easy way out for the moment.
- clients that expect __version__ to be numeric might not handle the string
- in existing deployments it might be hard to reduce all the already existing __version__ to stay undere the limit...
I still have to investigate into receiving and parsing XML replies from SOLR instead - making sure I keep the returned __version__ info in a Javascript string. Hoping that might work as a timely (but not as elegant) workaround.
was:
There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents. Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic.
Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision.
Here is what one can do to see this in action - all steps below with
1/ using the solr4 admin UI on
http://localhost:8983/solr/#/mycore/documents
2/ the request-handler box set to
/update?commit=true&versions=true
3/ by adding the following into the "documents" section on the page:
[1] for create
Using:
{ "id": "tst-abcd, "version": 1, "type": "test", "title": ["title"], "_version_": -1 }
Response:
{ "responseHeader": { "status": 0, "QTime": 1882 },
"adds": [ "tst-abcd", 1476172747866374100 ]
}
>> see the returned _version_ is a multiple of 100, always!
[2] update
Using:
{ "id": "tst-abcd", "version": 2, "type": "test", "title": ["title update"], "_version_": 1476172747866374100 }
Response Error:
{ "responseHeader": { "status": 409, "QTime": 51 },
"error": { "msg": "version conflict for tst-abcd expected=1476172747866374100 actual=1476172747866374144",
"code": 409 }}
>> notice how the error-message-string correctly mentions the real actual _version_ that is effective (not rounded to 100)
[3] corrected update, using that effective number
{ "id": "tst-abcd", "version": 2, "type": "test", "title": ["title update"], "_version_": 1476172747866374144 }
Response:
{ "responseHeader": { "status": 0, "QTime": 597 },
"adds": [ "tst-abcd", 1476173026894545000 ] }
Odd at first this behaviour is not shown with curl on the command line...
[1] create
$ curl "$solrbase/update?commit=true&versions=true" -H 'Content-type:application/json' -d '[{ "id": "tst-1234", "version": 1, "type": "test", "title": ["title"], "_version_": -1 }]'
response:
{"responseHeader":{"status":0,"QTime":587},"adds":["tst-1234",1476163269470191616]}
>> number is not rounded, looks good!
[2] update
$ curl "$solrbase/update?commit=true&versions=true" -H 'Content-type:application/json' -d '[{ "id": "tst-1234", "version": 2, "type": "test", "title": ["title updated"], "_version_": 1476163269470191616 }]'
response:
{"responseHeader":{"status":0,"QTime":512},"adds":["tst-1234",1476163320472928256]}
All this was pretty much a mistery to me untill I came across this:
http://stackoverflow.com/questions/15689790/parse-json-in-javascript-long-numbers-get-rounded
This looks like passing down the too big numbers in the _version_ as strings should avoid the issue. Or use numbers that aren't that big, since apparently: "The largest number JavaScript can handle without loss of precision is 9007199254740992" -- quoted from that stackoverflow page.
There are more references (below) talking about this being a Javascript limitation rather then a pure json-spec issue, nevertheless... it might be easier to adapt solr to deal with this know Javascript limitation and thus helping out the Javascript clients out there?
- http://stackoverflow.com/questions/307179/what-is-javascripts-max-int-whats-the-highest-integer-value-a-number-can-go-t
- http://stackoverflow.com/questions/13502398/json-integers-limit-on-size
In terms of backwards compatibility I don't see an easy way out for the moment.
- clients that expect _version_ to be numeric might not handle the string
- in existing deployments it might be hard to reduce all the already existing _version_ to stay undere the limit...
I still have to investigate into receiving and parsing XML replies from SOLR instead - making sure I keep the returned _version_ info in a Javascript string. Hoping that might work as a timely (but not as elegant) workaround.
> _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ ==> suggested resolution: json should communicate _version_ as string!
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-6364
> URL: https://issues.apache.org/jira/browse/SOLR-6364
> Project: Solr
> Issue Type: Bug
> Components: update
> Affects Versions: 4.9
> Environment: ubuntu 14.04 desktop 32bit
> Oracle Corporation Java HotSpot(TM) Server VM (1.7.0_45 24.45-b08)
> lucene-spec 4.9.0
> lucene-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:22:23
> solr-spec 4.9.0
> solr-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:34:03
> Reporter: Marc Portier
>
> There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents. Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic.
> Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision.
> Here is what one can do to see this in action - all steps below with
> 1/ using the solr4 admin UI on
> http://localhost:8983/solr/#/mycore/documents
> 2/ the request-handler box set to
> /update?commit=true&versions=true
> 3/ by adding the following into the "documents" section on the page:
> [1] for create
> Using:
> { "id": "tst-abcd, "version": 1, "type": "test", "title": ["title"], "_version_": -1 }
> Response:
> { "responseHeader": { "status": 0, "QTime": 1882 },
> "adds": [ "tst-abcd", 1476172747866374100 ]
> }
> >> see the returned __version__ is a multiple of 100, always!
> [2] update
> Using:
> { "id": "tst-abcd", "version": 2, "type": "test", "title": ["title update"], "_version_": 1476172747866374100 }
> Response Error:
> { "responseHeader": { "status": 409, "QTime": 51 },
> "error": { "msg": "version conflict for tst-abcd expected=1476172747866374100 actual=1476172747866374144",
> "code": 409 }}
> >> notice how the error-message-string correctly mentions the real actual __version__ that is effective (not rounded to 100)
> [3] corrected update, using that effective number
> { "id": "tst-abcd", "version": 2, "type": "test", "title": ["title update"], "_version_": 1476172747866374144 }
> Response:
> { "responseHeader": { "status": 0, "QTime": 597 },
> "adds": [ "tst-abcd", 1476173026894545000 ] }
> Odd at first this behaviour is not shown with curl on the command line...
> [1] create
> $ curl "$solrbase/update?commit=true&versions=true" -H 'Content-type:application/json' -d '[{ "id": "tst-1234", "version": 1, "type": "test", "title": ["title"], "_version_": -1 }]'
> response:
> {"responseHeader":{"status":0,"QTime":587},"adds":["tst-1234",1476163269470191616]}
> >> number is not rounded, looks good!
> [2] update
> $ curl "$solrbase/update?commit=true&versions=true" -H 'Content-type:application/json' -d '[{ "id": "tst-1234", "version": 2, "type": "test", "title": ["title updated"], "_version_": 1476163269470191616 }]'
> response:
> {"responseHeader":{"status":0,"QTime":512},"adds":["tst-1234",1476163320472928256]}
> All this was pretty much a mistery to me untill I came across this:
> http://stackoverflow.com/questions/15689790/parse-json-in-javascript-long-numbers-get-rounded
> This looks like passing down the too big numbers in the __version__ as strings should avoid the issue. Or use numbers that aren't that big, since apparently: "The largest number JavaScript can handle without loss of precision is 9007199254740992" -- quoted from that stackoverflow page.
> There are more references (below) talking about this being a Javascript limitation rather then a pure json-spec issue, nevertheless... it might be easier to adapt solr to deal with this know Javascript limitation and thus helping out the Javascript clients out there?
> - http://stackoverflow.com/questions/307179/what-is-javascripts-max-int-whats-the-highest-integer-value-a-number-can-go-t
> - http://stackoverflow.com/questions/13502398/json-integers-limit-on-size
> In terms of backwards compatibility I don't see an easy way out for the moment.
> - clients that expect __version__ to be numeric might not handle the string
> - in existing deployments it might be hard to reduce all the already existing __version__ to stay undere the limit...
> I still have to investigate into receiving and parsing XML replies from SOLR instead - making sure I keep the returned __version__ info in a Javascript string. Hoping that might work as a timely (but not as elegant) workaround.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org