[jira] [Commented] (SOLR-6364) _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ ==> suggested resolution: json should communicate _version_ as string!

["Hoss Man (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093395#comment-14093395 ] 

Hoss Man commented on SOLR-6364:
--------------------------------

bq. There seems to be a 100 based rounding active in the output/rendition/return of the version field of added documents.

No such rounding is occurring in solr (which is why you can't reproduce when you hit solr from curl).  What you are seeing appears to be the behavior of the javascript engine in your browser when it parses JSON containing numeric values larger then the javascript spec allows for numbers. (i see the same behavior in firefox)

bq. There are more references (below) talking about this being a Javascript limitation rather then a pure json-spec issue...

correct, the JSON spec has no limitiation on hte size of a numeric value.

bq. ...it might be easier to adapt solr to deal with this know Javascript limitation and thus helping out the Javascript clients out there?

we certainly should *not* change the default behavior of solr's JSON response format, since many languages have no problems with java "long" values in JSON -- but it does seem like it might be a good idea to add an option for dealing with this better, since it would certainly affect any javascript client parsing any "long" value (not just {{_version_}}

---

My suggestion would be a new "json.long" param (following the naming convention of "json.nl" and "json.wrf" params) that would control wether java "long" values should be returned as numerics or strings in the JSON response.



> _version_ value too big for javascript clients causing reported _version_ never matching internal _version_ ==> suggested resolution: json should communicate _version_ as string! 
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-6364
>                 URL: https://issues.apache.org/jira/browse/SOLR-6364
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 4.9
>         Environment: ubuntu 14.04 desktop 32bit
> Oracle Corporation Java HotSpot(TM) Server VM (1.7.0_45 24.45-b08)
> lucene-spec 4.9.0
> lucene-impl 4.9.0 1604085 - rmuir - 2014-06-20 06:22:23
> solr-spec  4.9.0
> solr-impl  4.9.0 1604085 - rmuir - 2014-06-20 06:34:03
>            Reporter: Marc Portier
>
> There seems to be a 100 based rounding active in the output/rendition/return of the _version_ field of added documents.  Internally on the solr side however the real number (non-rounded) is effective and introduces conflicts with the optimistic concurrency logic.
> Apparently this is to be expected in all Javascript clients, since the _version_ numbers used are too big to fit into Javascript Number variables without loss of precision.
> Here is what one can do to see this in action - all steps below with 
> 1/ using the solr4 admin UI on 
>     http://localhost:8983/solr/#/mycore/documents
> 2/ the request-handler box set to 
>     /update?commit=true&versions=true
> 3/ by adding the following into the "documents"  section on the page:
> [1] for create 
> Using:
> { "id": "tst-abcd, "version": 1, "type": "test", "title": ["title"], "_version_": -1 }
> Response:
> {  "responseHeader": {    "status": 0,    "QTime": 1882  },
>   "adds": [    "tst-abcd",    1476172747866374100  ]
> }
> >> see the returned __version__ is a multiple of 100, always!
> [2] update
> Using:
> { "id": "tst-abcd", "version": 2, "type": "test", "title": ["title update"], "_version_": 1476172747866374100 }
> Response Error:
> {  "responseHeader": {    "status": 409,    "QTime": 51  },
>   "error": {    "msg": "version conflict for tst-abcd expected=1476172747866374100 actual=1476172747866374144",
>     "code": 409  }}
> >> notice how the error-message-string correctly mentions the real actual __version__ that is effective (not rounded to 100)
> [3] corrected update, using that effective number
> { "id": "tst-abcd", "version": 2, "type": "test", "title": ["title update"], "_version_": 1476172747866374144 }
> Response:
> {  "responseHeader": {    "status": 0,    "QTime": 597  },
>   "adds": [    "tst-abcd",    1476173026894545000  ] }
> Odd at first this behaviour is not shown with curl on the command line...
> [1] create
> $ curl "$solrbase/update?commit=true&versions=true" -H 'Content-type:application/json' -d '[{ "id": "tst-1234", "version": 1, "type": "test", "title": ["title"], "_version_": -1 }]'
> response: 
> {"responseHeader":{"status":0,"QTime":587},"adds":["tst-1234",1476163269470191616]}
> >> number is not rounded, looks good!
> [2] update 
> $ curl "$solrbase/update?commit=true&versions=true" -H 'Content-type:application/json' -d '[{ "id": "tst-1234", "version": 2, "type": "test", "title": ["title updated"], "_version_": 1476163269470191616 }]'
> response: 
> {"responseHeader":{"status":0,"QTime":512},"adds":["tst-1234",1476163320472928256]}
> All this was pretty much a mistery to me untill I came across this:
> http://stackoverflow.com/questions/15689790/parse-json-in-javascript-long-numbers-get-rounded
> This looks like passing down the too big numbers in the __version__ as strings should avoid the issue. Or use numbers that aren't that big, since apparently: "The largest number JavaScript can handle without loss of precision is 9007199254740992"  -- quoted from that stackoverflow page.
> There are more references (below) talking about this being a Javascript limitation rather then a pure json-spec issue, nevertheless... it might be easier to adapt solr to deal with this know Javascript limitation and thus helping out the Javascript clients out there?
> - http://stackoverflow.com/questions/307179/what-is-javascripts-max-int-whats-the-highest-integer-value-a-number-can-go-t
> - http://stackoverflow.com/questions/13502398/json-integers-limit-on-size
> In terms of backwards compatibility I don't see an easy way out for the moment.  
> - clients that expect __version__ to be numeric might not handle the string
> - in existing deployments it might be hard to reduce all the already existing __version__ to stay undere the limit...
> I still have to investigate into receiving and parsing XML replies from SOLR instead - making sure I keep the returned __version__ info in a Javascript string.  Hoping that might work as a timely (but not as elegant) workaround.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org