You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Thomas Corthals <th...@klascement.net> on 2024/09/14 13:10:47 UTC
Solr 9.7.0 returns fields in different order with fl parameter
Hi,
Up until Solr 9.6.1, when using the fl parameter the fields were returned
in what seems to be the same order they are stored in. With Solr 9.7.0,
they are returned in a different order.
With the techproducts example on Solr 9.6.1:
{
"responseHeader":{
"status":0,
"QTime":2,
"params":{
"q":"price:[70 TO 80]",
"fl":"id,price"
}
},
"response":{
"numFound":1,
"start":0,
"numFoundExact":true,
"docs":[{
"id":"VS1GB400C3",
"price":74.99
}]
}
}
Same query and field list on Solr 9.7.0:
{
"responseHeader":{
"status":0,
"QTime":44,
"params":{
"q":"price:[70 TO 80]",
"fl":"id,price"
}
},
"response":{
"numFound":1,
"start":0,
"numFoundExact":true,
"docs":[{
"price":74.99,
"id":"VS1GB400C3"
}]
}
}
Is there a way to keep the returned field order consistent with how it used
to be? I'm looking at dozens of failed integration tests because of this
changed field order. And I can't change the order in the expected outcome
because the tests have to run against multiple Solr versions.
Thomas
Re: Solr 9.7.0 returns fields in different order with fl parameter
Posted by Thomas Corthals <th...@klascement.net>.
Thanks for the Jira link.
In another case that I know of, Luke requests can output JSON with
duplicate keys if you use show=doc and the document contains multiValued
fields. Always fun to parse in a language that doesn't support such a
structure.
Op za 14 sep 2024 21:16 schreef Gus Heck <gu...@gmail.com>:
> Unless something changed while I wasn't paying attention, order of fields
> in a response is not guaranteed, and some searching in Jira seems to
> confirm that ordering of fields in the response is a WONTFIX since 1.3:
> https://issues.apache.org/jira/browse/SOLR-1190
>
> One would not normally expect ordered keys in JSON anyway... According to
> its spec name/value pairs are unordered: https://www.json.org/json-en.html
> --> "An *object* is an unordered set of name/value pairs."
>
> There are a lot of places in our code where we do use LinkedHashMap to hold
> the fields, so reading the code might be slightly misleading, but I believe
> this is for the iteration cost benefits (scales vs actual entries, rather
> vs capacity) more than ordering (some exceptions in our codebase, see rant
> below). This use of LinkedHashMap may have led to an undocumented
> consistency that you have (unfortunately) come to rely upon.
>
> <rant tags="old, tired, hard to fix">
> There are some ways in which solr's usage of JSON is a bit dodgy (accepting
> illegal commas in some places instead of throwing an error... an
> intentional feature of noggit, but not one I like), and in one case that I
> know of, duplicate keys are allowed (in
>
> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-update-handlers.html#sending-json-update-commands
> )
> which IS *technically* legal json, but actively discouraged by the RFC:
> https://www.rfc-editor.org/rfc/rfc7159#section-4. Since the keys in that
> update format denote operations to be performed, order is significant by
> implication... something explicitly not guaranteed by the spec. I've always
> been annoyed by this design because it relies on behavior that the JSON
> spec doesn't ask fully compliant parsers to provide, and it relies on
> duplicate keys that many folks will fail to anticipate because it's so rare
> and not well supported by libraries. Even in the simpler json.org spec,
> there is reference to objects being intended to represent structures in
> other languages such as "*object*, record, struct, dictionary, hash table,
> keyed list, or associative array", most of which don't allow duplicate
> keys, so one could infer some intent even if they failed to exclude the
> behavior explicitly. Any mapping of the object into either Java or
> Javascript for manipulation requires custom processing, and custom data
> structures rather than a simple object mapper... should have been [ {
> "operation":"ADD", "parameters": {...}}, ... ] which is ordered and easily
> mapped with default mapper configurations widely available everywhere...
> </rant>
>
>
>
> On Sat, Sep 14, 2024 at 9:11 AM Thomas Corthals <th...@klascement.net>
> wrote:
>
> > Hi,
> >
> > Up until Solr 9.6.1, when using the fl parameter the fields were returned
> > in what seems to be the same order they are stored in. With Solr 9.7.0,
> > they are returned in a different order.
> >
> > With the techproducts example on Solr 9.6.1:
> >
> > {
> > "responseHeader":{
> > "status":0,
> > "QTime":2,
> > "params":{
> > "q":"price:[70 TO 80]",
> > "fl":"id,price"
> > }
> > },
> > "response":{
> > "numFound":1,
> > "start":0,
> > "numFoundExact":true,
> > "docs":[{
> > "id":"VS1GB400C3",
> > "price":74.99
> > }]
> > }
> > }
> >
> > Same query and field list on Solr 9.7.0:
> >
> > {
> > "responseHeader":{
> > "status":0,
> > "QTime":44,
> > "params":{
> > "q":"price:[70 TO 80]",
> > "fl":"id,price"
> > }
> > },
> > "response":{
> > "numFound":1,
> > "start":0,
> > "numFoundExact":true,
> > "docs":[{
> > "price":74.99,
> > "id":"VS1GB400C3"
> > }]
> > }
> > }
> >
> > Is there a way to keep the returned field order consistent with how it
> used
> > to be? I'm looking at dozens of failed integration tests because of this
> > changed field order. And I can't change the order in the expected outcome
> > because the tests have to run against multiple Solr versions.
> >
> > Thomas
> >
>
>
> --
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>
Re: Solr 9.7.0 returns fields in different order with fl parameter
Posted by Gus Heck <gu...@gmail.com>.
Unless something changed while I wasn't paying attention, order of fields
in a response is not guaranteed, and some searching in Jira seems to
confirm that ordering of fields in the response is a WONTFIX since 1.3:
https://issues.apache.org/jira/browse/SOLR-1190
One would not normally expect ordered keys in JSON anyway... According to
its spec name/value pairs are unordered: https://www.json.org/json-en.html
--> "An *object* is an unordered set of name/value pairs."
There are a lot of places in our code where we do use LinkedHashMap to hold
the fields, so reading the code might be slightly misleading, but I believe
this is for the iteration cost benefits (scales vs actual entries, rather
vs capacity) more than ordering (some exceptions in our codebase, see rant
below). This use of LinkedHashMap may have led to an undocumented
consistency that you have (unfortunately) come to rely upon.
<rant tags="old, tired, hard to fix">
There are some ways in which solr's usage of JSON is a bit dodgy (accepting
illegal commas in some places instead of throwing an error... an
intentional feature of noggit, but not one I like), and in one case that I
know of, duplicate keys are allowed (in
https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-update-handlers.html#sending-json-update-commands)
which IS *technically* legal json, but actively discouraged by the RFC:
https://www.rfc-editor.org/rfc/rfc7159#section-4. Since the keys in that
update format denote operations to be performed, order is significant by
implication... something explicitly not guaranteed by the spec. I've always
been annoyed by this design because it relies on behavior that the JSON
spec doesn't ask fully compliant parsers to provide, and it relies on
duplicate keys that many folks will fail to anticipate because it's so rare
and not well supported by libraries. Even in the simpler json.org spec,
there is reference to objects being intended to represent structures in
other languages such as "*object*, record, struct, dictionary, hash table,
keyed list, or associative array", most of which don't allow duplicate
keys, so one could infer some intent even if they failed to exclude the
behavior explicitly. Any mapping of the object into either Java or
Javascript for manipulation requires custom processing, and custom data
structures rather than a simple object mapper... should have been [ {
"operation":"ADD", "parameters": {...}}, ... ] which is ordered and easily
mapped with default mapper configurations widely available everywhere...
</rant>
On Sat, Sep 14, 2024 at 9:11 AM Thomas Corthals <th...@klascement.net>
wrote:
> Hi,
>
> Up until Solr 9.6.1, when using the fl parameter the fields were returned
> in what seems to be the same order they are stored in. With Solr 9.7.0,
> they are returned in a different order.
>
> With the techproducts example on Solr 9.6.1:
>
> {
> "responseHeader":{
> "status":0,
> "QTime":2,
> "params":{
> "q":"price:[70 TO 80]",
> "fl":"id,price"
> }
> },
> "response":{
> "numFound":1,
> "start":0,
> "numFoundExact":true,
> "docs":[{
> "id":"VS1GB400C3",
> "price":74.99
> }]
> }
> }
>
> Same query and field list on Solr 9.7.0:
>
> {
> "responseHeader":{
> "status":0,
> "QTime":44,
> "params":{
> "q":"price:[70 TO 80]",
> "fl":"id,price"
> }
> },
> "response":{
> "numFound":1,
> "start":0,
> "numFoundExact":true,
> "docs":[{
> "price":74.99,
> "id":"VS1GB400C3"
> }]
> }
> }
>
> Is there a way to keep the returned field order consistent with how it used
> to be? I'm looking at dozens of failed integration tests because of this
> changed field order. And I can't change the order in the expected outcome
> because the tests have to run against multiple Solr versions.
>
> Thomas
>
--
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)