You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Jane Sandberg <js...@princeton.edu> on 2023/07/28 14:33:05 UTC
JSON boolean query syntax with edismax as default QueryParser
Hi Solr colleagues,
On Solr 8.4.1, we’ve noticed that the following types of JSON DSL queries work if our luceneMatchVersion is 7.1 or lower, or if our default query parser is set to lucene:
{"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
However, if the query parser is set to edismax and the luceneMatchVersion is 7.2 or higher, the parsed query visible with debug=true becomes a complete mess, searching for the terms “bool” and “must”, rather than the terms we actually want to search for:
+(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 | (local_subject_unstem_search:bool)^15.0 | (author_unstem_search:bool)^40.0 […]
Also while debug=true, we noticed that the JSON DSL queries get converted into a querystring with local params: ”{!bool must=$_tt1 }”. So I am suspecting these two changes in Solr 7.2 as the reason we can’t use Boolean JSON queries with edismax and a recent luceneMatchVersion: https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes. Does that seem correct?
Also, could this be related to the question Benjamin Armintor asked on June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m specifically curious about whether a luceneMatchVersion of 7.1 or lower still works in Solr 9?
Thanks for your insights,
-Jane
--
Jane Sandberg (she/her)
Library Software Engineer, Discovery and Access Services
Re: JSON boolean query syntax with edismax as default QueryParser
Posted by Jane Sandberg <js...@princeton.edu>.
Thanks so much, David! I’d be happy to submit a PR if you point me in the right direction. And we’ll work on setting up a separate requestHandler for our advanced search in the meantime.
-Jane
From: David Smiley <da...@gmail.com>
Date: Tuesday, August 1, 2023 at 10:52 AM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: JSON boolean query syntax with edismax as default QueryParser
I filed https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSOLR-16916&data=05%7C01%7Cjs7389%40princeton.edu%7C37b82c5c00014b65d45008db92b81c9a%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638265091687761654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KXRIt3w61BIbUmnzdrtiMrir4z4EP8qIWJdSZ0TmyXc%3D&reserved=0<https://issues.apache.org/jira/browse/SOLR-16916>
And I also see in a debugger what's going on; I can advise you on an
internal fix if you wish to submit a PR. It'll be a one-liner, plus
whatever is needed for a test. Until this is resolved, simply use another
request handler just for JSON query DSL that does not change defType from
it's default "lucene" value.
~ David
On Tue, Aug 1, 2023 at 9:09 AM Jane Sandberg <js...@princeton.edu> wrote:
> defType=edismax is what we have historically used in our application.
> We’re only trying to use the JSON query DSL for one feature (an advanced
> search form), and are planning to continue using edismax for all the rest.
>
> -Jane
>
> From: David Smiley <da...@gmail.com>
> Date: Monday, July 31, 2023 at 8:53 PM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: JSON boolean query syntax with edismax as default QueryParser
> Why did you set defType=edismax in the /query handler in your
> solrconfig.xml?
>
> edismax is for handling user queries -- text directly from a text box.
> JSON Query DSL obviously isn't that.
>
> ~ David
>
>
> On Mon, Jul 31, 2023 at 3:09 PM Jane Sandberg <js...@princeton.edu>
> wrote:
>
> > Hi David,
> >
> > Thanks for looking into this, and for the security fix.
> >
> > My colleague and I put together a small repository to reproduce the
> > issue. It has a configset, a docker-compose file, and a README with the
> > steps to reproduce it on solr 8.4:
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpulibrary%2Fedismax-json-queries%2F&data=05%7C01%7Cjs7389%40princeton.edu%7C37b82c5c00014b65d45008db92b81c9a%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638265091687761654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=psWXR5qOZ8KlZdS6U70jlonu7TTBGUL0vQZM%2FwCN10Y%3D&reserved=0<https://github.com/pulibrary/edismax-json-queries/>
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpulibrary%2Fedismax-json-queries%2F&data=05%7C01%7Cjs7389%40princeton.edu%7C37b82c5c00014b65d45008db92b81c9a%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638265091687761654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=psWXR5qOZ8KlZdS6U70jlonu7TTBGUL0vQZM%2FwCN10Y%3D&reserved=0<https://github.com/pulibrary/edismax-json-queries/>>
> >
> > Appreciatively,
> >
> > -Jane
> >
> > From: David Smiley <da...@gmail.com>
> > Date: Monday, July 31, 2023 at 5:58 AM
> > To: users@solr.apache.org <us...@solr.apache.org>
> > Subject: Re: JSON boolean query syntax with edismax as default
> QueryParser
> > Hi Jane,
> >
> > That change in 7.2 does look like it's correlated, and I'm the one who
> > implemented it in the name of security.
> >
> > Could you walk me through a short series of steps to show the problem
> with
> > one of Solr's "example" setups like techproducts? Step one is run it,
> step
> > two is you sending the query via curl. Use whatever Solr version you
> want.
> >
> > ~ David
> >
> >
> > On Fri, Jul 28, 2023 at 10:33 AM Jane Sandberg <js...@princeton.edu>
> > wrote:
> >
> > > Hi Solr colleagues,
> > >
> > > On Solr 8.4.1, we’ve noticed that the following types of JSON DSL
> queries
> > > work if our luceneMatchVersion is 7.1 or lower, or if our default query
> > > parser is set to lucene:
> > >
> > >
> > >
> >
> {"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
> > >
> > > However, if the query parser is set to edismax and the
> luceneMatchVersion
> > > is 7.2 or higher, the parsed query visible with debug=true becomes a
> > > complete mess, searching for the terms “bool” and “must”, rather than
> the
> > > terms we actually want to search for:
> > >
> > >
> > > +(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 |
> > > (local_subject_unstem_search:bool)^15.0 |
> > (author_unstem_search:bool)^40.0
> > > […]
> > >
> > > Also while debug=true, we noticed that the JSON DSL queries get
> converted
> > > into a querystring with local params: ”{!bool must=$_tt1 }”. So I am
> > > suspecting these two changes in Solr 7.2 as the reason we can’t use
> > Boolean
> > > JSON queries with edismax and a recent luceneMatchVersion:
> > >
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7C37b82c5c00014b65d45008db92b81c9a%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638265091687761654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ez4YRZ2PUg1CXqPb3yc97iWhXCaXFFsSrndb3%2FO2sFo%3D&reserved=0<https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes>
> <
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7C37b82c5c00014b65d45008db92b81c9a%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638265091687761654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ez4YRZ2PUg1CXqPb3yc97iWhXCaXFFsSrndb3%2FO2sFo%3D&reserved=0<https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes>
> >
> > <
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7C37b82c5c00014b65d45008db92b81c9a%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638265091687761654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ez4YRZ2PUg1CXqPb3yc97iWhXCaXFFsSrndb3%2FO2sFo%3D&reserved=0<https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes>
> <
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7C37b82c5c00014b65d45008db92b81c9a%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638265091687761654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ez4YRZ2PUg1CXqPb3yc97iWhXCaXFFsSrndb3%2FO2sFo%3D&reserved=0<https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes>
> >
> > >.
> > > Does that seem correct?
> > >
> > > Also, could this be related to the question Benjamin Armintor asked on
> > > June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m
> > > specifically curious about whether a luceneMatchVersion of 7.1 or lower
> > > still works in Solr 9?
> > >
> > > Thanks for your insights,
> > >
> > > -Jane
> > >
> > > --
> > > Jane Sandberg (she/her)
> > > Library Software Engineer, Discovery and Access Services
> > >
> >
>
Re: JSON boolean query syntax with edismax as default QueryParser
Posted by David Smiley <da...@gmail.com>.
I filed https://issues.apache.org/jira/browse/SOLR-16916
And I also see in a debugger what's going on; I can advise you on an
internal fix if you wish to submit a PR. It'll be a one-liner, plus
whatever is needed for a test. Until this is resolved, simply use another
request handler just for JSON query DSL that does not change defType from
it's default "lucene" value.
~ David
On Tue, Aug 1, 2023 at 9:09 AM Jane Sandberg <js...@princeton.edu> wrote:
> defType=edismax is what we have historically used in our application.
> We’re only trying to use the JSON query DSL for one feature (an advanced
> search form), and are planning to continue using edismax for all the rest.
>
> -Jane
>
> From: David Smiley <da...@gmail.com>
> Date: Monday, July 31, 2023 at 8:53 PM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: JSON boolean query syntax with edismax as default QueryParser
> Why did you set defType=edismax in the /query handler in your
> solrconfig.xml?
>
> edismax is for handling user queries -- text directly from a text box.
> JSON Query DSL obviously isn't that.
>
> ~ David
>
>
> On Mon, Jul 31, 2023 at 3:09 PM Jane Sandberg <js...@princeton.edu>
> wrote:
>
> > Hi David,
> >
> > Thanks for looking into this, and for the security fix.
> >
> > My colleague and I put together a small repository to reproduce the
> > issue. It has a configset, a docker-compose file, and a README with the
> > steps to reproduce it on solr 8.4:
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpulibrary%2Fedismax-json-queries%2F&data=05%7C01%7Cjs7389%40princeton.edu%7Ca1e1def2ec954f95453d08db9242ca76%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638264587808266610%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9FoEGhqSWWk0PefmFFWzY%2Bd4bBN9sDSRtqHQa3DTlx4%3D&reserved=0
> <https://github.com/pulibrary/edismax-json-queries/>
> >
> > Appreciatively,
> >
> > -Jane
> >
> > From: David Smiley <da...@gmail.com>
> > Date: Monday, July 31, 2023 at 5:58 AM
> > To: users@solr.apache.org <us...@solr.apache.org>
> > Subject: Re: JSON boolean query syntax with edismax as default
> QueryParser
> > Hi Jane,
> >
> > That change in 7.2 does look like it's correlated, and I'm the one who
> > implemented it in the name of security.
> >
> > Could you walk me through a short series of steps to show the problem
> with
> > one of Solr's "example" setups like techproducts? Step one is run it,
> step
> > two is you sending the query via curl. Use whatever Solr version you
> want.
> >
> > ~ David
> >
> >
> > On Fri, Jul 28, 2023 at 10:33 AM Jane Sandberg <js...@princeton.edu>
> > wrote:
> >
> > > Hi Solr colleagues,
> > >
> > > On Solr 8.4.1, we’ve noticed that the following types of JSON DSL
> queries
> > > work if our luceneMatchVersion is 7.1 or lower, or if our default query
> > > parser is set to lucene:
> > >
> > >
> > >
> >
> {"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
> > >
> > > However, if the query parser is set to edismax and the
> luceneMatchVersion
> > > is 7.2 or higher, the parsed query visible with debug=true becomes a
> > > complete mess, searching for the terms “bool” and “must”, rather than
> the
> > > terms we actually want to search for:
> > >
> > >
> > > +(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 |
> > > (local_subject_unstem_search:bool)^15.0 |
> > (author_unstem_search:bool)^40.0
> > > […]
> > >
> > > Also while debug=true, we noticed that the JSON DSL queries get
> converted
> > > into a querystring with local params: ”{!bool must=$_tt1 }”. So I am
> > > suspecting these two changes in Solr 7.2 as the reason we can’t use
> > Boolean
> > > JSON queries with edismax and a recent luceneMatchVersion:
> > >
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7Ca1e1def2ec954f95453d08db9242ca76%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638264587808266610%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hMGAIWHtYmDP3Ma0huLryFW5AapvfVEMVUYP0b118jQ%3D&reserved=0
> <
> https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes
> >
> > <
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7Ca1e1def2ec954f95453d08db9242ca76%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638264587808266610%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hMGAIWHtYmDP3Ma0huLryFW5AapvfVEMVUYP0b118jQ%3D&reserved=0
> <
> https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes
> >
> > >.
> > > Does that seem correct?
> > >
> > > Also, could this be related to the question Benjamin Armintor asked on
> > > June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m
> > > specifically curious about whether a luceneMatchVersion of 7.1 or lower
> > > still works in Solr 9?
> > >
> > > Thanks for your insights,
> > >
> > > -Jane
> > >
> > > --
> > > Jane Sandberg (she/her)
> > > Library Software Engineer, Discovery and Access Services
> > >
> >
>
Re: JSON boolean query syntax with edismax as default QueryParser
Posted by Jane Sandberg <js...@princeton.edu>.
defType=edismax is what we have historically used in our application. We’re only trying to use the JSON query DSL for one feature (an advanced search form), and are planning to continue using edismax for all the rest.
-Jane
From: David Smiley <da...@gmail.com>
Date: Monday, July 31, 2023 at 8:53 PM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: JSON boolean query syntax with edismax as default QueryParser
Why did you set defType=edismax in the /query handler in your
solrconfig.xml?
edismax is for handling user queries -- text directly from a text box.
JSON Query DSL obviously isn't that.
~ David
On Mon, Jul 31, 2023 at 3:09 PM Jane Sandberg <js...@princeton.edu> wrote:
> Hi David,
>
> Thanks for looking into this, and for the security fix.
>
> My colleague and I put together a small repository to reproduce the
> issue. It has a configset, a docker-compose file, and a README with the
> steps to reproduce it on solr 8.4:
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpulibrary%2Fedismax-json-queries%2F&data=05%7C01%7Cjs7389%40princeton.edu%7Ca1e1def2ec954f95453d08db9242ca76%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638264587808266610%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9FoEGhqSWWk0PefmFFWzY%2Bd4bBN9sDSRtqHQa3DTlx4%3D&reserved=0<https://github.com/pulibrary/edismax-json-queries/>
>
> Appreciatively,
>
> -Jane
>
> From: David Smiley <da...@gmail.com>
> Date: Monday, July 31, 2023 at 5:58 AM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: JSON boolean query syntax with edismax as default QueryParser
> Hi Jane,
>
> That change in 7.2 does look like it's correlated, and I'm the one who
> implemented it in the name of security.
>
> Could you walk me through a short series of steps to show the problem with
> one of Solr's "example" setups like techproducts? Step one is run it, step
> two is you sending the query via curl. Use whatever Solr version you want.
>
> ~ David
>
>
> On Fri, Jul 28, 2023 at 10:33 AM Jane Sandberg <js...@princeton.edu>
> wrote:
>
> > Hi Solr colleagues,
> >
> > On Solr 8.4.1, we’ve noticed that the following types of JSON DSL queries
> > work if our luceneMatchVersion is 7.1 or lower, or if our default query
> > parser is set to lucene:
> >
> >
> >
> {"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
> >
> > However, if the query parser is set to edismax and the luceneMatchVersion
> > is 7.2 or higher, the parsed query visible with debug=true becomes a
> > complete mess, searching for the terms “bool” and “must”, rather than the
> > terms we actually want to search for:
> >
> >
> > +(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 |
> > (local_subject_unstem_search:bool)^15.0 |
> (author_unstem_search:bool)^40.0
> > […]
> >
> > Also while debug=true, we noticed that the JSON DSL queries get converted
> > into a querystring with local params: ”{!bool must=$_tt1 }”. So I am
> > suspecting these two changes in Solr 7.2 as the reason we can’t use
> Boolean
> > JSON queries with edismax and a recent luceneMatchVersion:
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7Ca1e1def2ec954f95453d08db9242ca76%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638264587808266610%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hMGAIWHtYmDP3Ma0huLryFW5AapvfVEMVUYP0b118jQ%3D&reserved=0<https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes>
> <
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7Ca1e1def2ec954f95453d08db9242ca76%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638264587808266610%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hMGAIWHtYmDP3Ma0huLryFW5AapvfVEMVUYP0b118jQ%3D&reserved=0<https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes>
> >.
> > Does that seem correct?
> >
> > Also, could this be related to the question Benjamin Armintor asked on
> > June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m
> > specifically curious about whether a luceneMatchVersion of 7.1 or lower
> > still works in Solr 9?
> >
> > Thanks for your insights,
> >
> > -Jane
> >
> > --
> > Jane Sandberg (she/her)
> > Library Software Engineer, Discovery and Access Services
> >
>
Re: JSON boolean query syntax with edismax as default QueryParser
Posted by David Smiley <da...@gmail.com>.
Why did you set defType=edismax in the /query handler in your
solrconfig.xml?
edismax is for handling user queries -- text directly from a text box.
JSON Query DSL obviously isn't that.
~ David
On Mon, Jul 31, 2023 at 3:09 PM Jane Sandberg <js...@princeton.edu> wrote:
> Hi David,
>
> Thanks for looking into this, and for the security fix.
>
> My colleague and I put together a small repository to reproduce the
> issue. It has a configset, a docker-compose file, and a README with the
> steps to reproduce it on solr 8.4:
> https://github.com/pulibrary/edismax-json-queries/
>
> Appreciatively,
>
> -Jane
>
> From: David Smiley <da...@gmail.com>
> Date: Monday, July 31, 2023 at 5:58 AM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: JSON boolean query syntax with edismax as default QueryParser
> Hi Jane,
>
> That change in 7.2 does look like it's correlated, and I'm the one who
> implemented it in the name of security.
>
> Could you walk me through a short series of steps to show the problem with
> one of Solr's "example" setups like techproducts? Step one is run it, step
> two is you sending the query via curl. Use whatever Solr version you want.
>
> ~ David
>
>
> On Fri, Jul 28, 2023 at 10:33 AM Jane Sandberg <js...@princeton.edu>
> wrote:
>
> > Hi Solr colleagues,
> >
> > On Solr 8.4.1, we’ve noticed that the following types of JSON DSL queries
> > work if our luceneMatchVersion is 7.1 or lower, or if our default query
> > parser is set to lucene:
> >
> >
> >
> {"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
> >
> > However, if the query parser is set to edismax and the luceneMatchVersion
> > is 7.2 or higher, the parsed query visible with debug=true becomes a
> > complete mess, searching for the terms “bool” and “must”, rather than the
> > terms we actually want to search for:
> >
> >
> > +(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 |
> > (local_subject_unstem_search:bool)^15.0 |
> (author_unstem_search:bool)^40.0
> > […]
> >
> > Also while debug=true, we noticed that the JSON DSL queries get converted
> > into a querystring with local params: ”{!bool must=$_tt1 }”. So I am
> > suspecting these two changes in Solr 7.2 as the reason we can’t use
> Boolean
> > JSON queries with edismax and a recent luceneMatchVersion:
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7Ce4c749a12c0e49600eaf08db91c5dec0%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638264051269788881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QhLXzlXcEnAp86czCzwUUHeio89aiAiAA46ZTkDrWDo%3D&reserved=0
> <
> https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes
> >.
> > Does that seem correct?
> >
> > Also, could this be related to the question Benjamin Armintor asked on
> > June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m
> > specifically curious about whether a luceneMatchVersion of 7.1 or lower
> > still works in Solr 9?
> >
> > Thanks for your insights,
> >
> > -Jane
> >
> > --
> > Jane Sandberg (she/her)
> > Library Software Engineer, Discovery and Access Services
> >
>
Re: JSON boolean query syntax with edismax as default QueryParser
Posted by Jane Sandberg <js...@princeton.edu>.
Hi David,
Thanks for looking into this, and for the security fix.
My colleague and I put together a small repository to reproduce the issue. It has a configset, a docker-compose file, and a README with the steps to reproduce it on solr 8.4: https://github.com/pulibrary/edismax-json-queries/
Appreciatively,
-Jane
From: David Smiley <da...@gmail.com>
Date: Monday, July 31, 2023 at 5:58 AM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: JSON boolean query syntax with edismax as default QueryParser
Hi Jane,
That change in 7.2 does look like it's correlated, and I'm the one who
implemented it in the name of security.
Could you walk me through a short series of steps to show the problem with
one of Solr's "example" setups like techproducts? Step one is run it, step
two is you sending the query via curl. Use whatever Solr version you want.
~ David
On Fri, Jul 28, 2023 at 10:33 AM Jane Sandberg <js...@princeton.edu> wrote:
> Hi Solr colleagues,
>
> On Solr 8.4.1, we’ve noticed that the following types of JSON DSL queries
> work if our luceneMatchVersion is 7.1 or lower, or if our default query
> parser is set to lucene:
>
>
> {"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
>
> However, if the query parser is set to edismax and the luceneMatchVersion
> is 7.2 or higher, the parsed query visible with debug=true becomes a
> complete mess, searching for the terms “bool” and “must”, rather than the
> terms we actually want to search for:
>
>
> +(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 |
> (local_subject_unstem_search:bool)^15.0 | (author_unstem_search:bool)^40.0
> […]
>
> Also while debug=true, we noticed that the JSON DSL queries get converted
> into a querystring with local params: ”{!bool must=$_tt1 }”. So I am
> suspecting these two changes in Solr 7.2 as the reason we can’t use Boolean
> JSON queries with edismax and a recent luceneMatchVersion:
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fdocs%2F7_2_0%2Fchanges%2FChanges.html%23v7.2.0.upgrade_notes&data=05%7C01%7Cjs7389%40princeton.edu%7Ce4c749a12c0e49600eaf08db91c5dec0%7C2ff601167431425db5af077d7791bda4%7C0%7C0%7C638264051269788881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QhLXzlXcEnAp86czCzwUUHeio89aiAiAA46ZTkDrWDo%3D&reserved=0<https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes>.
> Does that seem correct?
>
> Also, could this be related to the question Benjamin Armintor asked on
> June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m
> specifically curious about whether a luceneMatchVersion of 7.1 or lower
> still works in Solr 9?
>
> Thanks for your insights,
>
> -Jane
>
> --
> Jane Sandberg (she/her)
> Library Software Engineer, Discovery and Access Services
>
Re: JSON boolean query syntax with edismax as default QueryParser
Posted by David Smiley <da...@gmail.com>.
Hi Jane,
That change in 7.2 does look like it's correlated, and I'm the one who
implemented it in the name of security.
Could you walk me through a short series of steps to show the problem with
one of Solr's "example" setups like techproducts? Step one is run it, step
two is you sending the query via curl. Use whatever Solr version you want.
~ David
On Fri, Jul 28, 2023 at 10:33 AM Jane Sandberg <js...@princeton.edu> wrote:
> Hi Solr colleagues,
>
> On Solr 8.4.1, we’ve noticed that the following types of JSON DSL queries
> work if our luceneMatchVersion is 7.1 or lower, or if our default query
> parser is set to lucene:
>
>
> {"query":{"bool":{"must":[{"lucene":{"query":"plasticity","df":"title_a_index"}}]}}}
>
> However, if the query parser is set to edismax and the luceneMatchVersion
> is 7.2 or higher, the parsed query visible with debug=true becomes a
> complete mess, searching for the terms “bool” and “must”, rather than the
> terms we actually want to search for:
>
>
> +(DisjunctionMaxQuery(((author_main_unstem_search:bool)^1000.0 |
> (local_subject_unstem_search:bool)^15.0 | (author_unstem_search:bool)^40.0
> […]
>
> Also while debug=true, we noticed that the JSON DSL queries get converted
> into a querystring with local params: ”{!bool must=$_tt1 }”. So I am
> suspecting these two changes in Solr 7.2 as the reason we can’t use Boolean
> JSON queries with edismax and a recent luceneMatchVersion:
> https://solr.apache.org/docs/7_2_0/changes/Changes.html#v7.2.0.upgrade_notes.
> Does that seem correct?
>
> Also, could this be related to the question Benjamin Armintor asked on
> June 23 (subject: Changes to JSON query API/syntax in Solr 9.x?)? I’m
> specifically curious about whether a luceneMatchVersion of 7.1 or lower
> still works in Solr 9?
>
> Thanks for your insights,
>
> -Jane
>
> --
> Jane Sandberg (she/her)
> Library Software Engineer, Discovery and Access Services
>