You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Darin Amos <da...@gmail.com> on 2014/12/15 20:38:58 UTC
Is the QueryParser instance thread safe?
Hello,
I have a simple question, is an instance of the QueryParser created for every single request and is that object thread safe?
I will try not to get into too much of the specifics but I have a custom QueryParser and Search Component. Within my query parser when it is instantiated, I execute a lucene search and use that docset to help build and filter my main query. In my search component later on, I might need to build another query with the query parser and use that same filter docset. I would rather just hold onto the docset and reuse it rather than re-execute the query.
Thanks!
Darin
Re: Is the QueryParser instance thread safe?
Posted by Darin Amos <da...@gmail.com>.
No problem,
I have reviewed this content before except the blogs from McCandless,
I will look it over. I think the missing piece here that I did not
describe is that I am working with a SOLR instance that is embedded
within an already existing enterprise application, and is very tightly
coupled with that application (it's poorly designed). It's running
v4.3.0, the parent and child documents exist within the same core and
unfortunately I have no authority to change or restructure the index
as of today. The documents have all the same fields except child docs
have a single value parent ID field that is empty in the parents.
Where it gets tricky is a child document would have a multivalved
field with just one value, color=red, while the parent document has
the same field with ALL the possible values, size=red,blue,green.
This is the field I need to facet on. That is why it is challenging
for me to execute a query as I described. If I search for parents with
red children in stock(join query), I want to see all the other colors
in stock for those children. If I facet on the parent documents, I
will see facets that have no children in stock.
Thank you for continuing to respond to my emails, the last few weeks
has been a good learning experience.
Darin Amos
Sent from my iPad
> On Dec 15, 2014, at 5:35 PM, Mikhail Khludnev <mk...@griddynamics.com> wrote:
>
> Darin,
> I hardly follows your approach. I can propose you to check a few materials,
> just to sync terminology and align to the common problem definition.
> Here is a series of post where colleague of mine defines faceted navigation
> problem for eCommerce site:
> http://blog.griddynamics.com/2011/06/solr-experience-search-parent-child.html
> note, we changed the engine since then, we don't believe into
> term-positions anymore, however http://siren.solutions/siren/overview/
> seems successful with it (I don't really know the details).
> The similar problem definition and a solution from Mr McCandless
> http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html
> Some statements seems like you do dill-sideway faceting - check Mr
> McCandless again
> http://blog.mikemccandless.com/2013/02/drill-sideways-faceting-with-lucene.html
> Note, that machinery he describes is completely different to Solr ones
> http://wiki.apache.org/solr/SimpleFacetParameters#Multi-Select_Faceting_and_LocalParams
> Please, make sure how this particularly is implemented, because it seems
> quite relevant to what you do.
>
> Sorry, can't help you more.
>
>
>> On Tue, Dec 16, 2014 at 1:07 AM, Darin Amos <da...@gmail.com> wrote:
>>
>> Hi Mikhail,
>>
>> Sorry for having a few things unclear, I am creating a custom QParser, not
>> QueryParser, that was a slip up. I believe you have answered my question
>> though, and I think I have found a slightly better way to do the code
>> anyway.
>>
>> *Keep reading if curious. I could use some validation that what I am doing
>> is not completely stupid:
>>
>> I have been working on that query parser I have been sending emails out
>> about for a special join query use case. The custom parser allows you to
>> add filters to the child document set before the join happens. For example:
>>
>> q={!rollup from=parent
>> to=id}type:child&cfq=childField1:SomeValue&cfq=childField2:SomeValue
>>
>> There is a custom extended facet component. I want to facet the child
>> documents, not the final parent documents; except I want to facet ALL
>> children, not just the children that were joined from.
>>
>> The query parser also allows you to add another parameter that will apply
>> to ALL children, both before the join, and for the facets. ex:
>> q={!rollup from=parent
>> to=id}type:child&cfq=childField1:SomeValue&cfq=childField2:SomeValue&gcfq=childField3:someValue
>>
>> This goes back to a use case I emailed about a while ago. I want my client
>> to be able to execute a search like follows:
>> -Give me all shirts that have red items, only if it has red items in stock
>> -Give me the facets for all the in stock items for those products (aka
>> including the yellow, blue, green items)
>>
>> q={!rollup from=parent to=id}+(“shirts”….)&cfq=color:red&gcfq=inventory:[1
>> TO *]
>>
>> Since that global filter is used multiple times. Once in the query parser,
>> and once in the facet component, I just wanted to save it, and make sure I
>> was saving it safely.
>>
>> If anyone is curious, all my code is in a public github repository. (
>> https://github.com/damos/SolrRollupQuery <
>> https://github.com/damos/SolrRollupQuery>)
>>
>> Thanks
>>
>> Darin
>>
>>
>>
>>
>> In a nutshell, my query parser simply accepts a special filter query
>> “sfq=<Some Query I desire>”. The parser executes the query and puts the
>> docset into the request context to be used to filter multiple queries.
>>
>>> On Dec 15, 2014, at 3:45 PM, Mikhail Khludnev <
>> mkhludnev@griddynamics.com> wrote:
>>>
>>> Hello Darin,
>>>
>>> Let me try to answer below,
>>>
>>>> On Mon, Dec 15, 2014 at 10:38 PM, Darin Amos <da...@gmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I have a simple question, is an instance of the QueryParser created for
>>>> every single request and is that object thread safe?
>>>>
>>> QueryParser and QParser are request scope not-thread safe objects. eg
>>> QParser holds reference to SolrQueryRequest, and QueryParser holds parser
>>> field which is QParser.
>>>
>>>
>>>>
>>>> I will try not to get into too much of the specifics but I have a custom
>>>> QueryParser
>>>
>>> I wonder why. We used to extend QParserPlugin and QParser, rather than
>>> QueryParser.
>>>
>>>
>>>> and Search Component. Within my query parser when it is instantiated, I
>>>> execute a lucene search and use that docset to help build and filter my
>>>> main query. In my search component later on, I might need to build
>> another
>>>> query with the query parser and use that same filter docset. I would
>> rather
>>>> just hold onto the docset and reuse it rather than re-execute the query.
>>>>
>>>
>>> Here it's not clear whether you are talking about same request or about
>>> several ones. In case of former, you can pass whatever you need across
>> your
>>> componets,parsers, handlers, responsewriters, etc via
>>> SolrQueryRequest.getContext().
>>> Sharing across requests are more tricky, check how Solr uses filterCache.
>>> The regular extensions uses
>>> https://wiki.apache.org/solr/SolrCaching#User.2FGeneric_Caches to cache
>>> DocSets by Strings or perhaps really heavy queries, which are also valid
>>> across requests.
>>>
>>>
>>>>
>>>> Thanks!
>>>>
>>>> Darin
>>>>
>>>>
>>>
>>> --
>>> Sincerely yours
>>> Mikhail Khludnev
>>> Principal Engineer,
>>> Grid Dynamics
>>>
>>> <http://www.griddynamics.com>
>>> <mk...@griddynamics.com>
>>
>>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
Re: Is the QueryParser instance thread safe?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Darin,
I hardly follows your approach. I can propose you to check a few materials,
just to sync terminology and align to the common problem definition.
Here is a series of post where colleague of mine defines faceted navigation
problem for eCommerce site:
http://blog.griddynamics.com/2011/06/solr-experience-search-parent-child.html
note, we changed the engine since then, we don't believe into
term-positions anymore, however http://siren.solutions/siren/overview/
seems successful with it (I don't really know the details).
The similar problem definition and a solution from Mr McCandless
http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html
Some statements seems like you do dill-sideway faceting - check Mr
McCandless again
http://blog.mikemccandless.com/2013/02/drill-sideways-faceting-with-lucene.html
Note, that machinery he describes is completely different to Solr ones
http://wiki.apache.org/solr/SimpleFacetParameters#Multi-Select_Faceting_and_LocalParams
Please, make sure how this particularly is implemented, because it seems
quite relevant to what you do.
Sorry, can't help you more.
On Tue, Dec 16, 2014 at 1:07 AM, Darin Amos <da...@gmail.com> wrote:
>
> Hi Mikhail,
>
> Sorry for having a few things unclear, I am creating a custom QParser, not
> QueryParser, that was a slip up. I believe you have answered my question
> though, and I think I have found a slightly better way to do the code
> anyway.
>
> *Keep reading if curious. I could use some validation that what I am doing
> is not completely stupid:
>
> I have been working on that query parser I have been sending emails out
> about for a special join query use case. The custom parser allows you to
> add filters to the child document set before the join happens. For example:
>
> q={!rollup from=parent
> to=id}type:child&cfq=childField1:SomeValue&cfq=childField2:SomeValue
>
> There is a custom extended facet component. I want to facet the child
> documents, not the final parent documents; except I want to facet ALL
> children, not just the children that were joined from.
>
> The query parser also allows you to add another parameter that will apply
> to ALL children, both before the join, and for the facets. ex:
> q={!rollup from=parent
> to=id}type:child&cfq=childField1:SomeValue&cfq=childField2:SomeValue&gcfq=childField3:someValue
>
> This goes back to a use case I emailed about a while ago. I want my client
> to be able to execute a search like follows:
> -Give me all shirts that have red items, only if it has red items in stock
> -Give me the facets for all the in stock items for those products (aka
> including the yellow, blue, green items)
>
> q={!rollup from=parent to=id}+(“shirts”….)&cfq=color:red&gcfq=inventory:[1
> TO *]
>
> Since that global filter is used multiple times. Once in the query parser,
> and once in the facet component, I just wanted to save it, and make sure I
> was saving it safely.
>
> If anyone is curious, all my code is in a public github repository. (
> https://github.com/damos/SolrRollupQuery <
> https://github.com/damos/SolrRollupQuery>)
>
> Thanks
>
> Darin
>
>
>
>
> In a nutshell, my query parser simply accepts a special filter query
> “sfq=<Some Query I desire>”. The parser executes the query and puts the
> docset into the request context to be used to filter multiple queries.
>
> > On Dec 15, 2014, at 3:45 PM, Mikhail Khludnev <
> mkhludnev@griddynamics.com> wrote:
> >
> > Hello Darin,
> >
> > Let me try to answer below,
> >
> > On Mon, Dec 15, 2014 at 10:38 PM, Darin Amos <da...@gmail.com> wrote:
> >>
> >> Hello,
> >>
> >> I have a simple question, is an instance of the QueryParser created for
> >> every single request and is that object thread safe?
> >>
> > QueryParser and QParser are request scope not-thread safe objects. eg
> > QParser holds reference to SolrQueryRequest, and QueryParser holds parser
> > field which is QParser.
> >
> >
> >>
> >> I will try not to get into too much of the specifics but I have a custom
> >> QueryParser
> >
> > I wonder why. We used to extend QParserPlugin and QParser, rather than
> > QueryParser.
> >
> >
> >> and Search Component. Within my query parser when it is instantiated, I
> >> execute a lucene search and use that docset to help build and filter my
> >> main query. In my search component later on, I might need to build
> another
> >> query with the query parser and use that same filter docset. I would
> rather
> >> just hold onto the docset and reuse it rather than re-execute the query.
> >>
> >
> > Here it's not clear whether you are talking about same request or about
> > several ones. In case of former, you can pass whatever you need across
> your
> > componets,parsers, handlers, responsewriters, etc via
> > SolrQueryRequest.getContext().
> > Sharing across requests are more tricky, check how Solr uses filterCache.
> > The regular extensions uses
> > https://wiki.apache.org/solr/SolrCaching#User.2FGeneric_Caches to cache
> > DocSets by Strings or perhaps really heavy queries, which are also valid
> > across requests.
> >
> >
> >>
> >> Thanks!
> >>
> >> Darin
> >>
> >>
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> > <mk...@griddynamics.com>
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: Is the QueryParser instance thread safe?
Posted by Darin Amos <da...@gmail.com>.
Hi Mikhail,
Sorry for having a few things unclear, I am creating a custom QParser, not QueryParser, that was a slip up. I believe you have answered my question though, and I think I have found a slightly better way to do the code anyway.
*Keep reading if curious. I could use some validation that what I am doing is not completely stupid:
I have been working on that query parser I have been sending emails out about for a special join query use case. The custom parser allows you to add filters to the child document set before the join happens. For example:
q={!rollup from=parent to=id}type:child&cfq=childField1:SomeValue&cfq=childField2:SomeValue
There is a custom extended facet component. I want to facet the child documents, not the final parent documents; except I want to facet ALL children, not just the children that were joined from.
The query parser also allows you to add another parameter that will apply to ALL children, both before the join, and for the facets. ex:
q={!rollup from=parent to=id}type:child&cfq=childField1:SomeValue&cfq=childField2:SomeValue&gcfq=childField3:someValue
This goes back to a use case I emailed about a while ago. I want my client to be able to execute a search like follows:
-Give me all shirts that have red items, only if it has red items in stock
-Give me the facets for all the in stock items for those products (aka including the yellow, blue, green items)
q={!rollup from=parent to=id}+(“shirts”….)&cfq=color:red&gcfq=inventory:[1 TO *]
Since that global filter is used multiple times. Once in the query parser, and once in the facet component, I just wanted to save it, and make sure I was saving it safely.
If anyone is curious, all my code is in a public github repository. (https://github.com/damos/SolrRollupQuery <https://github.com/damos/SolrRollupQuery>)
Thanks
Darin
In a nutshell, my query parser simply accepts a special filter query “sfq=<Some Query I desire>”. The parser executes the query and puts the docset into the request context to be used to filter multiple queries.
> On Dec 15, 2014, at 3:45 PM, Mikhail Khludnev <mk...@griddynamics.com> wrote:
>
> Hello Darin,
>
> Let me try to answer below,
>
> On Mon, Dec 15, 2014 at 10:38 PM, Darin Amos <da...@gmail.com> wrote:
>>
>> Hello,
>>
>> I have a simple question, is an instance of the QueryParser created for
>> every single request and is that object thread safe?
>>
> QueryParser and QParser are request scope not-thread safe objects. eg
> QParser holds reference to SolrQueryRequest, and QueryParser holds parser
> field which is QParser.
>
>
>>
>> I will try not to get into too much of the specifics but I have a custom
>> QueryParser
>
> I wonder why. We used to extend QParserPlugin and QParser, rather than
> QueryParser.
>
>
>> and Search Component. Within my query parser when it is instantiated, I
>> execute a lucene search and use that docset to help build and filter my
>> main query. In my search component later on, I might need to build another
>> query with the query parser and use that same filter docset. I would rather
>> just hold onto the docset and reuse it rather than re-execute the query.
>>
>
> Here it's not clear whether you are talking about same request or about
> several ones. In case of former, you can pass whatever you need across your
> componets,parsers, handlers, responsewriters, etc via
> SolrQueryRequest.getContext().
> Sharing across requests are more tricky, check how Solr uses filterCache.
> The regular extensions uses
> https://wiki.apache.org/solr/SolrCaching#User.2FGeneric_Caches to cache
> DocSets by Strings or perhaps really heavy queries, which are also valid
> across requests.
>
>
>>
>> Thanks!
>>
>> Darin
>>
>>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
Re: Is the QueryParser instance thread safe?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hello Darin,
Let me try to answer below,
On Mon, Dec 15, 2014 at 10:38 PM, Darin Amos <da...@gmail.com> wrote:
>
> Hello,
>
> I have a simple question, is an instance of the QueryParser created for
> every single request and is that object thread safe?
>
QueryParser and QParser are request scope not-thread safe objects. eg
QParser holds reference to SolrQueryRequest, and QueryParser holds parser
field which is QParser.
>
> I will try not to get into too much of the specifics but I have a custom
> QueryParser
I wonder why. We used to extend QParserPlugin and QParser, rather than
QueryParser.
> and Search Component. Within my query parser when it is instantiated, I
> execute a lucene search and use that docset to help build and filter my
> main query. In my search component later on, I might need to build another
> query with the query parser and use that same filter docset. I would rather
> just hold onto the docset and reuse it rather than re-execute the query.
>
Here it's not clear whether you are talking about same request or about
several ones. In case of former, you can pass whatever you need across your
componets,parsers, handlers, responsewriters, etc via
SolrQueryRequest.getContext().
Sharing across requests are more tricky, check how Solr uses filterCache.
The regular extensions uses
https://wiki.apache.org/solr/SolrCaching#User.2FGeneric_Caches to cache
DocSets by Strings or perhaps really heavy queries, which are also valid
across requests.
>
> Thanks!
>
> Darin
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>