You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jamie Johnson <je...@gmail.com> on 2011/07/09 03:51:47 UTC
Query Rewrite
My organization is considering a few different approaches for indexing
vs query rewrite and I'm trying to figure out what would be required
in order to implement some form of query rewrite. Lets say my index
has 2 fields first name and last name. When the user does a query
name:bob I'd like to transform this into first_name:bob OR
last_name:bob. I know this example is trivial but our real index is
much more complex. Are there any extension points in solr for doing
such a thing? If not are there any utilities which I could use for
doing this? We are currently using the edismax query parser so I
suppose I could extend that to handle this but I'm starting from a
completely blank slate so any guidance would be appreciated.
Re: Query Rewrite
Posted by Chris Hostetter <ho...@fucit.org>.
: then in the CustomQueryParser I iterate over all the arguments adding
: each key/value to a Map. I then pass in this to the constructor of a
: basically copied ExtendedDismaxQParser (only difference is the added
: aliases and the logic to add those to the ExtendedSolrQParser).
:
: Now, the thing I hate about this is I had to pull pieces into my own
: class since some of the methods being called (in QueryUtils for
: instance) are not publicly available.
:
: Also I didn't expose parameters to allow this to be done per query
: level, but I'd most certainly take suggestions on how to do this. My
: use case doesn't require it but I'd be happy to make the modifications
: and try it out.
By all means, if you are interested in contributing what you have
to the project, please create a jira issue and attach it.
I suspect that the best way forward would be to adapt what you've changed
into the ExtendedDisxaxQParser directly, but it would also be useful to
see what pain points you had in using QueryUtils to try and make it easier
for other people to write their own parsers.
https://wiki.apache.org/solr/HowToContribute
"A half-baked patch in Jira, with no documentation, no tests
and no backwards compatibility is better than no patch at all."
-- Yonik's Law of Patches
-Hoss
Re: Query Rewrite
Posted by Jamie Johnson <je...@gmail.com>.
Ok, so I think I have something working relatively well. I have a few
issues which I'm not sure how to address though. Currently when I
create my ParserPlugin in solrconfig I do the following
<queryParser name="alias"
class="org.apache.solr.search.CustomQueryParserPlugin">
<str name="person_name">
person_name^1.0 person_name_first^0.5 person_name_last^0.5
</str>
</queryParser>
then in the CustomQueryParser I iterate over all the arguments adding
each key/value to a Map. I then pass in this to the constructor of a
basically copied ExtendedDismaxQParser (only difference is the added
aliases and the logic to add those to the ExtendedSolrQParser).
Now, the thing I hate about this is I had to pull pieces into my own
class since some of the methods being called (in QueryUtils for
instance) are not publicly available.
Also I didn't expose parameters to allow this to be done per query
level, but I'd most certainly take suggestions on how to do this. My
use case doesn't require it but I'd be happy to make the modifications
and try it out.
On Tue, Jul 12, 2011 at 5:08 PM, Jamie Johnson <je...@gmail.com> wrote:
> I'm not following where the aliasing feature I'm looking for is.
> Looking at the patch I didn't see it either. Essentially what I'm
> looking for is when a user searches for person_name that the query
> turns into person_name:john OR person_name_first:john OR
> person_name_last:john. I don't see anything like that here, am I just
> missing it?
>
> On Tue, Jul 12, 2011 at 3:06 PM, Chris Hostetter
> <ho...@fucit.org> wrote:
>>
>> : Thanks Hoss. I'm not really sure where to begin looking with this, I
>> : quickly read the JIRA but don't see mention of exposing the multiple
>> : aliases. Can you provide any more details?
>>
>> i refered to it as "uf" or "user fields" ... note the specific comment i
>> linked to in the first url, and the subsequent patch
>>
>> the colon bug in edismax is what hung me up at the time.
>>
>> :
>> : On Tue, Jul 12, 2011 at 1:19 PM, Chris Hostetter
>> : <ho...@fucit.org> wrote:
>> : > : Taking a closer look at this it seems as if the
>> : > : DisjunctionMaxQueryParser supports doing multiple aliases and
>> : > : generating multiple queries, I didn't see this same capability in the
>> : > : ExtendedDismaxQParser, am I just missing it? If this capability were
>> : >
>> : > it's never been exposed at a user level ... i started looking at adding it
>> : > to edismax but ran into a bug i couldn't uncover in the time i had to work
>> : > on it (which i think has since been fixed)...
>> : >
>> : > https://issues.apache.org/jira/browse/SOLR-1553?focusedCommentId=12839892&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12839892
>> : > https://issues.apache.org/jira/browse/SOLR-2409
>> : > https://issues.apache.org/jira/browse/SOLR-2368
>> : >
>> : >
>> : > -Hoss
>> : >
>> :
>>
>> -Hoss
>
Re: Query Rewrite
Posted by Jamie Johnson <je...@gmail.com>.
I'm not following where the aliasing feature I'm looking for is.
Looking at the patch I didn't see it either. Essentially what I'm
looking for is when a user searches for person_name that the query
turns into person_name:john OR person_name_first:john OR
person_name_last:john. I don't see anything like that here, am I just
missing it?
On Tue, Jul 12, 2011 at 3:06 PM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : Thanks Hoss. I'm not really sure where to begin looking with this, I
> : quickly read the JIRA but don't see mention of exposing the multiple
> : aliases. Can you provide any more details?
>
> i refered to it as "uf" or "user fields" ... note the specific comment i
> linked to in the first url, and the subsequent patch
>
> the colon bug in edismax is what hung me up at the time.
>
> :
> : On Tue, Jul 12, 2011 at 1:19 PM, Chris Hostetter
> : <ho...@fucit.org> wrote:
> : > : Taking a closer look at this it seems as if the
> : > : DisjunctionMaxQueryParser supports doing multiple aliases and
> : > : generating multiple queries, I didn't see this same capability in the
> : > : ExtendedDismaxQParser, am I just missing it? If this capability were
> : >
> : > it's never been exposed at a user level ... i started looking at adding it
> : > to edismax but ran into a bug i couldn't uncover in the time i had to work
> : > on it (which i think has since been fixed)...
> : >
> : > https://issues.apache.org/jira/browse/SOLR-1553?focusedCommentId=12839892&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12839892
> : > https://issues.apache.org/jira/browse/SOLR-2409
> : > https://issues.apache.org/jira/browse/SOLR-2368
> : >
> : >
> : > -Hoss
> : >
> :
>
> -Hoss
Re: Query Rewrite
Posted by Chris Hostetter <ho...@fucit.org>.
: Thanks Hoss. I'm not really sure where to begin looking with this, I
: quickly read the JIRA but don't see mention of exposing the multiple
: aliases. Can you provide any more details?
i refered to it as "uf" or "user fields" ... note the specific comment i
linked to in the first url, and the subsequent patch
the colon bug in edismax is what hung me up at the time.
:
: On Tue, Jul 12, 2011 at 1:19 PM, Chris Hostetter
: <ho...@fucit.org> wrote:
: > : Taking a closer look at this it seems as if the
: > : DisjunctionMaxQueryParser supports doing multiple aliases and
: > : generating multiple queries, I didn't see this same capability in the
: > : ExtendedDismaxQParser, am I just missing it? If this capability were
: >
: > it's never been exposed at a user level ... i started looking at adding it
: > to edismax but ran into a bug i couldn't uncover in the time i had to work
: > on it (which i think has since been fixed)...
: >
: > https://issues.apache.org/jira/browse/SOLR-1553?focusedCommentId=12839892&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12839892
: > https://issues.apache.org/jira/browse/SOLR-2409
: > https://issues.apache.org/jira/browse/SOLR-2368
: >
: >
: > -Hoss
: >
:
-Hoss
Re: Query Rewrite
Posted by Jamie Johnson <je...@gmail.com>.
Thanks Hoss. I'm not really sure where to begin looking with this, I
quickly read the JIRA but don't see mention of exposing the multiple
aliases. Can you provide any more details?
On Tue, Jul 12, 2011 at 1:19 PM, Chris Hostetter
<ho...@fucit.org> wrote:
> : Taking a closer look at this it seems as if the
> : DisjunctionMaxQueryParser supports doing multiple aliases and
> : generating multiple queries, I didn't see this same capability in the
> : ExtendedDismaxQParser, am I just missing it? If this capability were
>
> it's never been exposed at a user level ... i started looking at adding it
> to edismax but ran into a bug i couldn't uncover in the time i had to work
> on it (which i think has since been fixed)...
>
> https://issues.apache.org/jira/browse/SOLR-1553?focusedCommentId=12839892&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12839892
> https://issues.apache.org/jira/browse/SOLR-2409
> https://issues.apache.org/jira/browse/SOLR-2368
>
>
> -Hoss
>
Re: Query Rewrite
Posted by Chris Hostetter <ho...@fucit.org>.
: Taking a closer look at this it seems as if the
: DisjunctionMaxQueryParser supports doing multiple aliases and
: generating multiple queries, I didn't see this same capability in the
: ExtendedDismaxQParser, am I just missing it? If this capability were
it's never been exposed at a user level ... i started looking at adding it
to edismax but ran into a bug i couldn't uncover in the time i had to work
on it (which i think has since been fixed)...
https://issues.apache.org/jira/browse/SOLR-1553?focusedCommentId=12839892&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12839892
https://issues.apache.org/jira/browse/SOLR-2409
https://issues.apache.org/jira/browse/SOLR-2368
-Hoss
Re: Query Rewrite
Posted by Jamie Johnson <je...@gmail.com>.
Taking a closer look at this it seems as if the
DisjunctionMaxQueryParser supports doing multiple aliases and
generating multiple queries, I didn't see this same capability in the
ExtendedDismaxQParser, am I just missing it? If this capability were
there (and exposed via config options) this would be exactly what I
want. I'll check JIRA for this but if anyone is aware of it please
let me know.
On Sat, Jul 9, 2011 at 2:32 PM, Jamie Johnson <je...@gmail.com> wrote:
> I do use solr over HTTP. I'm not familiar with Nginx or it's URL
> rewrite capabilities. Would this be easier to rewrite a complex query
> using this then a custom query parser?
>
> On Sat, Jul 9, 2011 at 2:07 PM, Markus Jelsma
> <ma...@openindex.io> wrote:
>> If you use Solr over HTTP, then why don't you use URL rewriting? Nginx has
>> very powerful rewrite features and is capable of rewriting parts of the URL,
>> no need to write and compile and deploy code.
>>
>>> My organization is considering a few different approaches for indexing
>>> vs query rewrite and I'm trying to figure out what would be required
>>> in order to implement some form of query rewrite. Lets say my index
>>> has 2 fields first name and last name. When the user does a query
>>> name:bob I'd like to transform this into first_name:bob OR
>>> last_name:bob. I know this example is trivial but our real index is
>>> much more complex. Are there any extension points in solr for doing
>>> such a thing? If not are there any utilities which I could use for
>>> doing this? We are currently using the edismax query parser so I
>>> suppose I could extend that to handle this but I'm starting from a
>>> completely blank slate so any guidance would be appreciated.
>>
>
Re: Query Rewrite
Posted by Jamie Johnson <je...@gmail.com>.
I do use solr over HTTP. I'm not familiar with Nginx or it's URL
rewrite capabilities. Would this be easier to rewrite a complex query
using this then a custom query parser?
On Sat, Jul 9, 2011 at 2:07 PM, Markus Jelsma
<ma...@openindex.io> wrote:
> If you use Solr over HTTP, then why don't you use URL rewriting? Nginx has
> very powerful rewrite features and is capable of rewriting parts of the URL,
> no need to write and compile and deploy code.
>
>> My organization is considering a few different approaches for indexing
>> vs query rewrite and I'm trying to figure out what would be required
>> in order to implement some form of query rewrite. Lets say my index
>> has 2 fields first name and last name. When the user does a query
>> name:bob I'd like to transform this into first_name:bob OR
>> last_name:bob. I know this example is trivial but our real index is
>> much more complex. Are there any extension points in solr for doing
>> such a thing? If not are there any utilities which I could use for
>> doing this? We are currently using the edismax query parser so I
>> suppose I could extend that to handle this but I'm starting from a
>> completely blank slate so any guidance would be appreciated.
>
Re: Query Rewrite
Posted by Markus Jelsma <ma...@openindex.io>.
If you use Solr over HTTP, then why don't you use URL rewriting? Nginx has
very powerful rewrite features and is capable of rewriting parts of the URL,
no need to write and compile and deploy code.
> My organization is considering a few different approaches for indexing
> vs query rewrite and I'm trying to figure out what would be required
> in order to implement some form of query rewrite. Lets say my index
> has 2 fields first name and last name. When the user does a query
> name:bob I'd like to transform this into first_name:bob OR
> last_name:bob. I know this example is trivial but our real index is
> much more complex. Are there any extension points in solr for doing
> such a thing? If not are there any utilities which I could use for
> doing this? We are currently using the edismax query parser so I
> suppose I could extend that to handle this but I'm starting from a
> completely blank slate so any guidance would be appreciated.
Re: Query Rewrite
Posted by Jamie Johnson <je...@gmail.com>.
Also, I'm not sure if it matters or not but I'm using the edismax
query parser now, and don't wan to lose that functionality.
Really what I'd like is a way to configure the edismax parser so that
it would work on more than just the default field. So for instance if
the user specified first_name:bob I could do something with weights to
modify the query. Does that make sense?
On Sat, Jul 9, 2011 at 2:02 PM, Jamie Johnson <je...@gmail.com> wrote:
> If I did that how would I configure solr to use the modified query parser?
>
> On Sat, Jul 9, 2011 at 11:39 AM, Dmitry Kan <dm...@gmail.com> wrote:
>> you can try extending LuceneQParser. In its createParser method
>> (lucene 2.9.3 and solr 1.4) you can analyze the input query in the
>> param q and modify it accordingly.
>>
>> On 7/9/11, Jamie Johnson <je...@gmail.com> wrote:
>>> My organization is considering a few different approaches for indexing
>>> vs query rewrite and I'm trying to figure out what would be required
>>> in order to implement some form of query rewrite. Lets say my index
>>> has 2 fields first name and last name. When the user does a query
>>> name:bob I'd like to transform this into first_name:bob OR
>>> last_name:bob. I know this example is trivial but our real index is
>>> much more complex. Are there any extension points in solr for doing
>>> such a thing? If not are there any utilities which I could use for
>>> doing this? We are currently using the edismax query parser so I
>>> suppose I could extend that to handle this but I'm starting from a
>>> completely blank slate so any guidance would be appreciated.
>>>
>>
>>
>> --
>> Regards,
>>
>> Dmitry Kan
>>
>
Re: Query Rewrite
Posted by Jamie Johnson <je...@gmail.com>.
If I did that how would I configure solr to use the modified query parser?
On Sat, Jul 9, 2011 at 11:39 AM, Dmitry Kan <dm...@gmail.com> wrote:
> you can try extending LuceneQParser. In its createParser method
> (lucene 2.9.3 and solr 1.4) you can analyze the input query in the
> param q and modify it accordingly.
>
> On 7/9/11, Jamie Johnson <je...@gmail.com> wrote:
>> My organization is considering a few different approaches for indexing
>> vs query rewrite and I'm trying to figure out what would be required
>> in order to implement some form of query rewrite. Lets say my index
>> has 2 fields first name and last name. When the user does a query
>> name:bob I'd like to transform this into first_name:bob OR
>> last_name:bob. I know this example is trivial but our real index is
>> much more complex. Are there any extension points in solr for doing
>> such a thing? If not are there any utilities which I could use for
>> doing this? We are currently using the edismax query parser so I
>> suppose I could extend that to handle this but I'm starting from a
>> completely blank slate so any guidance would be appreciated.
>>
>
>
> --
> Regards,
>
> Dmitry Kan
>
Re: Query Rewrite
Posted by Jamie Johnson <je...@gmail.com>.
Thanks Dmitry, I think I've got this now.
On Sun, Jul 10, 2011 at 3:55 AM, Dmitry Kan <dm...@gmail.com> wrote:
> was typying this on-the-go from my phone, I meant LuceneQParserPlugin of
> course.
>
> On Sat, Jul 9, 2011 at 6:39 PM, Dmitry Kan <dm...@gmail.com> wrote:
>
>> you can try extending LuceneQParser. In its createParser method
>> (lucene 2.9.3 and solr 1.4) you can analyze the input query in the
>> param q and modify it accordingly.
>>
>> On 7/9/11, Jamie Johnson <je...@gmail.com> wrote:
>> > My organization is considering a few different approaches for indexing
>> > vs query rewrite and I'm trying to figure out what would be required
>> > in order to implement some form of query rewrite. Lets say my index
>> > has 2 fields first name and last name. When the user does a query
>> > name:bob I'd like to transform this into first_name:bob OR
>> > last_name:bob. I know this example is trivial but our real index is
>> > much more complex. Are there any extension points in solr for doing
>> > such a thing? If not are there any utilities which I could use for
>> > doing this? We are currently using the edismax query parser so I
>> > suppose I could extend that to handle this but I'm starting from a
>> > completely blank slate so any guidance would be appreciated.
>> >
>>
>>
>> --
>> Regards,
>>
>> Dmitry Kan
>>
>
>
>
> --
> Regards,
>
> Dmitry Kan
>
Re: Query Rewrite
Posted by Dmitry Kan <dm...@gmail.com>.
was typying this on-the-go from my phone, I meant LuceneQParserPlugin of
course.
On Sat, Jul 9, 2011 at 6:39 PM, Dmitry Kan <dm...@gmail.com> wrote:
> you can try extending LuceneQParser. In its createParser method
> (lucene 2.9.3 and solr 1.4) you can analyze the input query in the
> param q and modify it accordingly.
>
> On 7/9/11, Jamie Johnson <je...@gmail.com> wrote:
> > My organization is considering a few different approaches for indexing
> > vs query rewrite and I'm trying to figure out what would be required
> > in order to implement some form of query rewrite. Lets say my index
> > has 2 fields first name and last name. When the user does a query
> > name:bob I'd like to transform this into first_name:bob OR
> > last_name:bob. I know this example is trivial but our real index is
> > much more complex. Are there any extension points in solr for doing
> > such a thing? If not are there any utilities which I could use for
> > doing this? We are currently using the edismax query parser so I
> > suppose I could extend that to handle this but I'm starting from a
> > completely blank slate so any guidance would be appreciated.
> >
>
>
> --
> Regards,
>
> Dmitry Kan
>
--
Regards,
Dmitry Kan
Re: Query Rewrite
Posted by Dmitry Kan <dm...@gmail.com>.
you can try extending LuceneQParser. In its createParser method
(lucene 2.9.3 and solr 1.4) you can analyze the input query in the
param q and modify it accordingly.
On 7/9/11, Jamie Johnson <je...@gmail.com> wrote:
> My organization is considering a few different approaches for indexing
> vs query rewrite and I'm trying to figure out what would be required
> in order to implement some form of query rewrite. Lets say my index
> has 2 fields first name and last name. When the user does a query
> name:bob I'd like to transform this into first_name:bob OR
> last_name:bob. I know this example is trivial but our real index is
> much more complex. Are there any extension points in solr for doing
> such a thing? If not are there any utilities which I could use for
> doing this? We are currently using the edismax query parser so I
> suppose I could extend that to handle this but I'm starting from a
> completely blank slate so any guidance would be appreciated.
>
--
Regards,
Dmitry Kan