You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Yangrui Guo <gu...@gmail.com> on 2015/11/01 04:52:46 UTC
Kate Winslet vs Winslet Kate
Hi today I found an interesting aspect of solr. I imported IMDB data into
solr. The IMDB puts last name before first name for its person's name field
eg. "Winslet, Kate". When I search "Winslet Kate" with quotation marks I
could get the exact result. However if I search "Kate Winslet" or Kate AND
Winslet solr seem to return me all result containing either Kate or Winslet
which is similar to "Winslet Kate"~999999. From user perspective I
certainly want solr to treat Kate Winslet the same as Winslet Kate. Is
there anyway to make solr score higher for terms in the same field?
Yangrui
Re: Kate Winslet vs Winslet Kate
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I just had a thought that perhaps Complex Phrase parser could be useful here:
https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-ComplexPhraseQueryParser
You still need to mark that full name to search against specific
field, so it may or may not in a more general stream of user-provided
words.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 31 October 2015 at 23:52, Yangrui Guo <gu...@gmail.com> wrote:
> Hi today I found an interesting aspect of solr. I imported IMDB data into
> solr. The IMDB puts last name before first name for its person's name field
> eg. "Winslet, Kate". When I search "Winslet Kate" with quotation marks I
> could get the exact result. However if I search "Kate Winslet" or Kate AND
> Winslet solr seem to return me all result containing either Kate or Winslet
> which is similar to "Winslet Kate"~999999. From user perspective I
> certainly want solr to treat Kate Winslet the same as Winslet Kate. Is
> there anyway to make solr score higher for terms in the same field?
>
> Yangrui
Re: Kate Winslet vs Winslet Kate
Posted by Yangrui Guo <gu...@gmail.com>.
I've just read the post and it has addressed much of my issue. It is hard
to detect phrases and disambiguate phrases but some existing approaches
seem really promising.
On Sunday, November 1, 2015, Paul Libbrecht <pa...@hoplahup.net> wrote:
> Alexandre,
>
> I guess you are talking about that post:
>
>
> http://lucidworks.com/blog/2015/06/06/query-autofiltering-extended-language-logic-search/
>
> I think it is very often impossible to solve properly.
>
> Words such as "direction" have very many meanings and would come in
> different fields.
> In IMDB, words such as the names of persons would come in at least
> different roles; similarly, the actors' role's name is likely to match
> the family name of persons...
>
> Paul
>
>
>
> > As others indicated having intelligence to recognize the terms (e.g.
> > Kate should be in name) or some user indication to do so can make thing
> > more precise but is rarely done.
> > Alexandre Rafalovitch <mailto:arafalov@gmail.com <javascript:;>>
> > 1 novembre 2015 13:07
> > Which is what I believe Ted Sullivan is working on and presented at
> > the latest Lucene/Solr Revolution. His presentation does not seem to
> > be up, but he was writing about it on:
> > http://lucidworks.com/blog/author/tedsullivan/
>
> > Erick Erickson <mailto:erickerickson@gmail.com <javascript:;>>
> > 1 novembre 2015 07:40
> > Yeah, that's actually a tough one. You have no control over what the
> > user types,
> > you have to try to guess what they meant.
>
>
Re: Kate Winslet vs Winslet Kate
Posted by Yangrui Guo <gu...@gmail.com>.
Tried but still didn't get correct result. I guess the reason is because I
use block join with the document. My current solution is to use a name
tagged to extract persons then put name field restriction before it. This
will not work with all situations though. Thanks for the reply.
On Tuesday, November 3, 2015, Imtiaz Shakil Siddique <
shakilsust006@gmail.com> wrote:
> I think edismax query parser perfectly fits for your needs.
> You can make edismax search for query words on multiple fields using the
> "qf" parameter and you can also set the priority of those searched fields.
>
> Edismax also auto generates phrase query for specified fields . ( look into
> the "pf" and "pf2" parameter )
>
> Here is an example -->
> qf: name^3.0 text^1.0
> pf: name
> ps:2
> pf2:name text^0.5
>
> Now when you search for "Kate Winslet" you'll get docs matching the query
> word from name field.
> If you search for "Kate Winslet Movie" then "pf" will pick up "Kate
> Winslet" from name field and the "movie" will be picked up from text field.
> You can experiment the query time boosting to fine tune your needs.
>
> Regards
> Imtiaz Shakil Siddique
> On Nov 3, 2015 9:36 PM, "scott chu" <scott.chu@udngroup.com <javascript:;>>
> wrote:
>
> > solr-user,妳好
> >
> > With repsect to querying, Dismax makes solr query syntax quite like
> > Google's, you type simple keywords, you can boost them, you can use +/-
> > just like Google's. Meaning they give users a lot of covenince and less
> > boolean knowlege to establish intended query string. Normal Lucene search
> > syntax are treated as escaped characters except AND & OR. You can say
> > Dismax gives some room for phrase querying. eDismax improve something
> > of Dismax but it depends on if you need thorse improvement or not. You
> can
> > see it here:
> >
> https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser
> >
> >
> >
> > ----- Original Message -----
> > *From: *Yangrui Guo <guoyangrui@gmail.com <javascript:;>>
> > *To: *solr-user@lucene.apache.org <javascript:;>
> > *Date: *2015-11-01, 23:58:27
> > *Subject: *Re: Kate Winslet vs Winslet Kate
> >
> > Could you tell me more about the edismax approach? I'm new to it. Thanks
> a
> >
> > lot
> >
> > On Sunday, November 1, 2015, Erick Erickson <erickerickson@gmail.com
> <javascript:;>
> > <+erickerickson@gmail.com <javascript:;>>> wrote:
> >
> > > If your goal is to have docs with "kate" and "winslet"
> > > in the _name_ field be scored higher, just make that
> > > explicit as
> > > name:(kate AND winslet)
> > > perhaps boosting as
> > > name:(kate AND winslet)^10
> > > or add it as a clause
> > > q=kate AND winslet OR name:(kate AND winslet)^10
> > > or even
> > > q=kate AND winslet OR name:(kate AND winslet)^10 OR name:"kate
> > winslet"^20
> > >
> > >
> > > Or use edismax to do this kind of thing for you, that's
> > > its purpose.
> > >
> > > Best,
> > > Erick
> > >
> > > On Sun, Nov 1, 2015 at 7:06 AM, Yangrui Guo <guoyangrui@gmail.com
> <javascript:;>
> > <+guoyangrui@gmail.com <javascript:;>>
> > > <javascript:;>> wrote:
> > > > I debugged the query and found the query has been translated into
> > > > _text_:Kate AND _text_:Winslet, which _text_ is the default search
> > field.
> > > > Because my documents use parent/child relation it appeared that if
> > > there's
> > > > no exact match of Kate Winslet, solr will return all documents
> contains
> > > > "Kate" and "Winslet" in anywhere. However it will more sense if solr
> > can
> > > > rank docs that have "Kate" and "Winslet" in the same field higher. Of
> > > > course I can use some NLP tricks with named entity recognition but it
> > > would
> > > > be more expensive to develop.
> > > >
> > > > On Sunday, November 1, 2015, Paul Libbrecht <paul@hoplahup.net
> <javascript:;>
> > <+paul@hoplahup.net <javascript:;>>
> > > <javascript:;>> wrote:
> > > >
> > > >> Alexandre,
> > > >>
> > > >> I guess you are talking about that post:
> > > >>
> > > >>
> > > >>
> > >
> >
> http://lucidworks.com/blog/2015/06/06/query-autofiltering-extended-language-logic-search/
> > > >>
> > > >> I think it is very often impossible to solve properly.
> > > >>
> > > >> Words such as "direction" have very many meanings and would come in
> > > >> different fields.
> > > >> In IMDB, words such as the names of persons would come in at least
> > > >> different roles; similarly, the actors' role's name is likely to
> match
> > > >> the family name of persons...
> > > >>
> > > >> Paul
> > > >>
> > > >>
> > > >>
> > > >> > As others indicated having intelligence to recognize the terms
> (e.g.
> > > >> > Kate should be in name) or some user indication to do so can make
> > > thing
> > > >> > more precise but is rarely done.
> > > >> > Alexandre Rafalovitch <mailto:arafalov@gmail.com <javascript:;>
> > <+arafalov@gmail.com <javascript:;>> <javascript:;>
> > > <javascript:;>>
> > > >> > 1 novembre 2015 13:07
> > > >> > Which is what I believe Ted Sullivan is working on and presented
> at
> >
> > > >> > the latest Lucene/Solr Revolution. His presentation does not seem
> to
> > > >> > be up, but he was writing about it on:
> > > >> > http://lucidworks.com/blog/author/tedsullivan/
> > > >>
> > > >> > Erick Erickson <mailto:erickerickson@gmail.com <javascript:;>
> > <+erickerickson@gmail.com <javascript:;>> <javascript:;>
> > > <javascript:;>>
> > > >> > 1 novembre 2015 07:40
> > > >> > Yeah, that's actually a tough one. You have no control over what
> the
> > > >> > user types,
> > > >> > you have to try to guess what they meant.
> > > >>
> > > >>
> > >
> >
> >
> >
> > -----
> > 未在此訊息中找到病毒。
> > 已透過 AVG 檢查 - www.avg.com
> > 版本: 2015.0.6173 / 病毒庫: 4455/10925 - 發佈日期: 10/31/15
> >
> >
> >
>
Re: Kate Winslet vs Winslet Kate
Posted by Imtiaz Shakil Siddique <sh...@gmail.com>.
I think edismax query parser perfectly fits for your needs.
You can make edismax search for query words on multiple fields using the
"qf" parameter and you can also set the priority of those searched fields.
Edismax also auto generates phrase query for specified fields . ( look into
the "pf" and "pf2" parameter )
Here is an example -->
qf: name^3.0 text^1.0
pf: name
ps:2
pf2:name text^0.5
Now when you search for "Kate Winslet" you'll get docs matching the query
word from name field.
If you search for "Kate Winslet Movie" then "pf" will pick up "Kate
Winslet" from name field and the "movie" will be picked up from text field.
You can experiment the query time boosting to fine tune your needs.
Regards
Imtiaz Shakil Siddique
On Nov 3, 2015 9:36 PM, "scott chu" <sc...@udngroup.com> wrote:
> solr-user,妳好
>
> With repsect to querying, Dismax makes solr query syntax quite like
> Google's, you type simple keywords, you can boost them, you can use +/-
> just like Google's. Meaning they give users a lot of covenince and less
> boolean knowlege to establish intended query string. Normal Lucene search
> syntax are treated as escaped characters except AND & OR. You can say
> Dismax gives some room for phrase querying. eDismax improve something
> of Dismax but it depends on if you need thorse improvement or not. You can
> see it here:
> https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser
>
>
>
> ----- Original Message -----
> *From: *Yangrui Guo <gu...@gmail.com>
> *To: *solr-user@lucene.apache.org
> *Date: *2015-11-01, 23:58:27
> *Subject: *Re: Kate Winslet vs Winslet Kate
>
> Could you tell me more about the edismax approach? I'm new to it. Thanks a
>
> lot
>
> On Sunday, November 1, 2015, Erick Erickson <erickerickson@gmail.com
> <+e...@gmail.com>> wrote:
>
> > If your goal is to have docs with "kate" and "winslet"
> > in the _name_ field be scored higher, just make that
> > explicit as
> > name:(kate AND winslet)
> > perhaps boosting as
> > name:(kate AND winslet)^10
> > or add it as a clause
> > q=kate AND winslet OR name:(kate AND winslet)^10
> > or even
> > q=kate AND winslet OR name:(kate AND winslet)^10 OR name:"kate
> winslet"^20
> >
> >
> > Or use edismax to do this kind of thing for you, that's
> > its purpose.
> >
> > Best,
> > Erick
> >
> > On Sun, Nov 1, 2015 at 7:06 AM, Yangrui Guo <guoyangrui@gmail.com
> <+g...@gmail.com>
> > <javascript:;>> wrote:
> > > I debugged the query and found the query has been translated into
> > > _text_:Kate AND _text_:Winslet, which _text_ is the default search
> field.
> > > Because my documents use parent/child relation it appeared that if
> > there's
> > > no exact match of Kate Winslet, solr will return all documents contains
> > > "Kate" and "Winslet" in anywhere. However it will more sense if solr
> can
> > > rank docs that have "Kate" and "Winslet" in the same field higher. Of
> > > course I can use some NLP tricks with named entity recognition but it
> > would
> > > be more expensive to develop.
> > >
> > > On Sunday, November 1, 2015, Paul Libbrecht <paul@hoplahup.net
> <+p...@hoplahup.net>
> > <javascript:;>> wrote:
> > >
> > >> Alexandre,
> > >>
> > >> I guess you are talking about that post:
> > >>
> > >>
> > >>
> >
> http://lucidworks.com/blog/2015/06/06/query-autofiltering-extended-language-logic-search/
> > >>
> > >> I think it is very often impossible to solve properly.
> > >>
> > >> Words such as "direction" have very many meanings and would come in
> > >> different fields.
> > >> In IMDB, words such as the names of persons would come in at least
> > >> different roles; similarly, the actors' role's name is likely to match
> > >> the family name of persons...
> > >>
> > >> Paul
> > >>
> > >>
> > >>
> > >> > As others indicated having intelligence to recognize the terms (e.g.
> > >> > Kate should be in name) or some user indication to do so can make
> > thing
> > >> > more precise but is rarely done.
> > >> > Alexandre Rafalovitch <mailto:arafalov@gmail.com
> <+a...@gmail.com> <javascript:;>
> > <javascript:;>>
> > >> > 1 novembre 2015 13:07
> > >> > Which is what I believe Ted Sullivan is working on and presented at
>
> > >> > the latest Lucene/Solr Revolution. His presentation does not seem to
> > >> > be up, but he was writing about it on:
> > >> > http://lucidworks.com/blog/author/tedsullivan/
> > >>
> > >> > Erick Erickson <mailto:erickerickson@gmail.com
> <+e...@gmail.com> <javascript:;>
> > <javascript:;>>
> > >> > 1 novembre 2015 07:40
> > >> > Yeah, that's actually a tough one. You have no control over what the
> > >> > user types,
> > >> > you have to try to guess what they meant.
> > >>
> > >>
> >
>
>
>
> -----
> 未在此訊息中找到病毒。
> 已透過 AVG 檢查 - www.avg.com
> 版本: 2015.0.6173 / 病毒庫: 4455/10925 - 發佈日期: 10/31/15
>
>
>
Re: Kate Winslet vs Winslet Kate
Posted by scott chu <sc...@udngroup.com>.
solr-user,妳好
With repsect to querying, Dismax makes solr query syntax quite like Google's, you type simple keywords, you can boost them, you can use +/- just like Google's. Meaning they give users a lot of covenince and less boolean knowlege to establish intended query string. Normal Lucene search syntax are treated as escaped characters except AND & OR. You can say Dismax gives some room for phrase querying. eDismax improve something of Dismax but it depends on if you need thorse improvement or not. You can see it here: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser
----- Original Message -----
From: Yangrui Guo
To: solr-user@lucene.apache.org
Date: 2015-11-01, 23:58:27
Subject: Re: Kate Winslet vs Winslet Kate
Could you tell me more about the edismax approach? I'm new to it. Thanks a
lot
On Sunday, November 1, 2015, Erick Erickson <er...@gmail.com> wrote:
> If your goal is to have docs with "kate" and "winslet"
> in the _name_ field be scored higher, just make that
> explicit as
> name:(kate AND winslet)
> perhaps boosting as
> name:(kate AND winslet)^10
> or add it as a clause
> q=kate AND winslet OR name:(kate AND winslet)^10
> or even
> q=kate AND winslet OR name:(kate AND winslet)^10 OR name:"kate winslet"^20
>
>
> Or use edismax to do this kind of thing for you, that's
> its purpose.
>
> Best,
> Erick
>
> On Sun, Nov 1, 2015 at 7:06 AM, Yangrui Guo <guoyangrui@gmail.com
> <javascript:;>> wrote:
> > I debugged the query and found the query has been translated into
> > _text_:Kate AND _text_:Winslet, which _text_ is the default search field.
> > Because my documents use parent/child relation it appeared that if
> there's
> > no exact match of Kate Winslet, solr will return all documents contains
> > "Kate" and "Winslet" in anywhere. However it will more sense if solr can
> > rank docs that have "Kate" and "Winslet" in the same field higher. Of
> > course I can use some NLP tricks with named entity recognition but it
> would
> > be more expensive to develop.
> >
> > On Sunday, November 1, 2015, Paul Libbrecht <paul@hoplahup.net
> <javascript:;>> wrote:
> >
> >> Alexandre,
> >>
> >> I guess you are talking about that post:
> >>
> >>
> >>
> http://lucidworks.com/blog/2015/06/06/query-autofiltering-extended-language-logic-search/
> >>
> >> I think it is very often impossible to solve properly.
> >>
> >> Words such as "direction" have very many meanings and would come in
> >> different fields.
> >> In IMDB, words such as the names of persons would come in at least
> >> different roles; similarly, the actors' role's name is likely to match
> >> the family name of persons...
> >>
> >> Paul
> >>
> >>
> >>
> >> > As others indicated having intelligence to recognize the terms (e.g.
> >> > Kate should be in name) or some user indication to do so can make
> thing
> >> > more precise but is rarely done.
> >> > Alexandre Rafalovitch <mailto:arafalov@gmail.com <javascript:;>
> <javascript:;>>
> >> > 1 novembre 2015 13:07
> >> > Which is what I believe Ted Sullivan is working on and presented at
> >> > the latest Lucene/Solr Revolution. His presentation does not seem to
> >> > be up, but he was writing about it on:
> >> > http://lucidworks.com/blog/author/tedsullivan/
> >>
> >> > Erick Erickson <mailto:erickerickson@gmail.com <javascript:;>
> <javascript:;>>
> >> > 1 novembre 2015 07:40
> >> > Yeah, that's actually a tough one. You have no control over what the
> >> > user types,
> >> > you have to try to guess what they meant.
> >>
> >>
>
-----
未在此訊息中找到病毒。
已透過 AVG 檢查 - www.avg.com
版本: 2015.0.6173 / 病毒庫: 4455/10925 - 發佈日期: 10/31/15
Re: Kate Winslet vs Winslet Kate
Posted by Yangrui Guo <gu...@gmail.com>.
Could you tell me more about the edismax approach? I'm new to it. Thanks a
lot
On Sunday, November 1, 2015, Erick Erickson <er...@gmail.com> wrote:
> If your goal is to have docs with "kate" and "winslet"
> in the _name_ field be scored higher, just make that
> explicit as
> name:(kate AND winslet)
> perhaps boosting as
> name:(kate AND winslet)^10
> or add it as a clause
> q=kate AND winslet OR name:(kate AND winslet)^10
> or even
> q=kate AND winslet OR name:(kate AND winslet)^10 OR name:"kate winslet"^20
>
>
> Or use edismax to do this kind of thing for you, that's
> its purpose.
>
> Best,
> Erick
>
> On Sun, Nov 1, 2015 at 7:06 AM, Yangrui Guo <guoyangrui@gmail.com
> <javascript:;>> wrote:
> > I debugged the query and found the query has been translated into
> > _text_:Kate AND _text_:Winslet, which _text_ is the default search field.
> > Because my documents use parent/child relation it appeared that if
> there's
> > no exact match of Kate Winslet, solr will return all documents contains
> > "Kate" and "Winslet" in anywhere. However it will more sense if solr can
> > rank docs that have "Kate" and "Winslet" in the same field higher. Of
> > course I can use some NLP tricks with named entity recognition but it
> would
> > be more expensive to develop.
> >
> > On Sunday, November 1, 2015, Paul Libbrecht <paul@hoplahup.net
> <javascript:;>> wrote:
> >
> >> Alexandre,
> >>
> >> I guess you are talking about that post:
> >>
> >>
> >>
> http://lucidworks.com/blog/2015/06/06/query-autofiltering-extended-language-logic-search/
> >>
> >> I think it is very often impossible to solve properly.
> >>
> >> Words such as "direction" have very many meanings and would come in
> >> different fields.
> >> In IMDB, words such as the names of persons would come in at least
> >> different roles; similarly, the actors' role's name is likely to match
> >> the family name of persons...
> >>
> >> Paul
> >>
> >>
> >>
> >> > As others indicated having intelligence to recognize the terms (e.g.
> >> > Kate should be in name) or some user indication to do so can make
> thing
> >> > more precise but is rarely done.
> >> > Alexandre Rafalovitch <mailto:arafalov@gmail.com <javascript:;>
> <javascript:;>>
> >> > 1 novembre 2015 13:07
> >> > Which is what I believe Ted Sullivan is working on and presented at
> >> > the latest Lucene/Solr Revolution. His presentation does not seem to
> >> > be up, but he was writing about it on:
> >> > http://lucidworks.com/blog/author/tedsullivan/
> >>
> >> > Erick Erickson <mailto:erickerickson@gmail.com <javascript:;>
> <javascript:;>>
> >> > 1 novembre 2015 07:40
> >> > Yeah, that's actually a tough one. You have no control over what the
> >> > user types,
> >> > you have to try to guess what they meant.
> >>
> >>
>
Re: Kate Winslet vs Winslet Kate
Posted by Erick Erickson <er...@gmail.com>.
If your goal is to have docs with "kate" and "winslet"
in the _name_ field be scored higher, just make that
explicit as
name:(kate AND winslet)
perhaps boosting as
name:(kate AND winslet)^10
or add it as a clause
q=kate AND winslet OR name:(kate AND winslet)^10
or even
q=kate AND winslet OR name:(kate AND winslet)^10 OR name:"kate winslet"^20
Or use edismax to do this kind of thing for you, that's
its purpose.
Best,
Erick
On Sun, Nov 1, 2015 at 7:06 AM, Yangrui Guo <gu...@gmail.com> wrote:
> I debugged the query and found the query has been translated into
> _text_:Kate AND _text_:Winslet, which _text_ is the default search field.
> Because my documents use parent/child relation it appeared that if there's
> no exact match of Kate Winslet, solr will return all documents contains
> "Kate" and "Winslet" in anywhere. However it will more sense if solr can
> rank docs that have "Kate" and "Winslet" in the same field higher. Of
> course I can use some NLP tricks with named entity recognition but it would
> be more expensive to develop.
>
> On Sunday, November 1, 2015, Paul Libbrecht <pa...@hoplahup.net> wrote:
>
>> Alexandre,
>>
>> I guess you are talking about that post:
>>
>>
>> http://lucidworks.com/blog/2015/06/06/query-autofiltering-extended-language-logic-search/
>>
>> I think it is very often impossible to solve properly.
>>
>> Words such as "direction" have very many meanings and would come in
>> different fields.
>> In IMDB, words such as the names of persons would come in at least
>> different roles; similarly, the actors' role's name is likely to match
>> the family name of persons...
>>
>> Paul
>>
>>
>>
>> > As others indicated having intelligence to recognize the terms (e.g.
>> > Kate should be in name) or some user indication to do so can make thing
>> > more precise but is rarely done.
>> > Alexandre Rafalovitch <mailto:arafalov@gmail.com <javascript:;>>
>> > 1 novembre 2015 13:07
>> > Which is what I believe Ted Sullivan is working on and presented at
>> > the latest Lucene/Solr Revolution. His presentation does not seem to
>> > be up, but he was writing about it on:
>> > http://lucidworks.com/blog/author/tedsullivan/
>>
>> > Erick Erickson <mailto:erickerickson@gmail.com <javascript:;>>
>> > 1 novembre 2015 07:40
>> > Yeah, that's actually a tough one. You have no control over what the
>> > user types,
>> > you have to try to guess what they meant.
>>
>>
Re: Kate Winslet vs Winslet Kate
Posted by Yangrui Guo <gu...@gmail.com>.
I debugged the query and found the query has been translated into
_text_:Kate AND _text_:Winslet, which _text_ is the default search field.
Because my documents use parent/child relation it appeared that if there's
no exact match of Kate Winslet, solr will return all documents contains
"Kate" and "Winslet" in anywhere. However it will more sense if solr can
rank docs that have "Kate" and "Winslet" in the same field higher. Of
course I can use some NLP tricks with named entity recognition but it would
be more expensive to develop.
On Sunday, November 1, 2015, Paul Libbrecht <pa...@hoplahup.net> wrote:
> Alexandre,
>
> I guess you are talking about that post:
>
>
> http://lucidworks.com/blog/2015/06/06/query-autofiltering-extended-language-logic-search/
>
> I think it is very often impossible to solve properly.
>
> Words such as "direction" have very many meanings and would come in
> different fields.
> In IMDB, words such as the names of persons would come in at least
> different roles; similarly, the actors' role's name is likely to match
> the family name of persons...
>
> Paul
>
>
>
> > As others indicated having intelligence to recognize the terms (e.g.
> > Kate should be in name) or some user indication to do so can make thing
> > more precise but is rarely done.
> > Alexandre Rafalovitch <mailto:arafalov@gmail.com <javascript:;>>
> > 1 novembre 2015 13:07
> > Which is what I believe Ted Sullivan is working on and presented at
> > the latest Lucene/Solr Revolution. His presentation does not seem to
> > be up, but he was writing about it on:
> > http://lucidworks.com/blog/author/tedsullivan/
>
> > Erick Erickson <mailto:erickerickson@gmail.com <javascript:;>>
> > 1 novembre 2015 07:40
> > Yeah, that's actually a tough one. You have no control over what the
> > user types,
> > you have to try to guess what they meant.
>
>
Re: Kate Winslet vs Winslet Kate
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Alexandre,
I guess you are talking about that post:
http://lucidworks.com/blog/2015/06/06/query-autofiltering-extended-language-logic-search/
I think it is very often impossible to solve properly.
Words such as "direction" have very many meanings and would come in
different fields.
In IMDB, words such as the names of persons would come in at least
different roles; similarly, the actors' role's name is likely to match
the family name of persons...
Paul
> As others indicated having intelligence to recognize the terms (e.g.
> Kate should be in name) or some user indication to do so can make thing
> more precise but is rarely done.
> Alexandre Rafalovitch <ma...@gmail.com>
> 1 novembre 2015 13:07
> Which is what I believe Ted Sullivan is working on and presented at
> the latest Lucene/Solr Revolution. His presentation does not seem to
> be up, but he was writing about it on:
> http://lucidworks.com/blog/author/tedsullivan/
> Erick Erickson <ma...@gmail.com>
> 1 novembre 2015 07:40
> Yeah, that's actually a tough one. You have no control over what the
> user types,
> you have to try to guess what they meant.
Re: Kate Winslet vs Winslet Kate
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Which is what I believe Ted Sullivan is working on and presented at
the latest Lucene/Solr Revolution. His presentation does not seem to
be up, but he was writing about it on:
http://lucidworks.com/blog/author/tedsullivan/
Regards,
Alex.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 1 November 2015 at 01:40, Erick Erickson <er...@gmail.com> wrote:
> Yeah, that's actually a tough one. You have no control over what the user types,
> you have to try to guess what they meant.
>
> To do that right, you really have to have some meta-data besides what the user
> typed in, i.e. recognize "kate" and "winslet" are proper names and "movies" is
> something else and break up the query appropriately behind the scenes.
Re: Kate Winslet vs Winslet Kate
Posted by Paul Libbrecht <pa...@hoplahup.net>.
I believe that very many installations of solr actually need a query
expansion such as the one you describe below with an indexing of each
textual fields in multiple forms (string, straight
(whitespace/ideaograms), stemmed, phonetic).
Thanks to edismax, I think, you would do the following expansion:
- 2.0 for string match (same field only, complete value)
- 1.8 for straight match phrase (same field only, using a slop)
- 1.5 for straight match in bag of words
- 1.3 for stemmed match in bag of words
- 1.1 for phonetic match in bag of words
I think you can do that with edismax: sure about the parameter
distribution, just not sure about the pf usage, this might need two
straight fields which is quite cheap.
As others indicated having intelligence to recognize the terms (e.g.
Kate should be in name) or some user indication to do so can make thing
more precise but is rarely done.
Please note that this is just a suggestion. In particular, parameters
really need some testing and adjustment, I think.
Paul
> Erick Erickson <ma...@gmail.com>
> 1 novembre 2015 07:40
> Yeah, that's actually a tough one. You have no control over what the
> user types,
> you have to try to guess what they meant.
>
> To do that right, you really have to have some meta-data besides what
> the user
> typed in, i.e. recognize "kate" and "winslet" are proper names and
> "movies" is
> something else and break up the query appropriately behind the scenes.
>
> edismax might help here. You could copyField for everything into a
> bag_of_words field then boost the name field quite high relative to the
> bag_of_words field. That way, and _assuming_ that the bag_of_words
> field had all three words, then the user at least gets something.
>
> You can also do some tricks with edismax and the "pf" parameters. That
> option automatically takes the input and makes a phrase out of it against
> the field, so you get better scores for, say, the name field if it
> contains
> the phrase "kate winslet". doesn't help with the kate winslet movies
> though.
>
> On Sat, Oct 31, 2015 at 11:11 PM, Daniel Valdivia
> Daniel Valdivia <ma...@danielvaldivia.com>
> 1 novembre 2015 07:11
> Perhaps
>
> q=name:("Kate AND Winslet")
>
> q=name:("Kate Winslet")
>
> Sent from my iPhone
>
> Yangrui Guo <ma...@gmail.com>
> 1 novembre 2015 06:21
> Thanks for the reply. Putting the name: before the terms did the work. I
> just wanted to generalize the search query because users might be
> interested in querying Kate Winslet herself or her movies. If user enter
> query string "Kate Winslet movie", the query q=name:(Kate AND Winslet AND
> movie) will return nothing.
>
> Yangrui Guo
>
> On Saturday, October 31, 2015, Erick Erickson <er...@gmail.com>
>
> Erick Erickson <ma...@gmail.com>
> 1 novembre 2015 05:27
> There are a couple of anomalies here.
>
> 1> kate AND winslet
> What does the query look like if you add &debug=true to the statement
> and look at the "parsed_query" section of the return? My guess is you
> typed "q=name:kate AND winslet" which parses as "q=name:kate AND
> default_search_field:winslet" and are getting matches you don't
> expect. You need something like "q=name:(kate AND winslet)" or
> "q=name:kate AND name:winslet". Note that if you're using eDIsmax it's
> more complicated, but that should still honor the intent.
>
> 2> I have no idea why searching for "Kate Winslet" in quotes returns
> anything, I wouldn't expect it to unless you mean you type in "q=kate
> winslet" which is searching against your default field, not the name
> field.
>
> Best,
> Erick
> Yangrui Guo <ma...@gmail.com>
> 1 novembre 2015 04:52
> Hi today I found an interesting aspect of solr. I imported IMDB data into
> solr. The IMDB puts last name before first name for its person's name
> field
> eg. "Winslet, Kate". When I search "Winslet Kate" with quotation marks I
> could get the exact result. However if I search "Kate Winslet" or Kate AND
> Winslet solr seem to return me all result containing either Kate or
> Winslet
> which is similar to "Winslet Kate"~999999. From user perspective I
> certainly want solr to treat Kate Winslet the same as Winslet Kate. Is
> there anyway to make solr score higher for terms in the same field?
>
> Yangrui
>
Re: Kate Winslet vs Winslet Kate
Posted by Erick Erickson <er...@gmail.com>.
Yeah, that's actually a tough one. You have no control over what the user types,
you have to try to guess what they meant.
To do that right, you really have to have some meta-data besides what the user
typed in, i.e. recognize "kate" and "winslet" are proper names and "movies" is
something else and break up the query appropriately behind the scenes.
edismax might help here. You could copyField for everything into a
bag_of_words field then boost the name field quite high relative to the
bag_of_words field. That way, and _assuming_ that the bag_of_words
field had all three words, then the user at least gets something.
You can also do some tricks with edismax and the "pf" parameters. That
option automatically takes the input and makes a phrase out of it against
the field, so you get better scores for, say, the name field if it contains
the phrase "kate winslet". doesn't help with the kate winslet movies
though.
On Sat, Oct 31, 2015 at 11:11 PM, Daniel Valdivia
<ho...@danielvaldivia.com> wrote:
> Perhaps
>
> q=name:("Kate AND Winslet")
>
> q=name:("Kate Winslet")
>
> Sent from my iPhone
>
>> On Oct 31, 2015, at 10:21 PM, Yangrui Guo <gu...@gmail.com> wrote:
>>
>> Thanks for the reply. Putting the name: before the terms did the work. I
>> just wanted to generalize the search query because users might be
>> interested in querying Kate Winslet herself or her movies. If user enter
>> query string "Kate Winslet movie", the query q=name:(Kate AND Winslet AND
>> movie) will return nothing.
>>
>> Yangrui Guo
>>
>> On Saturday, October 31, 2015, Erick Erickson <er...@gmail.com>
>> wrote:
>>
>>> There are a couple of anomalies here.
>>>
>>> 1> kate AND winslet
>>> What does the query look like if you add &debug=true to the statement
>>> and look at the "parsed_query" section of the return? My guess is you
>>> typed "q=name:kate AND winslet" which parses as "q=name:kate AND
>>> default_search_field:winslet" and are getting matches you don't
>>> expect. You need something like "q=name:(kate AND winslet)" or
>>> "q=name:kate AND name:winslet". Note that if you're using eDIsmax it's
>>> more complicated, but that should still honor the intent.
>>>
>>> 2> I have no idea why searching for "Kate Winslet" in quotes returns
>>> anything, I wouldn't expect it to unless you mean you type in "q=kate
>>> winslet" which is searching against your default field, not the name
>>> field.
>>>
>>> Best,
>>> Erick
>>>
>>> On Sat, Oct 31, 2015 at 8:52 PM, Yangrui Guo <guoyangrui@gmail.com
>>> <javascript:;>> wrote:
>>>> Hi today I found an interesting aspect of solr. I imported IMDB data into
>>>> solr. The IMDB puts last name before first name for its person's name
>>> field
>>>> eg. "Winslet, Kate". When I search "Winslet Kate" with quotation marks I
>>>> could get the exact result. However if I search "Kate Winslet" or Kate
>>> AND
>>>> Winslet solr seem to return me all result containing either Kate or
>>> Winslet
>>>> which is similar to "Winslet Kate"~999999. From user perspective I
>>>> certainly want solr to treat Kate Winslet the same as Winslet Kate. Is
>>>> there anyway to make solr score higher for terms in the same field?
>>>>
>>>> Yangrui
>>>
Re: Kate Winslet vs Winslet Kate
Posted by Daniel Valdivia <ho...@danielvaldivia.com>.
Perhaps
q=name:("Kate AND Winslet")
q=name:("Kate Winslet")
Sent from my iPhone
> On Oct 31, 2015, at 10:21 PM, Yangrui Guo <gu...@gmail.com> wrote:
>
> Thanks for the reply. Putting the name: before the terms did the work. I
> just wanted to generalize the search query because users might be
> interested in querying Kate Winslet herself or her movies. If user enter
> query string "Kate Winslet movie", the query q=name:(Kate AND Winslet AND
> movie) will return nothing.
>
> Yangrui Guo
>
> On Saturday, October 31, 2015, Erick Erickson <er...@gmail.com>
> wrote:
>
>> There are a couple of anomalies here.
>>
>> 1> kate AND winslet
>> What does the query look like if you add &debug=true to the statement
>> and look at the "parsed_query" section of the return? My guess is you
>> typed "q=name:kate AND winslet" which parses as "q=name:kate AND
>> default_search_field:winslet" and are getting matches you don't
>> expect. You need something like "q=name:(kate AND winslet)" or
>> "q=name:kate AND name:winslet". Note that if you're using eDIsmax it's
>> more complicated, but that should still honor the intent.
>>
>> 2> I have no idea why searching for "Kate Winslet" in quotes returns
>> anything, I wouldn't expect it to unless you mean you type in "q=kate
>> winslet" which is searching against your default field, not the name
>> field.
>>
>> Best,
>> Erick
>>
>> On Sat, Oct 31, 2015 at 8:52 PM, Yangrui Guo <guoyangrui@gmail.com
>> <javascript:;>> wrote:
>>> Hi today I found an interesting aspect of solr. I imported IMDB data into
>>> solr. The IMDB puts last name before first name for its person's name
>> field
>>> eg. "Winslet, Kate". When I search "Winslet Kate" with quotation marks I
>>> could get the exact result. However if I search "Kate Winslet" or Kate
>> AND
>>> Winslet solr seem to return me all result containing either Kate or
>> Winslet
>>> which is similar to "Winslet Kate"~999999. From user perspective I
>>> certainly want solr to treat Kate Winslet the same as Winslet Kate. Is
>>> there anyway to make solr score higher for terms in the same field?
>>>
>>> Yangrui
>>
Re: Kate Winslet vs Winslet Kate
Posted by Yangrui Guo <gu...@gmail.com>.
Thanks for the reply. Putting the name: before the terms did the work. I
just wanted to generalize the search query because users might be
interested in querying Kate Winslet herself or her movies. If user enter
query string "Kate Winslet movie", the query q=name:(Kate AND Winslet AND
movie) will return nothing.
Yangrui Guo
On Saturday, October 31, 2015, Erick Erickson <er...@gmail.com>
wrote:
> There are a couple of anomalies here.
>
> 1> kate AND winslet
> What does the query look like if you add &debug=true to the statement
> and look at the "parsed_query" section of the return? My guess is you
> typed "q=name:kate AND winslet" which parses as "q=name:kate AND
> default_search_field:winslet" and are getting matches you don't
> expect. You need something like "q=name:(kate AND winslet)" or
> "q=name:kate AND name:winslet". Note that if you're using eDIsmax it's
> more complicated, but that should still honor the intent.
>
> 2> I have no idea why searching for "Kate Winslet" in quotes returns
> anything, I wouldn't expect it to unless you mean you type in "q=kate
> winslet" which is searching against your default field, not the name
> field.
>
> Best,
> Erick
>
> On Sat, Oct 31, 2015 at 8:52 PM, Yangrui Guo <guoyangrui@gmail.com
> <javascript:;>> wrote:
> > Hi today I found an interesting aspect of solr. I imported IMDB data into
> > solr. The IMDB puts last name before first name for its person's name
> field
> > eg. "Winslet, Kate". When I search "Winslet Kate" with quotation marks I
> > could get the exact result. However if I search "Kate Winslet" or Kate
> AND
> > Winslet solr seem to return me all result containing either Kate or
> Winslet
> > which is similar to "Winslet Kate"~999999. From user perspective I
> > certainly want solr to treat Kate Winslet the same as Winslet Kate. Is
> > there anyway to make solr score higher for terms in the same field?
> >
> > Yangrui
>
Re: Kate Winslet vs Winslet Kate
Posted by Erick Erickson <er...@gmail.com>.
There are a couple of anomalies here.
1> kate AND winslet
What does the query look like if you add &debug=true to the statement
and look at the "parsed_query" section of the return? My guess is you
typed "q=name:kate AND winslet" which parses as "q=name:kate AND
default_search_field:winslet" and are getting matches you don't
expect. You need something like "q=name:(kate AND winslet)" or
"q=name:kate AND name:winslet". Note that if you're using eDIsmax it's
more complicated, but that should still honor the intent.
2> I have no idea why searching for "Kate Winslet" in quotes returns
anything, I wouldn't expect it to unless you mean you type in "q=kate
winslet" which is searching against your default field, not the name
field.
Best,
Erick
On Sat, Oct 31, 2015 at 8:52 PM, Yangrui Guo <gu...@gmail.com> wrote:
> Hi today I found an interesting aspect of solr. I imported IMDB data into
> solr. The IMDB puts last name before first name for its person's name field
> eg. "Winslet, Kate". When I search "Winslet Kate" with quotation marks I
> could get the exact result. However if I search "Kate Winslet" or Kate AND
> Winslet solr seem to return me all result containing either Kate or Winslet
> which is similar to "Winslet Kate"~999999. From user perspective I
> certainly want solr to treat Kate Winslet the same as Winslet Kate. Is
> there anyway to make solr score higher for terms in the same field?
>
> Yangrui