You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Aaron Zeckoski <aa...@vt.edu> on 2010/02/10 15:04:43 UTC
Cannot get like exact searching to work
I am using SOLR 1.3 and my server is embedded and accessed using SOLRJ.
I would like to setup my searches so that exact matches are the first
results returned, followed by near matches, and finally token based
matches.
For example, if I have a summary field in schema which is created
using copyField from a bunch of other fields:
"My item title, keyword, other, stuff"
I want this search to match the item above first and foremost:
1) "My item title*"
Then this one:
2) "my item*"
and finally this one should also work:
3) "my title"
I tried creating a field to hold exact match data (summaryExact) which
actually works if I paste in the precise text but stops working as
soon as I add any wildcard to it. In other words I get no matches for
"My item title*" but I get 1 match for "My item title". I also tried
this:
(summary:"my item" || summaryExact:"my item*"^3)
but that results in 0 matches as well.
I could not quite figure out which tokenizer to use if I don't want
any tokens created but just want to trim and lowercase the string so
let me know if you have ideas on this. Basically, I want something
similar to DB "like" matching without case sensitivity and probably
trimmed as well. I don't really want the field to be tokenized though.
I am attaching my schema in case that helps.
I have spent a few days reading through the SOLR documentation and
forums and trying various things to get this to work but I just end up
making the matching worse when I make changes. I appreciate any
pointers, links, or ideas.
Thanks!
-AZ
--
Aaron Zeckoski (azeckoski (at) vt.edu)
Senior Research Engineer - CARET - University of Cambridge
https://twitter.com/azeckoski - http://www.linkedin.com/in/azeckoski
http://aaronz-sakai.blogspot.com/ - http://tinyurl.com/azprofile
Re: Cannot get like exact searching to work
Posted by Chris Hostetter <ho...@fucit.org>.
: > Can your query consist of more than one words?
:
: Yes, and I expect it almost always will (the query string is coming
: from a search box on a website).
...
: Actually it won't. The data I am indexing has extra spaces in front
: and is capitalized. I really need to be able to filter it through the
: lowercase and trim filter without tokenizing it.
...
: >> The idea is that a "phrase" match would be boosted over the
: >> normal
: >> token matches and would show up first in the listing. Let
This is starting to smell like an XY Problem...
http://people.apache.org/~hossman/#xyproblem
...you mentioned wanting prefix type queries to work, but that seems to be
based on your initial approach of using an "exact" (ie: untokenized) field
for your matches -- all of your examples seem to want matching at a "word"
level, not partial words.
If your ultimate goal is just that "exact' matches score higher then
documents containing all fo the same words in a differnet order (which
should score higher then docs only containing a few of the words) then i
think you are just making things harder for yourself then you really need
... "defType=dismax" should be able to solve all of your problems -- just
specify the field(s) you want to search in the qf and pf params and
documents with all the "words" in a phrase will appear first.
-Hoss
Re: Cannot get like exact searching to work
Posted by Ahmet Arslan <io...@yahoo.com>.
> > If you use string type for summaryExact you can run
> this query summaryExact:my\ item* It will bring you all
> documents begins with my item.
>
> Actually it won't. The data I am indexing has extra spaces
> in front
> and is capitalized. I really need to be able to filter it
> through the
> lowercase and trim filter without tokenizing it.
> Is there a way to apply filters to the string type (I am
> pretty sure
> there is not)?
As you said you cannot apply. But you can mimic it with KeywordTokenizer + TrimFilter + LowercaseFilter combination.
> This doesn't seem to align with the results I am seeing
> when I do searches. Are you saying that if I do a search like this it
> will boost the phrase matches while still doing token matches?
> q=summary:"my item"^5
Exactly. However you need to add individual terms also :
&q=summary:("my item"^5 my item)
Can you test this query? I think this is what you are looking for.
Just for your information: About multi-valued fields there is a positionIncrementGap value that puts space between multiple fields of this type on the same document, with the purpose of preventing false phrase matching across fields.
Re: Cannot get like exact searching to work
Posted by Aaron Zeckoski <aa...@vt.edu>.
On Thu, Feb 11, 2010 at 1:52 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>> What I really want is the equivalent of a match like this
>> along with
>> the normal tokenized matching (where the query has been
>> lowercased and
>> trimmed as well):
>> select * from blah where lowercase(column) like '%query%';
>> I think this is called a phrase match or something like
>> that.
>
> Can your query consist of more than one words?
Yes, and I expect it almost always will (the query string is coming
from a search box on a website).
>> However, wildcards cannot be used at the beginning of query so I
>> guess I can live with only being able to startsWith type matching until
>> that is fixed.
>
> With solr.ReversedWildcardFilterFactory it is possible. But it is in 1.4.0.
OK, so I may need to seriously look at SOLR 1.4 if I want to do a
"*stuff*" search.
>> For now I have tried to do that using this:
>> query = (summary:"my item" || summaryExact:"my item*"^3)
>> but I would do this if I could:
>> query = (summary:"my item" || summaryExact:"*my item*"^3)
>
> If you use string type for summaryExact you can run this query summaryExact:my\ item* It will bring you all documents begins with my item.
Actually it won't. The data I am indexing has extra spaces in front
and is capitalized. I really need to be able to filter it through the
lowercase and trim filter without tokenizing it.
Is there a way to apply filters to the string type (I am pretty sure
there is not)?
>> The idea is that a "phrase" match would be boosted over the
>> normal
>> token matches and would show up first in the listing. Let
>> me know if
>> more examples would help. I am happy to provide them.
>
> More examples will be great. Because boosting phrase match on a tokenized field can be achieved by something like "my item"^5 my item
> I didn't understand need of * operator.
> Also this query will retrieve documents below:
>
> something my item something
> my something item something
>
> We can say that it already behaves %like% query.
This doesn't seem to align with the results I am seeing when I do
searches. Are you saying that if I do a search like this it will boost
the phrase matches while still doing token matches?
q=summary:"my item"^5
or do I have to not use my summary field (the one I copy the other fields into).
-AZ
--
Aaron Zeckoski (azeckoski (at) vt.edu)
Senior Research Engineer - CARET - University of Cambridge
https://twitter.com/azeckoski - http://www.linkedin.com/in/azeckoski
http://aaronz-sakai.blogspot.com/ - http://tinyurl.com/azprofile
Re: Cannot get like exact searching to work
Posted by Ahmet Arslan <io...@yahoo.com>.
> I might be able to try this out though in general the
> project has a
> policy about only using released code (no trunk/unstable).
> https://issues.apache.org/jira/browse/SOLR-1604
> It looks like the kind of searching I want to do is not
> really
> supported in SOLR by default though. Is that correct?
* or ? operators inside phrases are not supported in Solr by default. In Lucene contrib package there is a ComplexPhraseQueryParser that allows them.
> I thought that was what my exact definition was doing
> except I also
> want the exact field to be lowercased and trimmed (which I
> don't want
> for all strings). Can you explain what is wrong with the
> current
> definition so I can fix it?
in your attachment the definition is:
<fieldType name="exact" class="solr.TextField" compressed="false" indexed="true" stored="true" />
I have never seen a field type definition without charfilter/tokenizer/tokenfiler chain. Usually string type is used for exact match.
> What I really want is the equivalent of a match like this
> along with
> the normal tokenized matching (where the query has been
> lowercased and
> trimmed as well):
> select * from blah where lowercase(column) like '%query%';
> I think this is called a phrase match or something like
> that.
Can your query consist of more than one words?
> However, wildcards cannot be used at the beginning of query so I
> guess I can live with only being able to startsWith type matching until
> that is fixed.
With solr.ReversedWildcardFilterFactory it is possible. But it is in 1.4.0.
> For now I have tried to do that using this:
> query = (summary:"my item" || summaryExact:"my item*"^3)
> but I would do this if I could:
> query = (summary:"my item" || summaryExact:"*my item*"^3)
If you use string type for summaryExact you can run this query summaryExact:my\ item* It will bring you all documents begins with my item.
> The idea is that a "phrase" match would be boosted over the
> normal
> token matches and would show up first in the listing. Let
> me know if
> more examples would help. I am happy to provide them.
More examples will be great. Because boosting phrase match on a tokenized field can be achieved by something like "my item"^5 my item
I didn't understand need of * operator.
Also this query will retrieve documents below:
something my item something
my something item something
We can say that it already behaves %like% query.
Re: Cannot get like exact searching to work
Posted by Aaron Zeckoski <aa...@vt.edu>.
On Thu, Feb 11, 2010 at 8:39 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>> I am using SOLR 1.3 and my server is
>> embedded and accessed using SOLRJ.
>> I would like to setup my searches so that exact matches are
>> the first
>> results returned, followed by near matches, and finally
>> token based
>> matches.
>> For example, if I have a summary field in schema which is
>> created
>> using copyField from a bunch of other fields:
>> "My item title, keyword, other, stuff"
>>
>> I want this search to match the item above first and
>> foremost:
>> 1) "My item title*"
>>
>> Then this one:
>> 2) "my item*"
>
> Wildcards inside phrases are not supported by default. You can use SOLR-1604 for that in solr 1.4.0. But i am not sure it will work with 1.3. Can you try?
I might be able to try this out though in general the project has a
policy about only using released code (no trunk/unstable).
https://issues.apache.org/jira/browse/SOLR-1604
It looks like the kind of searching I want to do is not really
supported in SOLR by default though. Is that correct?
>> I tried creating a field to hold exact match data
>> (summaryExact) which
>> actually works if I paste in the precise text but stops
>> working as
>> soon as I add any wildcard to it.
>
> Your <fieldType name="exact" definition is wrong. You can use directly string field type which is not analyzed/tonenized. Where string definiton is:
>
> <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>
I thought that was what my exact definition was doing except I also
want the exact field to be lowercased and trimmed (which I don't want
for all strings). Can you explain what is wrong with the current
definition so I can fix it?
>> I could not quite figure out which tokenizer to use if I
>> don't want
>> any tokens created but just want to trim and lowercase the
>> string so
>> let me know if you have ideas on this.
>
> KeywordTokenizerFactory + TrimFilterFactory + LowercaseFilterFactory combination can do that put punctuations won't be removed between tokens.
>
>> Basically, I want something
>> similar to DB "like" matching without case sensitivity and
>> probably
>> trimmed as well. I don't really want the field to be
>> tokenized though.
>
> Your examples seem you want to search something like startsWith? Can you explain more in detail?
What I really want is the equivalent of a match like this along with
the normal tokenized matching (where the query has been lowercased and
trimmed as well):
select * from blah where lowercase(column) like '%query%';
I think this is called a phrase match or something like that. However,
wildcards cannot be used at the beginning of query so I guess I can
live with only being able to startsWith type matching until that is
fixed. For now I have tried to do that using this:
query = (summary:"my item" || summaryExact:"my item*"^3)
but I would do this if I could:
query = (summary:"my item" || summaryExact:"*my item*"^3)
The idea is that a "phrase" match would be boosted over the normal
token matches and would show up first in the listing. Let me know if
more examples would help. I am happy to provide them.
> Also your <fieldType name="name" class="solr.StrField" ..> declation is also wrong. It should use class="solr.TextField".
OK, I will see if I can figure out how to correct that.
Thanks for all the help so far
-AZ
--
Aaron Zeckoski (azeckoski (at) vt.edu)
Senior Research Engineer - CARET - University of Cambridge
https://twitter.com/azeckoski - http://www.linkedin.com/in/azeckoski
http://aaronz-sakai.blogspot.com/ - http://tinyurl.com/azprofile
Re: Cannot get like exact searching to work
Posted by Ahmet Arslan <io...@yahoo.com>.
> I am using SOLR 1.3 and my server is
> embedded and accessed using SOLRJ.
> I would like to setup my searches so that exact matches are
> the first
> results returned, followed by near matches, and finally
> token based
> matches.
> For example, if I have a summary field in schema which is
> created
> using copyField from a bunch of other fields:
> "My item title, keyword, other, stuff"
>
> I want this search to match the item above first and
> foremost:
> 1) "My item title*"
>
> Then this one:
> 2) "my item*"
Wildcards inside phrases are not supported by default. You can use SOLR-1604 for that in solr 1.4.0. But i am not sure it will work with 1.3. Can you try?
> I tried creating a field to hold exact match data
> (summaryExact) which
> actually works if I paste in the precise text but stops
> working as
> soon as I add any wildcard to it.
Your <fieldType name="exact" definition is wrong. You can use directly string field type which is not analyzed/tonenized. Where string definiton is:
<fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>
> I could not quite figure out which tokenizer to use if I
> don't want
> any tokens created but just want to trim and lowercase the
> string so
> let me know if you have ideas on this.
KeywordTokenizerFactory + TrimFilterFactory + LowercaseFilterFactory combination can do that put punctuations won't be removed between tokens.
> Basically, I want something
> similar to DB "like" matching without case sensitivity and
> probably
> trimmed as well. I don't really want the field to be
> tokenized though.
Your examples seem you want to search something like startsWith? Can you explain more in detail?
Also your <fieldType name="name" class="solr.StrField" ..> declation is also wrong. It should use class="solr.TextField".