Advice on Exact Matching?

[Scott Gonyea <sc...@aitrus.org>]

Hi,

I am trying to make sure that when I search for text—regardless of
what that text is—that I get an exact match.  I'm *still* getting some
issues, and this last mile is becoming very painful.  The solr field,
for which I'm setting this up on, is pasted below my explanation.  I
appreciate any help.

Explanation:

I'm crawling websites with Nutch.  I'm performing some
mechanical-turk-like filtering and term matching.  The problem is,
there's some very gnarly behavior in Solr due to any number of
gotchas.

If I want to find *all* Solr documents that match
"[id]somejunk\hi[/id]" then life is instantly hell.

Likewise, lots of whitespace in between words throws it off " john
says hello,  how are you?"  I would love to be able to search for
these exact phrases.  If that's just not practical (I'm more than
willing to live with a bloated search index), what would some other
strategies be?

There's no MapReduce in Solr; I could attempt to do Hadoop-streaming,
but that's not very ideal for a variety of reasons.


Solr Schema.xml, fieldType "text" (no, this is not used everywhere;
only on 2 fields):


    <fieldType name="text"    class="solr.TextField"
positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1"     generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="1" splitOnCaseChange="1"/>
        <filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
        <filter class="solr.SynonymFilterFactory"
synonyms="synonyms.txt" expand="true" ignoreCase="true"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>


Thank you,
Scott Gonyea

Re: Advice on Exact Matching?

[Jonathan Rochkind <ro...@jhu.edu>]

There is a hacky kind of thing that Bill Dueber figured out for using 
multiple fields and dismax to BOOST "exact" matches, but include all 
matches in the result set.

You have to duplicate your data in a second non-tokenized field. Then 
you use dismax pf to super boost matches on the non-tokenized field. 
Because 'pf' is a phrase search, you don't run into trouble with dismax 
"pre-tokenization" on white space, even though it's a field that might 
have internal-token whitespace. (Using a non-tokenized field with dismax 
qf will basically never match a result with whitespace, unless it's 
phrase-quoted in query. But pf works.).

Because it was a non-tokenized field, it only matches (and triggers the 
dismax ps super boost) if it's an exact match. And it works. You CAN 
normalize your 'exact match' field in field analysis, removing 
punctuation or normalizing whitespace or whatever, and that works too, 
doing it both at index and query time analysis.



On 1/4/2011 4:28 PM, Chris Hostetter wrote:
> : I am trying to make sure that when I search for text—regardless of
> : what that text is—that I get an exact match.  I'm *still* getting some
> : issues, and this last mile is becoming very painful.  The solr field,
> : for which I'm setting this up on, is pasted below my explanation.  I
> : appreciate any help.
>
> if you are using a TextField with some analysis components, it's
> virtually impossible to get "exact" matches -- where my definition of
> exact is that the query text is character for character identical to the
> entire field value indexed.
>
> is your definition of exact match different?  i assme it must be since you
> are using TextField and talk about wanting to deal with whitespace between
> words.  so i think you need to explain a little bit better what your
> indexed data looks like, and what sample queries you expect to match that
> data (and equally important: what queries should *not* match thta data,
> and what data should *not* match those queries)
>
> : If I want to find *all* Solr documents that match
> : "[id]somejunk\hi[/id]" then life is instantly hell.
>
> 90% of the time when people have problems with "exact" matches it's
> because of QueryParser meta characters -- characters like ":", "[" and
> whitespace that the QUeryParser uses as instructions.  you can use the
> "raw" QParser to have every character treated as a literal....
>
> 	defType=raw
> 	q=[id]somejunk\hi[/id]
>
> -Hoss
>

Re: Advice on Exact Matching?

[Chris Hostetter <ho...@fucit.org>]

: I am trying to make sure that when I search for text—regardless of
: what that text is—that I get an exact match.  I'm *still* getting some
: issues, and this last mile is becoming very painful.  The solr field,
: for which I'm setting this up on, is pasted below my explanation.  I
: appreciate any help.

if you are using a TextField with some analysis components, it's 
virtually impossible to get "exact" matches -- where my definition of 
exact is that the query text is character for character identical to the 
entire field value indexed.

is your definition of exact match different?  i assme it must be since you 
are using TextField and talk about wanting to deal with whitespace between 
words.  so i think you need to explain a little bit better what your 
indexed data looks like, and what sample queries you expect to match that 
data (and equally important: what queries should *not* match thta data, 
and what data should *not* match those queries)

: If I want to find *all* Solr documents that match
: "[id]somejunk\hi[/id]" then life is instantly hell.

90% of the time when people have problems with "exact" matches it's 
because of QueryParser meta characters -- characters like ":", "[" and 
whitespace that the QUeryParser uses as instructions.  you can use the 
"raw" QParser to have every character treated as a literal....

	defType=raw
	q=[id]somejunk\hi[/id]

-Hoss

Re: Advice on Exact Matching?

[Erick Erickson <er...@gmail.com>]

You can gain a lot of insight into this kind of thing with the
admin/analysis page. Often the issue is that your tokenizing/
filtering isn't doing quite what you think. Try turning on the
debug checkboxes on that page and seeing what tokens are
generated at index and analysis page.

In particular, WordDelimiterFactory is often a surprise in how
it splits and recombines tokens. Including synonyms is another
potential issue. Not to mention the EnglishPorterFilterFactory.

If that's not helpful, could you paste some examples that you expect
to match that don't?

Best
Erick

On Thu, Dec 30, 2010 at 8:04 PM, Scott Gonyea <sc...@aitrus.org> wrote:

> Hi,
>
> I am trying to make sure that when I search for text—regardless of
> what that text is—that I get an exact match.  I'm *still* getting some
> issues, and this last mile is becoming very painful.  The solr field,
> for which I'm setting this up on, is pasted below my explanation.  I
> appreciate any help.
>
> Explanation:
>
> I'm crawling websites with Nutch.  I'm performing some
> mechanical-turk-like filtering and term matching.  The problem is,
> there's some very gnarly behavior in Solr due to any number of
> gotchas.
>
> If I want to find *all* Solr documents that match
> "[id]somejunk\hi[/id]" then life is instantly hell.
>
> Likewise, lots of whitespace in between words throws it off " john
> says hello,  how are you?"  I would love to be able to search for
> these exact phrases.  If that's just not practical (I'm more than
> willing to live with a bloated search index), what would some other
> strategies be?
>
> There's no MapReduce in Solr; I could attempt to do Hadoop-streaming,
> but that's not very ideal for a variety of reasons.
>
>
> Solr Schema.xml, fieldType "text" (no, this is not used everywhere;
> only on 2 fields):
>
>
>    <fieldType name="text"    class="solr.TextField"
> positionIncrementGap="100">
>      <analyzer>
>        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>        <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1"     generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="1" splitOnCaseChange="1"/>
>        <filter class="solr.EnglishPorterFilterFactory"
> protected="protwords.txt"/>
>        <filter class="solr.SynonymFilterFactory"
> synonyms="synonyms.txt" expand="true" ignoreCase="true"/>
>        <filter class="solr.LowerCaseFilterFactory"/>
>      </analyzer>
>    </fieldType>
>
>
> Thank you,
> Scott Gonyea
>