You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Rob Nikander <ro...@gmail.com> on 2014/08/28 19:38:33 UTC
indexing all suffixes to support leading wildcard?
Hi,
I've got some short fields (phone num, email) that I'd like to search using
good old string matching. (The full query is a boolean "or" that also uses
real text fields.) I see the warnings about wildcard queries that start
with *, and I'm wondering... do you think it would be a good idea to index
all the suffixes? Eg, a phone num 5551234, would become 7 values for the
"phoneNum" field: 4, 34, 234, etc. So "512*" would be a hit.
And maybe do something with the boosts so it doesn't overvalue the match
when it hits multiple values. ?
Rob
Re: indexing all suffixes to support leading wildcard?
Posted by Rob Nikander <ro...@gmail.com>.
Doh. Nevermind, I see it. I was searching with same analyzer that I used to
index. Usually that's right, but in this case, no.
Rob
On Fri, Aug 29, 2014 at 10:59 AM, Rob Nikander <ro...@gmail.com>
wrote:
> Thanks. That got the search working. Do you know if there's a trick for
> using FastVectorHighlighter with ngrams? I followed that doc's advice to
> use NGramTokenizer, and right now if the search matches "1234" it will only
> highlight "123".
>
> Rob
>
>
>
> On Fri, Aug 29, 2014 at 12:18 AM, Jack Krupansky <ja...@basetechnology.com>
> wrote:
>
>> Use the ngram token filter, and the a query of 512 would match by itself:
>> http://lucene.apache.org/core/4_9_0/analyzers-common/org/
>> apache/lucene/analysis/ngram/NGramTokenFilter.html
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Erick Erickson
>> Sent: Thursday, August 28, 2014 11:52 PM
>> To: java-user
>> Subject: Re: indexing all suffixes to support leading wildcard?
>>
>>
>> The "usual" approach is to index to a second field but.... backwards.
>> See ReverseStringFilter... Then all your leading wildcards
>> are really trailing wildcards in the reversed field.
>>
>> Best,
>> Erick
>>
>>
>> On Thu, Aug 28, 2014 at 10:38 AM, Rob Nikander <ro...@gmail.com>
>> wrote:
>>
>> Hi,
>>>
>>> I've got some short fields (phone num, email) that I'd like to search
>>> using
>>> good old string matching. (The full query is a boolean "or" that also
>>> uses
>>> real text fields.) I see the warnings about wildcard queries that start
>>> with *, and I'm wondering... do you think it would be a good idea to
>>> index
>>> all the suffixes? Eg, a phone num 5551234, would become 7 values for the
>>> "phoneNum" field: 4, 34, 234, etc. So "512*" would be a hit.
>>>
>>> And maybe do something with the boosts so it doesn't overvalue the match
>>> when it hits multiple values. ?
>>>
>>> Rob
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: indexing all suffixes to support leading wildcard?
Posted by Rob Nikander <ro...@gmail.com>.
Thanks. That got the search working. Do you know if there's a trick for
using FastVectorHighlighter with ngrams? I followed that doc's advice to
use NGramTokenizer, and right now if the search matches "1234" it will only
highlight "123".
Rob
On Fri, Aug 29, 2014 at 12:18 AM, Jack Krupansky <ja...@basetechnology.com>
wrote:
> Use the ngram token filter, and the a query of 512 would match by itself:
> http://lucene.apache.org/core/4_9_0/analyzers-common/org/
> apache/lucene/analysis/ngram/NGramTokenFilter.html
>
> -- Jack Krupansky
>
> -----Original Message----- From: Erick Erickson
> Sent: Thursday, August 28, 2014 11:52 PM
> To: java-user
> Subject: Re: indexing all suffixes to support leading wildcard?
>
>
> The "usual" approach is to index to a second field but.... backwards.
> See ReverseStringFilter... Then all your leading wildcards
> are really trailing wildcards in the reversed field.
>
> Best,
> Erick
>
>
> On Thu, Aug 28, 2014 at 10:38 AM, Rob Nikander <ro...@gmail.com>
> wrote:
>
> Hi,
>>
>> I've got some short fields (phone num, email) that I'd like to search
>> using
>> good old string matching. (The full query is a boolean "or" that also
>> uses
>> real text fields.) I see the warnings about wildcard queries that start
>> with *, and I'm wondering... do you think it would be a good idea to index
>> all the suffixes? Eg, a phone num 5551234, would become 7 values for the
>> "phoneNum" field: 4, 34, 234, etc. So "512*" would be a hit.
>>
>> And maybe do something with the boosts so it doesn't overvalue the match
>> when it hits multiple values. ?
>>
>> Rob
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: indexing all suffixes to support leading wildcard?
Posted by Jack Krupansky <ja...@basetechnology.com>.
Use the ngram token filter, and the a query of 512 would match by itself:
http://lucene.apache.org/core/4_9_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramTokenFilter.html
-- Jack Krupansky
-----Original Message-----
From: Erick Erickson
Sent: Thursday, August 28, 2014 11:52 PM
To: java-user
Subject: Re: indexing all suffixes to support leading wildcard?
The "usual" approach is to index to a second field but.... backwards.
See ReverseStringFilter... Then all your leading wildcards
are really trailing wildcards in the reversed field.
Best,
Erick
On Thu, Aug 28, 2014 at 10:38 AM, Rob Nikander <ro...@gmail.com>
wrote:
> Hi,
>
> I've got some short fields (phone num, email) that I'd like to search
> using
> good old string matching. (The full query is a boolean "or" that also
> uses
> real text fields.) I see the warnings about wildcard queries that start
> with *, and I'm wondering... do you think it would be a good idea to index
> all the suffixes? Eg, a phone num 5551234, would become 7 values for the
> "phoneNum" field: 4, 34, 234, etc. So "512*" would be a hit.
>
> And maybe do something with the boosts so it doesn't overvalue the match
> when it hits multiple values. ?
>
> Rob
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: indexing all suffixes to support leading wildcard?
Posted by Erick Erickson <er...@gmail.com>.
The "usual" approach is to index to a second field but.... backwards.
See ReverseStringFilter... Then all your leading wildcards
are really trailing wildcards in the reversed field.
Best,
Erick
On Thu, Aug 28, 2014 at 10:38 AM, Rob Nikander <ro...@gmail.com>
wrote:
> Hi,
>
> I've got some short fields (phone num, email) that I'd like to search using
> good old string matching. (The full query is a boolean "or" that also uses
> real text fields.) I see the warnings about wildcard queries that start
> with *, and I'm wondering... do you think it would be a good idea to index
> all the suffixes? Eg, a phone num 5551234, would become 7 values for the
> "phoneNum" field: 4, 34, 234, etc. So "512*" would be a hit.
>
> And maybe do something with the boosts so it doesn't overvalue the match
> when it hits multiple values. ?
>
> Rob
>