You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by Walter Underwood <wu...@netflix.com> on 2006/11/07 23:20:13 UTC

Adding Phonetic Search to Solr

I haven't found fuzzy or phonetic search in Solr, and I have a couple
of approaches I might try:

1. Adding fuzzy to the DisMax specs.

2. Adding a phonetic token filter and relying on the per-field analyzer
support.

Option 2 seems like it would be a lot faster in production, and
probably easier to implement. Does that seem right?

How do I specify the new token filter factory in the schema file?
I don't quite get the mapping from solr.FooFilterFactory to
org.apache.solr.analysis.FooFilterFactory.

wunder
-- 
Walter Underwood
Search Guru, Netflix



Re: Adding Phonetic Search to Solr

Posted by Walter Underwood <wu...@netflix.com>.
On 11/7/06 3:26 PM, "Mike Klaas" <mi...@gmail.com> wrote:

> Is the state of the art in phonetic token generation reasonable?  I've
> been rather disappointed with some implementations (eg. SOUNDEX in
> MySQL, MSSQL).

SOUNDEX is excellent technology for its time, but its time was 1920.

Double Metaphone is far more complex and works fairly well. There is
an Apache commons codec implementation available. It is certainly
good enough for matching proper names, like Moody and Mudie or
Cathy and Kathie.

There are some commercial phonetic coders, but I don't have any
experience with those.

wunder
-- 
Walter Underwood
Search Guru, Netflix



Re: Re: Adding Phonetic Search to Solr

Posted by Mike Klaas <mi...@gmail.com>.
On 11/7/06, Walter Underwood <wu...@netflix.com> wrote:
> On 11/7/06 2:30 PM, "Mike Klaas" <mi...@gmail.com> wrote:
> > On 11/7/06, Walter Underwood <wu...@netflix.com> wrote:
> >>
> >> 1. Adding fuzzy to the DisMax specs.
> >
> > What do you envisage the implementation looking like?
>
> Probably continue with the template-like patterns already there.
>
>   title^2.0   (search title field with boost of 2.0)
>   title~      (search title field with fuzzy matching)

Ah, I see what you mean.  Seems reasonable (as someone who is utter
unfamiliar with fuzzy queries in lucene).

<>
> Ah, I missed the <analyzer> example with a stock Lucene analyzer.
> Oops. I still need to write an Analyzer, because there is no standard
> phonetic search in Lucene today. There are some patches and addons
> floating around.

Is the state of the art in phonetic token generation reasonable?  I've
been rather disappointed with some implementations (eg. SOUNDEX in
MySQL, MSSQL).

cheers,
-Mike

Re: Adding Phonetic Search to Solr

Posted by Yonik Seeley <yo...@apache.org>.
On 11/7/06, Mike Klaas <mi...@gmail.com> wrote:
> > 2. Adding a phonetic token filter and relying on the per-field analyzer
> > support.
> >
> > Option 2 seems like it would be a lot faster in production, and
> > probably easier to implement. Does that seem right?
>
> I'm not sure why any modification to solr would be necessary.  You
> could add a field with a phonetic analyzer and use copyField to copy
> your search fields to it.  Search will use the modified analyzer
> automatically.

I assumed he meant the implementation of the filter/analyzer itself
(unless there is already a Lucene filter we can use).  Unless there is
a lucene analyzer (which can be used directly), a FilterFactory for
the filter will have to be added, but that's easy.

> > How do I specify the new token filter factory in the schema file?
> > I don't quite get the mapping from solr.FooFilterFactory to
> > org.apache.solr.analysis.FooFilterFactory.

When Solr was an internal CNET project, the package name was just solr
(or actually, just solar).  I kept the short version as a form of
backward compatibility, and I sort of liked it because it made the
schema less verbose.

Anyway there is a set of standard  packages that will be checked if
the class name isn't found as-is.  I thought it was documented but I
can't seem to find it...


-Yonik

Re: Adding Phonetic Search to Solr

Posted by Chris Hostetter <ho...@fucit.org>.
: >> 1. Adding fuzzy to the DisMax specs.
: >
: > What do you envisage the implementation looking like?
:
: Probably continue with the template-like patterns already there.
:
:   title^2.0   (search title field with boost of 2.0)
:   title~      (search title field with fuzzy matching)

Interesting idea ... that's really a seperate idea from "Phonetic" search
right? ... fuzzy searching (ala: FuzzyQuery) is really a seperate beast
from phonetics (which i assume would best be implimented using your second
idea of a new TokenFilter)

I'm all in favor of both ideas ... but i wonder if adding fuzzy support to
dismax would be better done with a new param similar to "pf" (ie "ff" for
fuzzy fields and "fslop" for the min similaity)

: Still, it seems like others might want to use a phonetic token
: filter with the <filter> specs. I'd be glad to contribute that,
: if others think it would be useful.

yeah .. you may want to just write a generic Lucene TokenFilter that does
the phonetics, submit it as a page to Lucene-Java and then it's triial to
whip up a TokenFilterFactory to use it in Solr.



-Hoss


Re: Adding Phonetic Search to Solr

Posted by Walter Underwood <wu...@netflix.com>.
On 11/7/06 2:30 PM, "Mike Klaas" <mi...@gmail.com> wrote:
> On 11/7/06, Walter Underwood <wu...@netflix.com> wrote:
>> 
>> 1. Adding fuzzy to the DisMax specs.
> 
> What do you envisage the implementation looking like?

Probably continue with the template-like patterns already there.

  title^2.0   (search title field with boost of 2.0)
  title~      (search title field with fuzzy matching)

>> 2. Adding a phonetic token filter and relying on the per-field analyzer
>> support.
> 
> I'm not sure why any modification to solr would be necessary.  You
> could add a field with a phonetic analyzer and use copyField to copy
> your search fields to it.  Search will use the modified analyzer
> automatically.

Ah, I missed the <analyzer> example with a stock Lucene analyzer.
Oops. I still need to write an Analyzer, because there is no standard
phonetic search in Lucene today. There are some patches and addons
floating around.

Still, it seems like others might want to use a phonetic token
filter with the <filter> specs. I'd be glad to contribute that,
if others think it would be useful.

wunder
-- 
Walter Underwood
Search Guru, Netflix



Re: Adding Phonetic Search to Solr

Posted by Mike Klaas <mi...@gmail.com>.
On 11/7/06, Walter Underwood <wu...@netflix.com> wrote:
> I haven't found fuzzy or phonetic search in Solr, and I have a couple
> of approaches I might try:
>
> 1. Adding fuzzy to the DisMax specs.

What do you envisage the implementation looking like?

> 2. Adding a phonetic token filter and relying on the per-field analyzer
> support.
>
> Option 2 seems like it would be a lot faster in production, and
> probably easier to implement. Does that seem right?

I'm not sure why any modification to solr would be necessary.  You
could add a field with a phonetic analyzer and use copyField to copy
your search fields to it.  Search will use the modified analyzer
automatically.

-Mike

Re: Adding Phonetic Search to Solr

Posted by Chris Hostetter <ho...@fucit.org>.
: 2. Adding a phonetic token filter and relying on the per-field analyzer
: support.
:
: Option 2 seems like it would be a lot faster in production, and
: probably easier to implement. Does that seem right?

yep, just write your Analyzer (or TokenFilter) and drop it in.

: How do I specify the new token filter factory in the schema file?
: I don't quite get the mapping from solr.FooFilterFactory to
: org.apache.solr.analysis.FooFilterFactory.

Classes *should* be specified using the fully qualified package, but for
convincince, if Solr sees a package name of just "solr" it checks a few
known packages based on context (ie: org.apache.solr.analysis,
o.a.s.schema, etc...)

As discussed in SOLR-68, getting the classloader to find your custom
analyzer is currently a pain in the ass .. the easiest thing to do that
should work in any servlet container is add it directly to the solr.war
... but i'm hopeingto make that unneccessary soon.



-Hoss