You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Aad Nales <aa...@rotterdam-cs.com> on 2005/02/09 13:23:57 UTC
sounds like spellcheck
In my Clipper days I could build an index on English words using a
technique that was called soundex. Searching in that index resulted in
hits of words that sounded the same. From what i remember this technique
only worked for English. Has it ever been generalized?
What i am trying to solve is this. A customer is looking for a solution
to spelling mistakes made by children (upto 10) when typing in queries.
The site is Dutch. Common mistakes are 'sgool' when searching for
'school'. The 'normal' spellcheckers and suggestors typically generate a
list where the 'sounds like' candidates' are too far away from the
result. So what I am thinking about doing is this:
1. create a parser that takes a word and creates a soundindex entry.
2. create list of 'correctly' spelled words either based on the index of
the website or on some kind of dictionary.
2a. perhaps create a n-gram index based on these words
3. accept a query, figure out that a spelling mistake has been made
3a find alternatives by parsing the query and searching the 'sound like
index' and then calculate and order the results
Steps 2 and 3 have been discussed at length in this forum and have even
made it to the sandbox. What I am left with is 1.
My thinking is processing a series of replacement statements that go like:
--
g sounds like ch if the immediate predecessor is an s.
o sounds like oo if the immediate predecessor is a consonant
--
But before I takes this to the next step I am wondering if anybody has
created or thought up alternative solutions?
Cheers,
Aad
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: sounds like spellcheck
Posted by Aad Nales <aa...@rotterdam-cs.com>.
Morus Walter wrote:
>
>Unfortunatley the website seems to be down.
>
>
>
Do you have the url? The sources are off course very welcome as well.
Cheers,
Aad
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: sounds like spellcheck
Posted by Morus Walter <mo...@tanto.de>.
Aad Nales writes:
>
> Steps 2 and 3 have been discussed at length in this forum and have even
> made it to the sandbox. What I am left with is 1.
>
> My thinking is processing a series of replacement statements that go like:
> --
> g sounds like ch if the immediate predecessor is an s.
> o sounds like oo if the immediate predecessor is a consonant
> --
>
> But before I takes this to the next step I am wondering if anybody has
> created or thought up alternative solutions?
>
An implementation of a rule based system to create such a pronounciation
form, can be found in a library called makelib that is part of an editor
named leanedit.
Unfortunatley the website seems to be down.
The lib is LGPL. If you're interested, I can send you a copy of the
sources. The only ruleset available is german though.
Morus
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: sounds like spellcheck
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Feb 9, 2005, at 7:23 AM, Aad Nales wrote:
> In my Clipper days I could build an index on English words using a
> technique that was called soundex. Searching in that index resulted in
> hits of words that sounded the same. From what i remember this
> technique only worked for English. Has it ever been generalized?
I do not know how Soundex/Metaphone/Double Metaphone work with
non-English languages, but these algorithms are in Jakarta Commons
Codec. I used the Metaphone algorithm as a custom analyzer example in
Lucene in Action. You'll see it in the source code distribution under
src/lia/analysis/codec. I did a couple of variations, one that adds
the metaphoned version as a token in the same position and one that
simply replaces it in the token stream.
I even envisioned this sounds-like feature being used for children. I
was mulling over this idea while having lunch with my son one day last
spring (he was 5 at the time). I asked him how to spell "cool cat" and
he replied "c-o-l c-a-t". I tried it out with the metaphone algorithm
and it matches!
http://www.lucenebook.com/search?query=cool+cat
Erik
>
> What i am trying to solve is this. A customer is looking for a
> solution to spelling mistakes made by children (upto 10) when typing
> in queries. The site is Dutch. Common mistakes are 'sgool' when
> searching for 'school'. The 'normal' spellcheckers and suggestors
> typically generate a list where the 'sounds like' candidates' are too
> far away from the result. So what I am thinking about doing is this:
>
> 1. create a parser that takes a word and creates a soundindex entry.
>
> 2. create list of 'correctly' spelled words either based on the index
> of the website or on some kind of dictionary.
> 2a. perhaps create a n-gram index based on these words
>
> 3. accept a query, figure out that a spelling mistake has been made
> 3a find alternatives by parsing the query and searching the 'sound
> like index' and then calculate and order the results
>
> Steps 2 and 3 have been discussed at length in this forum and have
> even made it to the sandbox. What I am left with is 1.
>
> My thinking is processing a series of replacement statements that go
> like:
> --
> g sounds like ch if the immediate predecessor is an s.
> o sounds like oo if the immediate predecessor is a consonant
> --
>
> But before I takes this to the next step I am wondering if anybody has
> created or thought up alternative solutions?
>
> Cheers,
> Aad
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: sounds like spellcheck
Posted by Aad Nales <aa...@rotterdam-cs.com>.
Thanks for the reference to Metaphone et al. This is the direction I am
looking for. What I don't get is why so much of the 'knowledge' of these
algoritms is stored in the 'process'. I guess it has to be performance.
cheers,
Aad
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: sounds like spellcheck [auf Viren geprueft]
Posted by Andrzej Bialecki <ab...@getopt.org>.
Jonathan O'Connor wrote:
> Aad,
> Are you trying to check the spelling of English words by Dutch children?
> Then, Phonetix or any of these other solutions may not be perfect.
>>>From my little knowledge of Dutch, a "g" is some sort of velar fricative
> (pronounced at the back of throat). And "ch" in english is also a velar
> fricative.
> You have to hope that the soundex/metaphone rules are broad enough to be
> used by both languages.
Soundex and Metaphone have been specifically designed for English. I
know from my experience with Swedish and Polish that their results for
other languages can range from mediocre to extremely bad. You should
definitely not blindly trust them, but perform careful tests using some
test corpus.
For Slavic languages you can use Daitch-Mokotoff instead, but I have no
idea about Dutch...
Another suggestion: did you try the solution developed by Dave Spencer
(look for NGramSpeller)?
--
Best regards,
Andrzej Bialecki
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: sounds like spellcheck [auf Viren geprueft]
Posted by Jonathan O'Connor <Jo...@xcom.de>.
Aad, Well at least that's easier.
Ciao,
Jonathan O'Connor
XCOM Dublin
Aad Nales <aa...@rotterdam-cs.com>
09/02/2005 16:16
Please respond to
"Lucene Users List" <lu...@jakarta.apache.org>
To
Lucene Users List <lu...@jakarta.apache.org>
cc
Subject
Re: sounds like spellcheck [auf Viren geprueft]
Jonathan O'Connor wrote:
>Aad,
>Are you trying to check the spelling of English words by Dutch children?
>
>
Uh no, I am trying to correct the spelling of Dutch words by Dutch
children who, as most children do, make phonetic spelling mistakes.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
*** Aktuelle Veranstaltungen der XCOM AG ***
XCOM laedt ein zur IBM Workplace Roadshow in Frankfurt (16.02.2005), Duesseldorf (23.02.2005) und Berlin (02.03.2005)
Anmeldung und Information unter http://lotus.xcom.de/events
Workshop-Reihe "Mobilisierung von Lotus Notes Applikationen" in Frankfurt (17.02.2005), Duesseldorf (24.02.2005) und Berlin (05.03.2005)
Anmeldung und Information unter http://lotus.xcom.de/events
*** XCOM AG Legal Disclaimer ***
Diese E-Mail einschliesslich ihrer Anhaenge ist vertraulich und ist allein fur den Gebrauch durch den vorgesehenen Empfaenger bestimmt. Dritten ist das Lesen, Verteilen oder Weiterleiten dieser E-Mail untersagt. Wir bitten, eine fehlgeleitete E-Mail unverzueglich vollstaendig zu loeschen und uns eine Nachricht zukommen zu lassen.
This email may contain material that is confidential and for the sole use of the intended recipient. Any review, distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Re: sounds like spellcheck [auf Viren geprueft]
Posted by Aad Nales <aa...@rotterdam-cs.com>.
Jonathan O'Connor wrote:
>Aad,
>Are you trying to check the spelling of English words by Dutch children?
>
>
Uh no, I am trying to correct the spelling of Dutch words by Dutch
children who, as most children do, make phonetic spelling mistakes.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: sounds like spellcheck [auf Viren geprueft]
Posted by Jonathan O'Connor <Jo...@xcom.de>.
Aad,
Are you trying to check the spelling of English words by Dutch children?
Then, Phonetix or any of these other solutions may not be perfect.
>From my little knowledge of Dutch, a "g" is some sort of velar fricative
(pronounced at the back of throat). And "ch" in english is also a velar
fricative.
You have to hope that the soundex/metaphone rules are broad enough to be
used by both languages.
Interesting little problem. No J2EE libraries to call, just static String
convertToSoundex(String word) to implement. Ah if only I could do more of
that sort of coding.
Ciao,
Jonathan O'Connor
XCOM Dublin
Kelvin Tan <ke...@relevanz.com>
09/02/2005 13:03
Please respond to
"Lucene Users List" <lu...@jakarta.apache.org>
To
Lucene Users List <lu...@jakarta.apache.org>
cc
Subject
Re: sounds like spellcheck [auf Viren geprueft]
Hey Aad, I believe
http://jakarta.apache.org/lucene/docs/contributions.html has a link to
Phonetix (
http://www.companywebstore.de/tangentum/mirror/en/products/phonetix/index.html
), an LGPL-licensed lib for phonetic algorithms like Soundex, Metaphone
and DoubleMetaphone. There are Lucene adapters.
As to the suitability of the algorithms, I haven't taken a look at the
Phonetix implementation, but if
http://spottedtiger.tripod.com/D_Language/D_DoubleMetaPhone.html is
anything to go by (do a search for "dutch"), then it should meet your
needs, or at least won't be difficult to customize.
Is that what you're looking for?
k
On Wed, 09 Feb 2005 13:23:57 +0100, Aad Nales wrote:
> In my Clipper days I could build an index on English words using a
> technique that was called soundex. Searching in that index resulted
> in hits of words that sounded the same. From what i remember this
> technique only worked for English. Has it ever been generalized?
>
> What i am trying to solve is this. A customer is looking for a
> solution to spelling mistakes made by children (upto 10) when
> typing in queries. The site is Dutch. Common mistakes are 'sgool'
> when searching for 'school'. The 'normal' spellcheckers and
> suggestors typically generate a list where the 'sounds like'
> candidates' are too far away from the result. So what I am thinking
> about doing is this:
>
> 1. create a parser that takes a word and creates a soundindex entry.
>
> 2. create list of 'correctly' spelled words either based on the
> index of the website or on some kind of dictionary.
> 2a. perhaps create a n-gram index based on these words
>
> 3. accept a query, figure out that a spelling mistake has been made
> 3a find alternatives by parsing the query and searching the 'sound
> like index' and then calculate and order the results
>
> Steps 2 and 3 have been discussed at length in this forum and have
> even made it to the sandbox. What I am left with is 1.
>
> My thinking is processing a series of replacement statements that
> go like: --
> g sounds like ch if the immediate predecessor is an s. o sounds
> like oo if the immediate predecessor is a consonant --
>
> But before I takes this to the next step I am wondering if anybody
> has created or thought up alternative solutions?
>
> Cheers,
> Aad
>
>
> --------------------------------------------------------------------
> - To unsubscribe, e-mail: lucene-user-
> unsubscribe@jakarta.apache.org For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
*** Aktuelle Veranstaltungen der XCOM AG ***
XCOM laedt ein zur IBM Workplace Roadshow in Frankfurt (16.02.2005), Duesseldorf (23.02.2005) und Berlin (02.03.2005)
Anmeldung und Information unter http://lotus.xcom.de/events
Workshop-Reihe "Mobilisierung von Lotus Notes Applikationen" in Frankfurt (17.02.2005), Duesseldorf (24.02.2005) und Berlin (05.03.2005)
Anmeldung und Information unter http://lotus.xcom.de/events
*** XCOM AG Legal Disclaimer ***
Diese E-Mail einschliesslich ihrer Anhaenge ist vertraulich und ist allein fur den Gebrauch durch den vorgesehenen Empfaenger bestimmt. Dritten ist das Lesen, Verteilen oder Weiterleiten dieser E-Mail untersagt. Wir bitten, eine fehlgeleitete E-Mail unverzueglich vollstaendig zu loeschen und uns eine Nachricht zukommen zu lassen.
This email may contain material that is confidential and for the sole use of the intended recipient. Any review, distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Re: sounds like spellcheck
Posted by Kelvin Tan <ke...@relevanz.com>.
Hey Aad, I believe http://jakarta.apache.org/lucene/docs/contributions.html has a link to Phonetix (http://www.companywebstore.de/tangentum/mirror/en/products/phonetix/index.html), an LGPL-licensed lib for phonetic algorithms like Soundex, Metaphone and DoubleMetaphone. There are Lucene adapters.
As to the suitability of the algorithms, I haven't taken a look at the Phonetix implementation, but if http://spottedtiger.tripod.com/D_Language/D_DoubleMetaPhone.html is anything to go by (do a search for "dutch"), then it should meet your needs, or at least won't be difficult to customize.
Is that what you're looking for?
k
On Wed, 09 Feb 2005 13:23:57 +0100, Aad Nales wrote:
> In my Clipper days I could build an index on English words using a
> technique that was called soundex. Searching in that index resulted
> in hits of words that sounded the same. From what i remember this
> technique only worked for English. Has it ever been generalized?
>
> What i am trying to solve is this. A customer is looking for a
> solution to spelling mistakes made by children (upto 10) when
> typing in queries. The site is Dutch. Common mistakes are 'sgool'
> when searching for 'school'. The 'normal' spellcheckers and
> suggestors typically generate a list where the 'sounds like'
> candidates' are too far away from the result. So what I am thinking
> about doing is this:
>
> 1. create a parser that takes a word and creates a soundindex entry.
>
> 2. create list of 'correctly' spelled words either based on the
> index of the website or on some kind of dictionary.
> 2a. perhaps create a n-gram index based on these words
>
> 3. accept a query, figure out that a spelling mistake has been made
> 3a find alternatives by parsing the query and searching the 'sound
> like index' and then calculate and order the results
>
> Steps 2 and 3 have been discussed at length in this forum and have
> even made it to the sandbox. What I am left with is 1.
>
> My thinking is processing a series of replacement statements that
> go like: --
> g sounds like ch if the immediate predecessor is an s. o sounds
> like oo if the immediate predecessor is a consonant --
>
> But before I takes this to the next step I am wondering if anybody
> has created or thought up alternative solutions?
>
> Cheers,
> Aad
>
>
> --------------------------------------------------------------------
> - To unsubscribe, e-mail: lucene-user-
> unsubscribe@jakarta.apache.org For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org