You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michael Sokolov <ms...@gmail.com> on 2019/04/16 18:08:00 UTC
umlauts / diacritic expansion
I'm learning how to index/search German today and understanding that
vowels with umlauts are conventionally expanded into two ASCII
characters, eg "für" -> "fuer", so people may search for the expanded
form "fuer", but they might also search with the diacritic, and
finally they might lazily search using the stripped form "fur".
My question: is there a standard CharFilter or TokenFilter that
expands to both (ASCII) forms, for characters with umlauts and perhaps
other diacritics I might be unaware of in other languages having
similar multiple renderings in ASCII?
-Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: umlauts / diacritic expansion
Posted by Michael Sokolov <ms...@gmail.com>.
Right, AsciiFoldingFilter seems to map Ü [LATIN CAPITAL LETTER U
WITH DIAERESIS] to "U" not "UE".
On Wed, Apr 17, 2019 at 12:26 AM Ralf Heyde <ra...@gmx.de> wrote:
>
> Ah sorry, Asciifolding for umlauts will result in ue/ae - ß/ss etc
>
> You could allow a distance of 1 or 2 given you use levenshtein distance - this might be close to what you need.
>
> Von meinem iPhone gesendet
>
> > Am 16.04.2019 um 20:08 schrieb Michael Sokolov <ms...@gmail.com>:
> >
> > I'm learning how to index/search German today and understanding that
> > vowels with umlauts are conventionally expanded into two ASCII
> > characters, eg "für" -> "fuer", so people may search for the expanded
> > form "fuer", but they might also search with the diacritic, and
> > finally they might lazily search using the stripped form "fur".
> >
> > My question: is there a standard CharFilter or TokenFilter that
> > expands to both (ASCII) forms, for characters with umlauts and perhaps
> > other diacritics I might be unaware of in other languages having
> > similar multiple renderings in ASCII?
> >
> > -Mike
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: umlauts / diacritic expansion
Posted by Ralf Heyde <ra...@gmx.de>.
Ah sorry, Asciifolding for umlauts will result in ue/ae - ß/ss etc
You could allow a distance of 1 or 2 given you use levenshtein distance - this might be close to what you need.
Von meinem iPhone gesendet
> Am 16.04.2019 um 20:08 schrieb Michael Sokolov <ms...@gmail.com>:
>
> I'm learning how to index/search German today and understanding that
> vowels with umlauts are conventionally expanded into two ASCII
> characters, eg "für" -> "fuer", so people may search for the expanded
> form "fuer", but they might also search with the diacritic, and
> finally they might lazily search using the stripped form "fur".
>
> My question: is there a standard CharFilter or TokenFilter that
> expands to both (ASCII) forms, for characters with umlauts and perhaps
> other diacritics I might be unaware of in other languages having
> similar multiple renderings in ASCII?
>
> -Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: umlauts / diacritic expansion
Posted by Ralf Heyde <ra...@gmx.de>.
Hey,
Take a look at Asciifoldingfilter - this one is quite generic.
Does this answer your question?
Cheers Ralf
Von meinem iPhone gesendet
> Am 16.04.2019 um 20:08 schrieb Michael Sokolov <ms...@gmail.com>:
>
> I'm learning how to index/search German today and understanding that
> vowels with umlauts are conventionally expanded into two ASCII
> characters, eg "für" -> "fuer", so people may search for the expanded
> form "fuer", but they might also search with the diacritic, and
> finally they might lazily search using the stripped form "fur".
>
> My question: is there a standard CharFilter or TokenFilter that
> expands to both (ASCII) forms, for characters with umlauts and perhaps
> other diacritics I might be unaware of in other languages having
> similar multiple renderings in ASCII?
>
> -Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org