You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michael Sokolov <ms...@gmail.com> on 2019/04/16 18:08:00 UTC

umlauts / diacritic expansion

I'm learning how to index/search German today and understanding that
vowels with umlauts are conventionally expanded into two ASCII
characters, eg  "für" -> "fuer", so people may search for the expanded
form "fuer", but they might also search with the diacritic, and
finally they might lazily search using the stripped form "fur".

My question: is there a standard CharFilter or TokenFilter that
expands to both (ASCII) forms, for characters with umlauts and perhaps
other diacritics I might be unaware of in other languages having
similar multiple renderings in ASCII?

-Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: umlauts / diacritic expansion

Posted by Michael Sokolov <ms...@gmail.com>.
Right, AsciiFoldingFilter seems to map  Ü  [LATIN CAPITAL LETTER U
WITH DIAERESIS] to "U" not "UE".

On Wed, Apr 17, 2019 at 12:26 AM Ralf Heyde <ra...@gmx.de> wrote:
>
> Ah sorry, Asciifolding for umlauts will result in ue/ae - ß/ss etc
>
> You could allow a distance of 1 or 2 given you use levenshtein distance - this might be close to what you need.
>
> Von meinem iPhone gesendet
>
> > Am 16.04.2019 um 20:08 schrieb Michael Sokolov <ms...@gmail.com>:
> >
> > I'm learning how to index/search German today and understanding that
> > vowels with umlauts are conventionally expanded into two ASCII
> > characters, eg  "für" -> "fuer", so people may search for the expanded
> > form "fuer", but they might also search with the diacritic, and
> > finally they might lazily search using the stripped form "fur".
> >
> > My question: is there a standard CharFilter or TokenFilter that
> > expands to both (ASCII) forms, for characters with umlauts and perhaps
> > other diacritics I might be unaware of in other languages having
> > similar multiple renderings in ASCII?
> >
> > -Mike
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: umlauts / diacritic expansion

Posted by Ralf Heyde <ra...@gmx.de>.
Ah sorry, Asciifolding for umlauts will result in ue/ae - ß/ss etc

You could allow a distance of 1 or 2 given you use levenshtein distance - this might be close to what you need. 

Von meinem iPhone gesendet

> Am 16.04.2019 um 20:08 schrieb Michael Sokolov <ms...@gmail.com>:
> 
> I'm learning how to index/search German today and understanding that
> vowels with umlauts are conventionally expanded into two ASCII
> characters, eg  "für" -> "fuer", so people may search for the expanded
> form "fuer", but they might also search with the diacritic, and
> finally they might lazily search using the stripped form "fur".
> 
> My question: is there a standard CharFilter or TokenFilter that
> expands to both (ASCII) forms, for characters with umlauts and perhaps
> other diacritics I might be unaware of in other languages having
> similar multiple renderings in ASCII?
> 
> -Mike
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: umlauts / diacritic expansion

Posted by Ralf Heyde <ra...@gmx.de>.
Hey,

Take a look at Asciifoldingfilter - this one is quite generic.

Does this answer your question?

Cheers Ralf

Von meinem iPhone gesendet

> Am 16.04.2019 um 20:08 schrieb Michael Sokolov <ms...@gmail.com>:
> 
> I'm learning how to index/search German today and understanding that
> vowels with umlauts are conventionally expanded into two ASCII
> characters, eg  "für" -> "fuer", so people may search for the expanded
> form "fuer", but they might also search with the diacritic, and
> finally they might lazily search using the stripped form "fur".
> 
> My question: is there a standard CharFilter or TokenFilter that
> expands to both (ASCII) forms, for characters with umlauts and perhaps
> other diacritics I might be unaware of in other languages having
> similar multiple renderings in ASCII?
> 
> -Mike
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org