You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Dyga, Adam" <ad...@beumergroup.com> on 2012/11/19 10:46:41 UTC

German 'ue' -> 'u' conversion

Hello,

I have two questin regarding handling German umlauts in Lucene:

1. I'm trying to find a way to convert German Umlauts written as 'ue', 'ae', etc to folded form 'u', 'a' and so on.
This is done by GermanAnalyzer (and German2StemFilter used by it), but unfortunately it also does stemming which is very undesired in my case.
Is there any other filter that can do only the 'ua' -> 'u' conversion?

2. Is there any filter that does 'ü' -> 'ue' (NOT 'u') conversion? What I'm trying to achieve is that word "über" should be found in the index whenever the user searches for " über" or "ueber" , but NOT "uber". 

Regards,
AD

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: German 'ue' -> 'u' conversion

Posted by Wouter Heijke <wh...@xs4all.nl>.
Hi,
We use a solution where we have our own implementation similar to
ASCIIFoldingFilter for German language specific characters (and also
French and Dutch).

Wouter

> Hello,
>
> I have two questin regarding handling German umlauts in Lucene:
>
> 1. I'm trying to find a way to convert German Umlauts written as 'ue',
> 'ae', etc to folded form 'u', 'a' and so on.
> This is done by GermanAnalyzer (and German2StemFilter used by it), but
> unfortunately it also does stemming which is very undesired in my case.
> Is there any other filter that can do only the 'ua' -> 'u' conversion?
>
> 2. Is there any filter that does 'ü' -> 'ue' (NOT 'u') conversion? What
> I'm trying to achieve is that word "über" should be found in the index
> whenever the user searches for " über" or "ueber" , but NOT "uber".
>
> Regards,
> AD
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: German 'ue' -> 'u' conversion

Posted by "Dyga, Adam" <ad...@beumergroup.com>.
Yes, that would solve my question 2. I can convert all umlauts to 'ue', 'ae', etc form before the tokens get to other filters and it should work fine.

Thanks,
Adam


-----Original Message-----
From: Igal @ getRailo.org [mailto:igal@getrailo.org] 
Sent: 19 listopada 2012 11:15
To: java-user@lucene.apache.org
Subject: Re: German 'ue' -> 'u' conversion

if your needs are so specific -- you can always build a NormalizeCharMap and use MappingCharFilter


Igal


On 11/19/2012 2:11 AM, Dyga, Adam wrote:
> I did, but none of them can do it (at least in default configuration).
>
> Regards,
> AD
>
> -----Original Message-----
> From: Igal @ getRailo.org [mailto:igal@getrailo.org]
> Sent: 19 listopada 2012 11:10
> To: java-user@lucene.apache.org
> Subject: Re: German 'ue' -> 'u' conversion
>
> look for filters that use the ICU4J library
>
>
> On 11/19/2012 2:08 AM, Lutz Fechner wrote:
>> Hi,
>>
>> we use a modified ISOLatin1AccentFilter bit to replace German accents by ae, oe, ue and so on for that purpose.
>>
>> In the code you will see a switch for the characters.
>>
>>
>> You need to change it from
>>
>> case '\u00E4' : // small ä
>>             output[outputPos++] = 'a';
>>             output[outputPos++] = 'e';
>>             break;
>>
>> To something like this
>>
>> case '\u00E4' : // small ä
>>             output[outputPos++] = 'a';
>> 	    break;
>>
>> for the characters you want to replace.
>>
>>
>> Best Regards
>>
>> Lutz Fechner
>>
>>
>>
>>
>> -----Original Message-----
>> From: Dyga, Adam [mailto:adam.dyga@beumergroup.com]
>> Sent: Montag, 19. November 2012 10:47
>> To: java-user@lucene.apache.org
>> Subject: German 'ue' -> 'u' conversion
>>
>> Hello,
>>
>> I have two questin regarding handling German umlauts in Lucene:
>>
>> 1. I'm trying to find a way to convert German Umlauts written as 'ue', 'ae', etc to folded form 'u', 'a' and so on.
>> This is done by GermanAnalyzer (and German2StemFilter used by it), but unfortunately it also does stemming which is very undesired in my case.
>> Is there any other filter that can do only the 'ua' -> 'u' conversion?
>>
>> 2. Is there any filter that does 'ü' -> 'ue' (NOT 'u') conversion? What I'm trying to achieve is that word "über" should be found in the index whenever the user searches for " über" or "ueber" , but NOT "uber".
>>
>> Regards,
>> AD
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: German 'ue' -> 'u' conversion

Posted by "Igal @ getRailo.org" <ig...@getrailo.org>.
if your needs are so specific -- you can always build a NormalizeCharMap 
and use MappingCharFilter


Igal


On 11/19/2012 2:11 AM, Dyga, Adam wrote:
> I did, but none of them can do it (at least in default configuration).
>
> Regards,
> AD
>
> -----Original Message-----
> From: Igal @ getRailo.org [mailto:igal@getrailo.org]
> Sent: 19 listopada 2012 11:10
> To: java-user@lucene.apache.org
> Subject: Re: German 'ue' -> 'u' conversion
>
> look for filters that use the ICU4J library
>
>
> On 11/19/2012 2:08 AM, Lutz Fechner wrote:
>> Hi,
>>
>> we use a modified ISOLatin1AccentFilter bit to replace German accents by ae, oe, ue and so on for that purpose.
>>
>> In the code you will see a switch for the characters.
>>
>>
>> You need to change it from
>>
>> case '\u00E4' : // small ä
>>             output[outputPos++] = 'a';
>>             output[outputPos++] = 'e';
>>             break;
>>
>> To something like this
>>
>> case '\u00E4' : // small ä
>>             output[outputPos++] = 'a';
>> 	    break;
>>
>> for the characters you want to replace.
>>
>>
>> Best Regards
>>
>> Lutz Fechner
>>
>>
>>
>>
>> -----Original Message-----
>> From: Dyga, Adam [mailto:adam.dyga@beumergroup.com]
>> Sent: Montag, 19. November 2012 10:47
>> To: java-user@lucene.apache.org
>> Subject: German 'ue' -> 'u' conversion
>>
>> Hello,
>>
>> I have two questin regarding handling German umlauts in Lucene:
>>
>> 1. I'm trying to find a way to convert German Umlauts written as 'ue', 'ae', etc to folded form 'u', 'a' and so on.
>> This is done by GermanAnalyzer (and German2StemFilter used by it), but unfortunately it also does stemming which is very undesired in my case.
>> Is there any other filter that can do only the 'ua' -> 'u' conversion?
>>
>> 2. Is there any filter that does 'ü' -> 'ue' (NOT 'u') conversion? What I'm trying to achieve is that word "über" should be found in the index whenever the user searches for " über" or "ueber" , but NOT "uber".
>>
>> Regards,
>> AD
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: German 'ue' -> 'u' conversion

Posted by "Dyga, Adam" <ad...@beumergroup.com>.
I did, but none of them can do it (at least in default configuration).

Regards,
AD

-----Original Message-----
From: Igal @ getRailo.org [mailto:igal@getrailo.org] 
Sent: 19 listopada 2012 11:10
To: java-user@lucene.apache.org
Subject: Re: German 'ue' -> 'u' conversion

look for filters that use the ICU4J library


On 11/19/2012 2:08 AM, Lutz Fechner wrote:
> Hi,
>
> we use a modified ISOLatin1AccentFilter bit to replace German accents by ae, oe, ue and so on for that purpose.
>
> In the code you will see a switch for the characters.
>
>
> You need to change it from
>
> case '\u00E4' : // small ä
>            output[outputPos++] = 'a';
>            output[outputPos++] = 'e';
>            break;
>
> To something like this
>
> case '\u00E4' : // small ä
>            output[outputPos++] = 'a';
> 	    break;
>
> for the characters you want to replace.
>
>
> Best Regards
>
> Lutz Fechner
>
>
>
>
> -----Original Message-----
> From: Dyga, Adam [mailto:adam.dyga@beumergroup.com]
> Sent: Montag, 19. November 2012 10:47
> To: java-user@lucene.apache.org
> Subject: German 'ue' -> 'u' conversion
>
> Hello,
>
> I have two questin regarding handling German umlauts in Lucene:
>
> 1. I'm trying to find a way to convert German Umlauts written as 'ue', 'ae', etc to folded form 'u', 'a' and so on.
> This is done by GermanAnalyzer (and German2StemFilter used by it), but unfortunately it also does stemming which is very undesired in my case.
> Is there any other filter that can do only the 'ua' -> 'u' conversion?
>
> 2. Is there any filter that does 'ü' -> 'ue' (NOT 'u') conversion? What I'm trying to achieve is that word "über" should be found in the index whenever the user searches for " über" or "ueber" , but NOT "uber".
>
> Regards,
> AD
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

-- 
Igal Sapir
Railo - Open Source CFML Engine
http://getrailo.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: German 'ue' -> 'u' conversion

Posted by "Igal @ getRailo.org" <ig...@getrailo.org>.
look for filters that use the ICU4J library


On 11/19/2012 2:08 AM, Lutz Fechner wrote:
> Hi,
>
> we use a modified ISOLatin1AccentFilter bit to replace German accents by ae, oe, ue and so on for that purpose.
>
> In the code you will see a switch for the characters.
>
>
> You need to change it from
>
> case '\u00E4' : // small ä
>            output[outputPos++] = 'a';
>            output[outputPos++] = 'e';
>            break;
>
> To something like this
>
> case '\u00E4' : // small ä
>            output[outputPos++] = 'a';
> 	    break;
>
> for the characters you want to replace.
>
>
> Best Regards
>
> Lutz Fechner
>
>
>
>
> -----Original Message-----
> From: Dyga, Adam [mailto:adam.dyga@beumergroup.com]
> Sent: Montag, 19. November 2012 10:47
> To: java-user@lucene.apache.org
> Subject: German 'ue' -> 'u' conversion
>
> Hello,
>
> I have two questin regarding handling German umlauts in Lucene:
>
> 1. I'm trying to find a way to convert German Umlauts written as 'ue', 'ae', etc to folded form 'u', 'a' and so on.
> This is done by GermanAnalyzer (and German2StemFilter used by it), but unfortunately it also does stemming which is very undesired in my case.
> Is there any other filter that can do only the 'ua' -> 'u' conversion?
>
> 2. Is there any filter that does 'ü' -> 'ue' (NOT 'u') conversion? What I'm trying to achieve is that word "über" should be found in the index whenever the user searches for " über" or "ueber" , but NOT "uber".
>
> Regards,
> AD
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

-- 
Igal Sapir
Railo - Open Source CFML Engine
http://getrailo.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: German 'ue' -> 'u' conversion

Posted by Lutz Fechner <LF...@hubwoo.com>.
Hi,

we use a modified ISOLatin1AccentFilter bit to replace German accents by ae, oe, ue and so on for that purpose.

In the code you will see a switch for the characters.


You need to change it from 

case '\u00E4' : // small ä
          output[outputPos++] = 'a';
          output[outputPos++] = 'e';
          break;

To something like this

case '\u00E4' : // small ä
          output[outputPos++] = 'a';          
	    break;

for the characters you want to replace.


Best Regards

Lutz Fechner




-----Original Message-----
From: Dyga, Adam [mailto:adam.dyga@beumergroup.com] 
Sent: Montag, 19. November 2012 10:47
To: java-user@lucene.apache.org
Subject: German 'ue' -> 'u' conversion

Hello,

I have two questin regarding handling German umlauts in Lucene:

1. I'm trying to find a way to convert German Umlauts written as 'ue', 'ae', etc to folded form 'u', 'a' and so on.
This is done by GermanAnalyzer (and German2StemFilter used by it), but unfortunately it also does stemming which is very undesired in my case.
Is there any other filter that can do only the 'ua' -> 'u' conversion?

2. Is there any filter that does 'ü' -> 'ue' (NOT 'u') conversion? What I'm trying to achieve is that word "über" should be found in the index whenever the user searches for " über" or "ueber" , but NOT "uber". 

Regards,
AD

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org