You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Christian Winkler (JIRA)" <ji...@apache.org> on 2015/09/18 17:33:05 UTC
[jira] [Updated] (LUCENE-6809) DictionaryCompoundWordTokenFilter
should respect minSubwordSize also for fragments
[ https://issues.apache.org/jira/browse/LUCENE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Winkler updated LUCENE-6809:
--------------------------------------
Summary: DictionaryCompoundWordTokenFilter should respect minSubwordSize also for fragments (was: {{DictionaryCompoundWordTokenFilter}} should respect minSubwordSize also for fragments)
> DictionaryCompoundWordTokenFilter should respect minSubwordSize also for fragments
> ----------------------------------------------------------------------------------
>
> Key: LUCENE-6809
> URL: https://issues.apache.org/jira/browse/LUCENE-6809
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 5.3, Trunk
> Reporter: Christian Winkler
>
> {{DictionaryCompoundWordTokenFilter}} is very useful for building German search indices. However this can also lead to ambiguities as words might be extracted which have a completely different meaning. Most often this happens if the remaining parts of the word are too small.
> Example: {{schwein}} (German for pig) contains {{wein}} (German for wine). Even if {{minSubwordSize}} is set to {{4}}, {{wein}} gets extracted though {{sch}} is shorter than 4 characters.
> We could solve this by requiring all word parts to be part of the dictionary (at most 3), but this creates problems with compound words of more than three nouns.
> Therefore we have built an alternate solution where {{minSubwordSize}} is also applied to the rest of the fragments. We have tested this in several (large) customer indices and it is working much better than before.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org