You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jean-Baptiste Lespiau (JIRA)" <ji...@apache.org> on 2016/04/26 23:38:12 UTC
[jira] [Resolved] (LUCENE-7004) Duplicate tokens using
WordDelimiterFilter for a specific configuration
[ https://issues.apache.org/jira/browse/LUCENE-7004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Baptiste Lespiau resolved LUCENE-7004.
-------------------------------------------
Resolution: Implemented
Change to status to implemented, hoping someone will review it.
Be careful that LUCENE-7003 is a dependency.
> Duplicate tokens using WordDelimiterFilter for a specific configuration
> -----------------------------------------------------------------------
>
> Key: LUCENE-7004
> URL: https://issues.apache.org/jira/browse/LUCENE-7004
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Jean-Baptiste Lespiau
> Priority: Minor
> Attachments: FIX-LUCENE-7004.PATCH, TEST-LUCENE-7004.PATCH, wdf-analysis.png
>
>
> When using both the options PRESERVE_ORIGINAL|SPLIT_ON_CASE_CHANGE|CONCATENATE_ALL using the WordDelimiterFilter, we have duplicate tokens on strings contaning only case changes.
> When using the SPLIT_ON_CASE_CHANGE option, "abcDef" is split into "abc", "Def".
> When having PRESERVE_ORIGINAL, we keep "abcDef".
> However, when one uses CONCATENATE_ALL (or CATENATE_WORDS ?), it also adds another token built from the concatenation of the splited words, giving "abcDef" again.
> I'm not 100% certain that token filters should not produce duplicate tokens (same word, same start and end positions). Can someone confirm this is a bug ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org