You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Lespiau (JIRA)" <ji...@apache.org> on 2016/01/28 23:00:41 UTC
[jira] [Comment Edited] (SOLR-8606) Duplicate tokens using
WordDelimiterFilter for a specific configuration
[ https://issues.apache.org/jira/browse/SOLR-8606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122373#comment-15122373 ]
Lespiau edited comment on SOLR-8606 at 1/28/16 10:00 PM:
---------------------------------------------------------
Indeed, there is no incorrect match.
You can move this issue to its correct place. I didn't know there was a different place for Lucene.
The resulting token for "abcDef" are :
"abcDef", "abc", "abcDef", "Def"
The 2 "abcDef" are the same, except for the "positionIncrement", since the first one degins a new word, and the second one does not.
termAtt=abcDef, startOffset=0, endOffset=6, typeAtt=word,
positionIncrement=1, positionLength=1, keywordAtt=null]
termAtt=abcDef, startOffset=0, endOffset=6, typeAtt=word,
positionIncrement=0, positionLength=1, keywordAtt=null
What do you mean by the same *term* position ? There is no *term* attribute for me in a token.
was (Author: gueust):
Indeed, there is no incorrect match.
You can move this issue to its correct place. I didn't know there was a different placefor Lucene.
The resulting token for "abcDef" are :
"abcDef", "abc", "abcDef", "Def"
The 2 "abcDef" are the same, except for the "positionIncrement", since the first one degins a new word, and the second one does not.
termAtt=abcDef, startOffset=0, endOffset=6, typeAtt=word,
positionIncrement=1, positionLength=1, keywordAtt=null]
termAtt=abcDef, startOffset=0, endOffset=6, typeAtt=word,
positionIncrement=0, positionLength=1, keywordAtt=null
What do you mean by the same *term* position ? There is no *term* attribute for me in a token.
> Duplicate tokens using WordDelimiterFilter for a specific configuration
> -----------------------------------------------------------------------
>
> Key: SOLR-8606
> URL: https://issues.apache.org/jira/browse/SOLR-8606
> Project: Solr
> Issue Type: Bug
> Reporter: Lespiau
> Priority: Minor
> Attachments: SOLR-8686-TEST.patch
>
>
> When using both the options PRESERVE_ORIGINAL| SPLIT_ON_CASE_CHANGE and CONCATENATE_ALL|CATENATE_WORDS using the WordDelimiterFilter, we have duplicate tokens on strings contaning only case changes.
> When using the SPLIT_ON_CASE_CHANGE option, "abcDef" is split into "abc", "Def".
> When having PRESERVE_ORIGINAL, we keep "abcDef".
> However, when one uses CONCATENATE_ALL or CATENATE_WORDS, it also adds an other token built from the concatenation of the splited words, giving "abcDef" again.
> I'm not 100% certain that token filters should not produce duplicate tokens (same word, same start and end positions). Can someone confirm this is a bug ?
> I supply a patch that gives a test explosing the incorrect behavior.
> I'm willing to work on the following days to fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org