You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "David Smiley (Jira)" <ji...@apache.org> on 2020/09/28 12:24:00 UTC
[jira] [Commented] (LUCENE-9458) WordDelimiterGraphFilter (and
non-graph) should tie-break order using end offset
[ https://issues.apache.org/jira/browse/LUCENE-9458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17203188#comment-17203188 ]
David Smiley commented on LUCENE-9458:
--------------------------------------
In the updated PR, I only changed WDGF, reverting my attempt to fix WDF.
> WordDelimiterGraphFilter (and non-graph) should tie-break order using end offset
> --------------------------------------------------------------------------------
>
> Key: LUCENE-9458
> URL: https://issues.apache.org/jira/browse/LUCENE-9458
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/analysis
> Reporter: David Smiley
> Assignee: David Smiley
> Priority: Minor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> WordDelimiterGraphFilter and WordDelimiterFilter do not consult the end offset in their sub-token _ordering_. In the event of a tie-break, I propose the longer token come first. This usually happens already, but not always, and so this also feels like an inconsistency when you see it. This issue can be thought of as a bug fix to LUCENE-9006 or an improvement; I have no strong feelings on the issue classification. Before reading further, definitely read that issue.
> I see this is a problem when using CATENATE_ALL with either GENERATE_WORD_PARTS xor GENERATE_NUMBER_PARTS when the input ends with that part not being generated. Consider the input: "other-9" and let's assume we want to catenate all, generate word parts, but nothing else (not numbers). This should be tokenized in this order: "other9", "other" but today is emitted in reverse order.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org