[jira] [Created] (LUCENE-9006) Ensure WordDelimiterGraphFilter always emits catenateAll token early

["David Wayne Smiley (Jira)" <ji...@apache.org>]

David Wayne Smiley created LUCENE-9006:
------------------------------------------

             Summary: Ensure WordDelimiterGraphFilter always emits catenateAll token early
                 Key: LUCENE-9006
                 URL: https://issues.apache.org/jira/browse/LUCENE-9006
             Project: Lucene - Core
          Issue Type: Improvement
          Components: modules/analysis
            Reporter: David Wayne Smiley
            Assignee: David Wayne Smiley


Ideally, the first token of WDGF is the preserveOriginal (if configured to emit), and the second should be the catenateAll (if configured to emit).  The deprecated WDF does this but WDGF can sometimes put the first other token earlier when there is a non-emitted candidate sub-token.

Example input "8-other" when only generateWordParts and catenateAll -- *not* generateNumberParts.  WDGF internally sees the '8' but moves on.  Ultimately, the "other" token and the catenated "8other" will appear at the same internal position, which by luck fools the sorter to emit "other" first.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org