You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@lucene.apache.org by "Alan Woodward (JIRA)" <ji...@apache.org> on 2018/07/12 11:36:00 UTC

[jira] [Created] (LUCENE-8395) WordDelimiterGraphFilter can incorrectly add holes to a TokenStream

Alan Woodward created LUCENE-8395:
-------------------------------------

             Summary: WordDelimiterGraphFilter can incorrectly add holes to a TokenStream
                 Key: LUCENE-8395
                 URL: https://issues.apache.org/jira/browse/LUCENE-8395
             Project: Lucene - Core
          Issue Type: New Feature
            Reporter: Alan Woodward
            Assignee: Alan Woodward
         Attachments: LUCENE-8395.patch

If a token consists entirely of delimiter characters, then WordDelimiterGraphFilter will remove the token and insert a hole into the TokenStream.  However, it does this even if {{preserve_original}} is set, which results in an extra gap being added to the stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org