You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@lucene.apache.org by "Alan Woodward (JIRA)" <ji...@apache.org> on 2019/03/11 12:44:00 UTC

[jira] [Created] (LUCENE-8719) FixedShingleFilter can miss paths at the end of a TokenStream

Alan Woodward created LUCENE-8719:
-------------------------------------

             Summary: FixedShingleFilter can miss paths at the end of a TokenStream
                 Key: LUCENE-8719
                 URL: https://issues.apache.org/jira/browse/LUCENE-8719
             Project: Lucene - Core
          Issue Type: Bug
            Reporter: Alan Woodward
            Assignee: Alan Woodward


If we have a set of paths of differing lengths at the end of a TokenStream, FixedShingleFilter will currently stop when the first path hits the end of the stream, but this may miss extra longer paths.  For example, the stream "a b:3/c d e f" with a shingle size of 3 should produce paths "a b f", "a c d", "c d e" and "d e f", but at the moment FixedShingleFilter will stop when it finds that the path "b f" is too short to produce a shingle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org