You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Sebastian Kirsch (JIRA)" <ji...@apache.org> on 2006/08/07 23:16:15 UTC
[jira] Commented: (LUCENE-400) NGramFilter -- construct n-grams
from a TokenStream
[ http://issues.apache.org/jira/browse/LUCENE-400?page=comments#action_12426327 ]
Sebastian Kirsch commented on LUCENE-400:
-----------------------------------------
Hi Otis,
I did not figure out the problem. Getting rid of Commons Collection should be no problem; I am just using them as FIFOs. However, I do not have the time at the moment to implement this.
Kind regards, Sebastian
> NGramFilter -- construct n-grams from a TokenStream
> ---------------------------------------------------
>
> Key: LUCENE-400
> URL: http://issues.apache.org/jira/browse/LUCENE-400
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: unspecified
> Environment: Operating System: All
> Platform: All
> Reporter: Sebastian Kirsch
> Assigned To: Lucene Developers
> Priority: Minor
> Attachments: NGramAnalyzerWrapper.java, NGramAnalyzerWrapperTest.java, NGramFilter.java, NGramFilterTest.java
>
>
> This filter constructs n-grams (token combinations up to a fixed size, sometimes
> called "shingles") from a token stream.
> The filter sets start offsets, end offsets and position increments, so
> highlighting and phrase queries should work.
> Position increments > 1 in the input stream are replaced by filler tokens
> (tokens with termText "_" and endOffset - startOffset = 0) in the output
> n-grams. (Position increments > 1 in the input stream are usually caused by
> removing some tokens, eg. stopwords, from a stream.)
> The filter uses CircularFifoBuffer and UnboundedFifoBuffer from Apache
> Commons-Collections.
> Filter, test case and an analyzer are attached.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org