[jira] [Created] (LUCENE-5674) A new token filter: SubSequence

["Nitzan Shaked (JIRA)" <ji...@apache.org>]

Nitzan Shaked created LUCENE-5674:
-------------------------------------

             Summary: A new token filter: SubSequence
                 Key: LUCENE-5674
                 URL: https://issues.apache.org/jira/browse/LUCENE-5674
             Project: Lucene - Core
          Issue Type: Improvement
          Components: core/other
            Reporter: Nitzan Shaked
            Priority: Minor
         Attachments: 0001-SubSeqFilter.patch

A new configurable token filter which, given a token breaks it into sub-parts and outputs consecutive sub-sequences of those sub-parts.

Useful for, for example, using during indexing to generate variations on domain names, so that "www.google.com" can be found by searching for "google.com", or "www.google.com".

Parameters:

sepRegexp: A regular expression used split incoming tokens into sub-parts.
glue: A string used to concatenate sub-parts together when creating sub-sequences.
minLen: Minimum length (in sub-parts) of output sub-sequences
maxLen: Maximum length (in sub-parts) of output sub-sequences (0 for unlimited; negative numbers for token length in sub-parts minus specified length)
anchor: Anchor.START to output only prefixes, or Anchor.END to output only suffixes, or Anchor.NONE to output any sub-sequence
withOriginal: whether to output also the original token

Note: tests will follow, I currently believe BaseTokenStreamTestCase is broken/mis-designed for such use cases, I will open a separate Jira for that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org