[jira] Reopened: (LUCENE-759) Add n-gram tokenizers to contrib/analyzers

["Otis Gospodnetic (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Otis Gospodnetic reopened LUCENE-759:
-------------------------------------

    Lucene Fields: [New, Patch Available]  (was: [New])

Reopening, because I'm bringing in Adam Hiatt's modifications that he uploaded in a patch for SOLR-81.  Adam's changes allow this tokenizer to create n-grams whose sizes are specified as a min-max range.

This patch fixes a bug in Adam's code, but has another bug that I don't know how to fix now.
Adam's bug:
  input: abcde
  minGram: 1
  maxGram: 3
  output: a ab abc  -- and this is where tokenizing stopped, which was wrong, it should have continued: b bc bcd c cd cde d de e

Otis' bug:
  input: abcde
  minGeam: 1
  maxGram: 3
  output: e de cde d cd bcd c bc abc b ab -- and this is where tokenizing stops, which is wrong, it should generate one more n-gram: a

This bug won't hurt SOLR-81, but it should be fixed.

> Add n-gram tokenizers to contrib/analyzers
> ------------------------------------------
>
>                 Key: LUCENE-759
>                 URL: https://issues.apache.org/jira/browse/LUCENE-759
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>            Reporter: Otis Gospodnetic
>            Priority: Minor
>         Attachments: LUCENE-759.patch
>
>
> It would be nice to have some n-gram-capable tokenizers in contrib/analyzers.  Patch coming shortly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org