You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Adrien Grand (JIRA)" <ji...@apache.org> on 2013/04/27 00:18:17 UTC
[jira] [Resolved] (LUCENE-2947) NGramTokenizer shouldn't trim
whitespace
[ https://issues.apache.org/jira/browse/LUCENE-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrien Grand resolved LUCENE-2947.
----------------------------------
Resolution: Fixed
NGramTokenizer doesn't trim whitespaces anymore (LUCENE-4955).
> NGramTokenizer shouldn't trim whitespace
> ----------------------------------------
>
> Key: LUCENE-2947
> URL: https://issues.apache.org/jira/browse/LUCENE-2947
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
> Affects Versions: 3.0.3
> Reporter: David Byrne
> Priority: Minor
> Attachments: LUCENE-2947.patch, NGramTokenizerTest.java
>
>
> Before I tokenize my strings, I am padding them with white space:
> String foobar = " " + foo + " " + bar + " ";
> When constructing term vectors from ngrams, this strategy has a couple benefits. First, it places special emphasis on the starting and ending of a word. Second, it improves the similarity between phrases with swapped words. " foo bar " matches " bar foo " more closely than "foo bar" matches "bar foo".
> The problem is that Lucene's NGramTokenizer trims whitespace. This forces me to do some preprocessing on my strings before I can tokenize them:
> foobar.replaceAll(" ","$"); //arbitrary char not in my data
> This is undocumented, so users won't realize their strings are being trim()'ed, unless they look through the source, or examine the tokens manually.
> I am proposing NGramTokenizer should be changed to respect whitespace. Is there a compelling reason against this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org