[jira] [Created] (LUCENE-6874) WhitespaceTokenizer should tokenize on NBSP

["David Smiley (JIRA)" <ji...@apache.org>]

David Smiley created LUCENE-6874:
------------------------------------

             Summary: WhitespaceTokenizer should tokenize on NBSP
                 Key: LUCENE-6874
                 URL: https://issues.apache.org/jira/browse/LUCENE-6874
             Project: Lucene - Core
          Issue Type: Improvement
          Components: modules/analysis
            Reporter: David Smiley
            Priority: Minor


WhitespaceTokenizer uses [Character.isWhitespace |http://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-int-] to decide what is whitespace.  Here's a pertinent excerpt:

bq. It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', '\u2007', '\u202F')

Perhaps Character.isWhitespace should have been called isLineBreakableWhitespace?

I think WhitespaceTokenizer should tokenize on this.  I am aware it's easy to work around but why leave this trap in by default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org