[jira] [Created] (OPENNLP-1947) TokenizerME changes keepNewLines on the shared WhitespaceTokenizer.INSTANCE

["Richard Zowalla (Jira)" <ji...@apache.org>]

Richard Zowalla created OPENNLP-1947:
----------------------------------------

             Summary: TokenizerME changes keepNewLines on the shared WhitespaceTokenizer.INSTANCE
                 Key: OPENNLP-1947
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1947
             Project: OpenNLP
          Issue Type: Bug
            Reporter: Richard Zowalla


{code:java}
// TokenizerME.java:195\-196
WhitespaceTokenizer whitespaceTokenizer = WhitespaceTokenizer.INSTANCE;
whitespaceTokenizer.setKeepNewLines\(keepNewLines\);
{code}

{{TokenizerME}} sets {{keepNewLines}} on the process\-wide singleton on every {{tokenizePos}} call.

_Consequences:_
\* After any {{TokenizerME}} with {{keepNewLines=true}} has run, every other user of {{WhitespaceTokenizer.INSTANCE}} gets {{\n}} and {{\r}} back as tokens. About 25 production callers are affected, e.g. {{NameSample.parse}}, {{POSSample.parse}}, {{TokenSample}}, {{DocumentSampleStream}}, {{SentimentSampleStream}}, the brat and OntoNotes readers, and the CLI tools.
\* {{TokenizerME}} is annotated {{@ThreadSafe}}, but two instances with different {{keepNewLines}} settings race on the same field.

_Repro:_ run a {{TokenizerME}} with {{setKeepNewLines\(true\)}}, then {{WhitespaceTokenizer.INSTANCE.tokenize\("a\nb"\)}} returns {{\[a, \n, b\]}}.

_Fix:_ use a local {{WhitespaceTokenizer}} in {{TokenizerME}}, or pass {{keepNewLines}} as a parameter. Longer term, make {{INSTANCE}} immutable \(deprecate {{setKeepNewLines}} on it\).

The open regex\-removal PRs add more callers of {{INSTANCE}} \(OPENNLP\-1928/1929/1930/1933\), so this should land first.





--
This message was sent by Atlassian Jira
(v8.20.10#820010)