[jira] [Created] (OPENNLP-1929) BasicContextGenerator splits on the separator as a regular expression

["Kristian Rickert (Jira)" <ji...@apache.org>]

Kristian Rickert created OPENNLP-1929:
-----------------------------------------

             Summary: BasicContextGenerator splits on the separator as a regular expression
                 Key: OPENNLP-1929
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1929
             Project: OpenNLP
          Issue Type: Bug
          Components: Machine Learning
            Reporter: Kristian Rickert
            Assignee: Kristian Rickert


{{BasicContextGenerator(String sep)}} documents its argument as a "separator character", but {{getContext}} calls {{String.split(separator)}}, which compiles the argument as a regular expression on every call.

Consequences for a caller who follows the Javadoc:
* {{"|"}}, {{"."}}, {{"+"}}, {{"*"}}, {{"?"}} split on every character or on nothing.
* {{"("}} or {{"["}} throw {{PatternSyntaxException}} at prediction time, not at construction.
* An empty separator splits the input into single characters.
* If an application takes the separator from configuration, a crafted pattern is re-evaluated on every context string, which is a backtracking exposure.

No code in the repository calls the separator constructor; it arrived with the maxent modularization (OPENNLP-1740) and has no test.

Fix: split on the literal separator with a cursor scan, keeping {{String.split}} trailing-empty semantics; reject an empty separator with {{IllegalArgumentException}} at construction; state in the Javadoc that the separator is literal. Tests: default space, a metacharacter separator, a multi-character separator, leading and trailing separators, empty and null separator.

Compatibility: external code that intentionally passes a pattern such as {{\s+}} relied on undocumented behavior; 3.0 is the release to correct it.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)