[jira] [Created] (OPENNLP-1189) Token model creation fails without at least one tag

["Jeff Zemerick (JIRA)" <ji...@apache.org>]

Jeff Zemerick created OPENNLP-1189:
--------------------------------------

             Summary: Token model creation fails without at least one <SPLIT> tag
                 Key: OPENNLP-1189
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1189
             Project: OpenNLP
          Issue Type: Bug
          Components: Tokenizer
    Affects Versions: 1.8.4
            Reporter: Jeff Zemerick


The tokenizer training documentation for 1.8.4 states that "Tokens are either separated by a whitespace or by a special <SPLIT> tag." However, it appears that training files if the training data does not contain at least one <SPLIT> tag. To reproduce:

Training on the sample data works fine:

Pierre Vinken<SPLIT>, 61 years old<SPLIT>, will join the board as a nonexecutive director Nov. 29<SPLIT>.
Mr. Vinken is chairman of Elsevier N.V.<SPLIT>, the Dutch publishing group<SPLIT>.
Rudolph Agnew<SPLIT>, 55 years old and former chairman of Consolidated Gold Fields PLC<SPLIT>,
 was named a nonexecutive director of this British industrial conglomerate<SPLIT>.

Replacing the <SPLIT> tags with whitespace causes the training to fail with InsufficientTrainingDataException:

Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC ,
 was named a nonexecutive director of this British industrial conglomerate .

Modifying the training data to contain a single <SPLIT> tag causes training to complete successfully:

Pierre Vinken<SPLIT>, 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC ,
 was named a nonexecutive director of this British industrial conglomerate .

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)