You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@opennlp.apache.org by "martin (JIRA)" <ji...@apache.org> on 2017/06/29 05:53:00 UTC

[jira] [Created] (OPENNLP-1099) Is this a typical tokenization issue?

martin created OPENNLP-1099:
-------------------------------

             Summary: Is this a typical tokenization issue?
                 Key: OPENNLP-1099
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1099
             Project: OpenNLP
          Issue Type: Bug
          Components: Lemmatizer
            Reporter: martin


I am testing openNLP and found some significant tokenization issue involving punctuation.  

Thank you Costco!
i love costco!
I love Costco!!
FUCK IKEA.

In all these cases, the last punctuation is not split so "Costco!" and "IKEA." are treated as one token. This looks like a systematic problem. 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)