You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@opennlp.apache.org by "Joern Kottmann (JIRA)" <ji...@apache.org> on 2014/02/18 14:28:21 UTC

[jira] [Updated] (OPENNLP-618) Tokenize "can't"

     [ https://issues.apache.org/jira/browse/OPENNLP-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joern Kottmann updated OPENNLP-618:
-----------------------------------

    Component/s:     (was: OpenNLP ML)

> Tokenize "can't"
> ----------------
>
>                 Key: OPENNLP-618
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-618
>             Project: OpenNLP
>          Issue Type: Question
>         Environment: Windows 7 Enterprise, Java SE 1.7.0_40-b43
>            Reporter: tonylxc
>
> When I use OpenNLP's tokenizer to tokenize this sentence "I can't do it.", I get tokens like "[I] [ca] [n't] [do] [it] [.]". Isn't it supposed to be something like "[I] [can] ['] [t] [do] [it] [.]" or "[I] [can't] [do] [it] [.]"? I know LanguageTool's tokenizer gives tokens like "[I] [can] ['] [t] [do] [it] [.]".



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)