You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@opennlp.apache.org by "tonylxc (JIRA)" <ji...@apache.org> on 2013/11/14 22:45:21 UTC

[jira] [Updated] (OPENNLP-618) Tokenize "can't"

     [ https://issues.apache.org/jira/browse/OPENNLP-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

tonylxc updated OPENNLP-618:
----------------------------

    Description: When I use OpenNLP's tokenizer to tokenize this sentence "I can't do it", I get tokens like "[I] [ca] [n't] [do] [it] [.]". Isn't it supposed to be something like "[I] [can] ['] [t] [do] [it] [.]" or "[I] [can't] [do] [it] [.]"? I know LanguageTool's tokenizer gives tokens like "[I] [can] ['] [t] [do] [it] [.]".  (was: When I use OpenNLP's tokenizer to tokenize this sentence "I can't do it", I get tokens like "[I] [ca] [n't] [do] [it] [.]". Isn't it supposed to be something like "[I] [can] ['] [t] [do] [it] [.]" or "[I] [can't] [do] [it] [.]"?)

> Tokenize "can't"
> ----------------
>
>                 Key: OPENNLP-618
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-618
>             Project: OpenNLP
>          Issue Type: Question
>          Components: OpenNLP ML
>         Environment: Windows 7 Enterprise, Java SE 1.7.0_40-b43
>            Reporter: tonylxc
>            Priority: Critical
>
> When I use OpenNLP's tokenizer to tokenize this sentence "I can't do it", I get tokens like "[I] [ca] [n't] [do] [it] [.]". Isn't it supposed to be something like "[I] [can] ['] [t] [do] [it] [.]" or "[I] [can't] [do] [it] [.]"? I know LanguageTool's tokenizer gives tokens like "[I] [can] ['] [t] [do] [it] [.]".



--
This message was sent by Atlassian JIRA
(v6.1#6144)