You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by "Greg Holmberg (JIRA)" <ui...@incubator.apache.org> on 2007/01/04 18:35:27 UTC

[jira] Commented: (UIMA-151) Add project for uima whitespace tokenizer implementation

    [ https://issues.apache.org/jira/browse/UIMA-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462262 ] 

Greg Holmberg commented on UIMA-151:
------------------------------------

May I suggest using the ICU Unicode package: http://icu.sourceforge.net

It already has a whitespace tokenizer for Java that understands all the Unicode conventions (apparently these are not trivial).  See the com.ibm.icu.text.BreakIterator and com.ibm.icu.util.StringTokenizer classes.  They also have a nice article, "Text Boundary Analysis in Java", here: http://icu.sourceforge.net/docs/papers/text_boundary_analysis_in_java



> Add project for uima whitespace tokenizer implementation
> --------------------------------------------------------
>
>                 Key: UIMA-151
>                 URL: https://issues.apache.org/jira/browse/UIMA-151
>             Project: UIMA
>          Issue Type: New Feature
>          Components: Sandbox
>    Affects Versions: 2.1
>            Reporter: Michael Baessler
>         Assigned To: Michael Baessler
>             Fix For: 2.1
>
>
> Add project for uima whitespace tokenizer implementation

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira