[jira] [Commented] (OPENNLP-1201) add bailout way for certain languages in order to use POS features

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OPENNLP-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623586#comment-16623586 ] 

ASF GitHub Bot commented on OPENNLP-1201:
-----------------------------------------

kojisekig opened a new pull request #334: OPENNLP-1201: [experimental] add TokenNameFinderFactory and FeatureGe…
URL: https://github.com/apache/opennlp/pull/334
 
 
   …nerator so that user can add an auxialiary information, e.g. POS tag, to a token when running TokenNameFinderTrainer
   
   Thank you for contributing to Apache OpenNLP.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
        in the commit message?
   
   - [ ] Does your PR title start with OPENNLP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
   
   - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] Have you ensured that the full suite of tests is executed via mvn clean install at the root opennlp folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file in opennlp folder?
   - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found in opennlp folder?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> add bailout way for certain languages in order to use POS features
> ------------------------------------------------------------------
>
>                 Key: OPENNLP-1201
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1201
>             Project: OpenNLP
>          Issue Type: Improvement
>          Components: Command Line Interface, Formats
>    Affects Versions: 1.8.4
>            Reporter: Koji Sekiguchi
>            Priority: Major
>
> As OpenNLP tools depend on the fact that text being processed needs to be tokenized in advance (in other words, words in the text are separated each other by space), it is difficult for uses who use certain languages (e.g. CJK) to use POS (Part-of-Speech) features.
> To simplify the explanation, consider using NameFinder for Japanese text. In NameFinder tools (Train, Eval, Recognize), they require that users should provide Japanese text which has already been tokenized, but once we tokenize Japanese text, it loses POS information. (I think Chinese language has same problem)
> Let me describe this problem for western language users :) (English, French, Italian, etc.) without using Japanese letters. I’ll try to use English alphabet, instead.
> Suppose you have a sentence text “isentthemachine” which you want to give NameFinder, you use morphological analyzer in order to tokenize the sentence. There are two possible sequence of tokens:
> - i (PPSS) / sent (VBD) / the (AT) / machine (NP)
> - i (PPSS) / sent (VBD) / them (PPO) / a (AT) / chine (NP)
> As you noticed, morphological analyzer not only tokenizes the sentence, but also tags POS tag to each token. Same thing takes place in Japanese language (and Chinese language, I think).
> However, in OpenNLP feature generator API, it accepts sequence of tokens thru API i.e. `String[] tokens`, I cannot produce POS feature in the feature generator.
> To solve this problem (and to invite many users to our community), I’d like to suggest that OpenNLP tools allow users to add optional information to each tokenized word.
> For example, one can give the following text when using NameFinder tools.
> {code}
> $ cat en-ner.train
> I/PPSS sent/VBD the/AT machine/NP
> {code}
> When using such text, they must inform the tool that the token has POS tag in the text by using a certain option e.g. -postag
> {code}
> $ opennlp TokenNameFinderTrainer -data en-ner.train -model en-ner.bin -postag
> {code}
> We can maintain the backward compatibility to set -postag false by default and in this case, existing feature generators work exactly the same as before. If a user set -postag option in the command line, the existing feature generators eliminate “/POS” part of token “word/POS” in the text so that they can produce same features as before.
> I’d like to add a simple feature generator which generates only “POS” part of token “word/POS” in the text, in addition to managing -postag option. This simple feature generator allows Japanese/Chinese users to produce precise POS features.
> I’d like to focus on NameFinder in this ticket (Let me add this option to other tools (chunker, classifier, etc.) in another ticket, if needed).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)