[jira] [Resolved] (OPENNLP-711) SentenceDetectorME::sentPosDetect() with useTokenEnd=false

["Joern Kottmann (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joern Kottmann resolved OPENNLP-711.
------------------------------------
    Resolution: Fixed

The proposed fix is applied. Please verify that it works for you.

To acknowledge that the issue is fixed for you please *close* the issue.

> SentenceDetectorME::sentPosDetect() with useTokenEnd=false
> ----------------------------------------------------------
>
>                 Key: OPENNLP-711
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-711
>             Project: OpenNLP
>          Issue Type: Bug
>          Components: Sentence Detector
>    Affects Versions: 1.6.0
>            Reporter: Eugen Hanussek
>            Priority: Minor
>             Fix For: 1.6.0
>
>
> I trained the SentenceModel with a german korpus and wondered about the results for the following input (a mark indicates the expected split):
> {code:xml}
> "I am hungry.Ich bin Mr. Bean.Ein guter Satz."
>              ^                ^
> {code}
> The result was 3 sentences. Good, but the split was not at the eosChar. It was after the token with the eosChar: "I am hungry.Ich" , "bin Mr. Bean.Ein", ...
> After some debugging I found out that I have to set useTokenEnd=false in the SentenceDetectorFactory-ctor.
> And then I found a *little bug in SentenceDetectorME* when the span is calculated:
> {code:java}
>   public Span[] sentPosDetect(String s) {
> ...
>       if (bestOutcome.equals(SPLIT) && isAcceptableBreak(s, index, cint)) {
>         if (index != cint) {
>           if (useTokenEnd) {
>             positions.add(getFirstNonWS(s, getFirstWS(s,cint + 1)));
>           }
>           else {
>             positions.add(getFirstNonWS(s,cint)); // this should be positions.add(getFirstNonWS(s,cint + 1)); 
>           }
>           sentProbs.add(probs[model.getIndex(bestOutcome)]);
>         }
>         index = cint + 1;
>       }
> ...
> {code}
> This change has only impact on models with useTokenEnd=false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)