You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@opennlp.apache.org by "Nishant Shrivastava (Jira)" <ji...@apache.org> on 2025/05/11 20:01:00 UTC

[jira] [Created] (OPENNLP-1736) NGramLanguageModel - Allow choice of smoothing/discounting algorithm

Nishant Shrivastava created OPENNLP-1736:
--------------------------------------------

             Summary: NGramLanguageModel - Allow choice of smoothing/discounting algorithm
                 Key: OPENNLP-1736
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1736
             Project: OpenNLP
          Issue Type: Wish
          Components: language model
    Affects Versions: 2.5.4
            Reporter: Nishant Shrivastava


Currently, NGramLanguageModel uses stupid backoff to deal with “zero probability n-grams”. https://issues.apache.org/jira/browse/OPENNLP-986

It will be useful, if we can refactor it to pass a smoothing/discounting logic from outside.
This will allow us to add implementations of other smoothing/discounting techniques (e.g. below) in future.

https://en.wikipedia.org/wiki/Kneser%E2%80%93Ney_smoothing
https://en.wikipedia.org/wiki/Good%E2%80%93Turing_frequency_estimation



--
This message was sent by Atlassian Jira
(v8.20.10#820010)