You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steve Rowe (JIRA)" <ji...@apache.org> on 2017/12/12 05:01:02 UTC
[jira] [Updated] (LUCENE-2899) Add OpenNLP Analysis capabilities as
a module
[ https://issues.apache.org/jira/browse/LUCENE-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Rowe updated LUCENE-2899:
-------------------------------
Attachment: LUCENE-2899.patch
Patch, lots of changes (see below), I think it's ready to go (precommit and all Lucene/Solr tests pass). My plan is to wait a couple days for review, then commit if there are no objections.
Changes since the last patch:
* Corrected some test model training data.
* Switched OpenNLPTokenizer to require both the sentence and tokenizer models - I couldn't think of a reason to support specification of only one of them.
* OpenNLPTokenizer now extends SegmentingTokenizerBase, and uses an OpenNLP sentence segmenter via shim class OpenNLPSentenceBreakIterator. End-of-sentence tokens are marked by setting a bit on the FlagsAttribute. All OpenNLP-based filters operate on sentences.
* OpenNLPLemmatizerFilter now supports dictionary-based and/or model-based lemmatization; if both are configured, dictionary-based tokenization is performed first, and then out-of-vocabulary tokens are lemmatized using the model.
* Each analysis operation is now in its own component (previously OpenNLPFilter did multiple things).
* Removed the end-of-sentence hack in OpenNLPPOSFilter (described in an earlier comment on this issue) - periods are no longer appended to sentences prior to pos tagging.
* Named entity recognition is now handled in an update request processor in the analysis-extras Solr contrib (though the NER model loading machinery remains in OpenNLPOpsFactory in the lucene/analysis/opennlp module).
* Added ref guide docs.
* Added CHANGES.txt entries.
> Add OpenNLP Analysis capabilities as a module
> ---------------------------------------------
>
> Key: LUCENE-2899
> URL: https://issues.apache.org/jira/browse/LUCENE-2899
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 4.9, 6.0
>
> Attachments: LUCENE-2899-6.1.0.patch, LUCENE-2899-RJN.patch, LUCENE-2899.patch, LUCENE-2899.patch, LUCENE-2899.patch, LUCENE-2899.patch, LUCENE-2899.patch, LUCENE-2899.patch, OpenNLPFilter.java, OpenNLPTokenizer.java
>
>
> Now that OpenNLP is an ASF project and has a nice license, it would be nice to have a submodule (under analysis) that exposed capabilities for it. Drew Farris, Tom Morton and I have code that does:
> * Sentence Detection as a Tokenizer (could also be a TokenFilter, although it would have to change slightly to buffer tokens)
> * NamedEntity recognition as a TokenFilter
> We are also planning a Tokenizer/TokenFilter that can put parts of speech as either payloads (PartOfSpeechAttribute?) on a token or at the same position.
> I'd propose it go under:
> modules/analysis/opennlp
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org