You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/09/02 10:55:22 UTC
(opennlp) branch OPENNLP-1909-resource-installer updated (bca31fc38 -> b1b38039f)
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1909-resource-installer
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit bca31fc38 OPENNLP-1902: Verify cached models against their SHA-512 checksum (#1239)
omit 44ac89eae model: Document the converged download path for hunspell and mecab
omit c2c6aa698 util: Converge dictionary downloads and unpacking on ResourceInstaller
omit 0eeefacd1 util: Pin converged catalog installs and mecab pax and reinstall behavior (failing tests)
omit 02d9f3c22 Merge branch 'OPENNLP-1894-lattice-cjk' into OPENNLP-1909-resource-installer
omit 317404aaf Merge branch 'OPENNLP-1893-hunspell' into OPENNLP-1909-resource-installer
omit 762435880 util: Address review: extract repeated test literals, tighten entry-ceiling prose
omit 6a6f109ea model: Document required remote checksums, the entry ceiling, and its overrides
omit 53cb0111d util: Bound archive entry counts and read ceiling overrides at class load
omit 966c2ac27 util: Pin entry-count budget and ceiling startup overrides (failing tests)
omit 6d45a69c7 util: Require checksums for remote sources and refuse promotion over existing files
omit bcea1bfce util: Pin required remote checksums and vacant-target promotion (failing tests)
omit 5357f64a9 model: Clarify installer expansion limits
omit 53c3c26c1 util: Bound tar metadata expansion and maximum-size skips
omit 8e1fff479 util: Pin tar skip and metadata expansion limits (failing tests)
omit 3d90f4f1b model: Document installer schemes, tar formats, and the Limits builder
omit 65d1154ec util: Bound timeouts, schemes, promotion, and tar headers
omit 6929207f6 util: Pin installer and tar hardening (failing tests)
omit a700e5930 util: Pin installer boundaries, defaults, and redirect statuses
omit 7ace48f45 model: Document installer limits, SHA-512, and staged installation
omit adf155ef7 util: Bound and stage ResourceInstaller downloads
omit ae6c3bca2 util: Pin hardened ResourceInstaller contract (failing tests)
omit 39b576e45 OPENNLP-1893: Fail loud on COMPOUNDRULE, IGNORE, KEEPCASE and gate full-strip rules behind FULLSTRIP
omit fe53f4c99 OPENNLP-1893: Expose silent COMPOUNDRULE, IGNORE, KEEPCASE and ungated full-strip rules with failing tests
omit 85cb63f50 OPENNLP-1894: Extract dictionary files from the archive root only
omit f3e2d3bc0 OPENNLP-1894: Bound matrix cells separately so real distributions load
omit 8e14a692c OPENNLP-1894: Add a remote-gated end-to-end test over the catalog dictionaries
omit 163e3960d OPENNLP-1894: Reconcile the shared download test files with the sibling PR
omit 9f811674f OPENNLP-1893: Reconcile the shared download test files with the sibling PR
omit e6d39bd6f OPENNLP-1893: Use a numeric character reference for the no-break space in the manual
omit 43a782513 OPENNLP-1893: Address review: unbox the boundary lookups and publish the stream ceiling
omit f171dcad1 OPENNLP-1894: Address review: cite named formats and align test conventions
omit cdcba0074 util: Address review: cite the installer mirror test and drop an em dash
omit aa07fbd57 util: Store *.bin sources packed in ResourceInstaller
omit b5ff4e112 util: Address review: fold tar detection into TarStream, tighten javadoc
omit 871c53e4c model: Document archived-resource installation with a mirror-tested example
omit 0826827fc util: Usage example and hardening tests for the resource installer and tar reader, corrected javadoc
omit f8c991ab6 util: General installer for user-supplied third-party resources
omit 722f3bdf4 OPENNLP-1893: Trigger CI for the DownloadUtil sync commit
omit ec43621bb OPENNLP-1894: Trigger CI for the budget-override commit
omit 39afc7e59 OPENNLP-1893: Sync shared DownloadUtil with the startup-overridable download ceiling
omit e2a437acd OPENNLP-1894: Make download and extraction byte budgets overridable at startup
omit 0b27d5ec1 OPENNLP-1893: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog
omit b81f34840 OPENNLP-1894: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog
omit 8cede5e29 OPENNLP-1893: Bound stream size and match affix conditions by code point
omit b79cc3104 OPENNLP-1894: Cap archive extract budgets and harden MeCab load edges
omit ba8ab36bb OPENNLP-1893: Fail loud on result-altering unsupported affix directives
omit 263035581 OPENNLP-1894: Move lattice tokenizer types into opennlp-api
omit d8cc10e29 OPENNLP-1893: Address review: fold the affix twins, extract tags, complete javadoc
omit 7a98076c8 OPENNLP-1894: Address review: complete javadoc, hoist constants, and fold fixture duplication
omit e0b2b4d68 OPENNLP-1893: Add {@inheritDoc} to the stemmer overrides and trim empty javadoc lines
omit fdc9f93a4 OPENNLP-1894: Trim parsed lines as Unicode whitespace and document the tokenizer overrides
omit dd0d2d928 OPENNLP-1893: Apply the review-convention pass: factual license prose, split null contracts, thread-safety annotations
omit 8d5afa229 OPENNLP-1893: Add hunspell manual coverage with mirror-tested examples
omit 7b7d2c458 OPENNLP-1893: Honor the blocking flags, circumfixes, and compound positioning
omit 2f0eab31e OPENNLP-1893: Decompose unanalyzed words into two flagged compound parts
omit e80f4d193 OPENNLP-1893: Walk only the affix rules that can apply, bucketed by their boundary character
omit 98320f530 OPENNLP-1893: Resolve numeric dictionary flags through the AF alias table
omit d16f3284c OPENNLP-1893: Read flags as code points, tolerate trailing morphology, and stem nothing from nothing
omit 322fd1a5a OPENNLP-1893: Cut morphology like hunspell does, accept UTF-8 flags and strip-only rules, and read the parser through the whitespace seam
omit bc718eed4 OPENNLP-1893: Document Hunspell dictionary acquisition with a license-preserving download helper
omit d37af0d32 OPENNLP-1893: Usage, threading, and malformed-input tests for the Hunspell engine, precise javadoc
omit ed7999288 OPENNLP-1893: Twofold suffix analysis through Hunspell continuation classes
omit 3dc0950e7 OPENNLP-1893: Hunspell-format affix engine over user-supplied dictionaries
omit 701f1db57 OPENNLP-1894: Apply the review-convention pass and drop unreferenced lexicon accessors
omit f28c02fc6 OPENNLP-1894: Document lattice CJK tokenization with a mirror-tested example
omit cac8a31ea OPENNLP-1894: Chain lattice nodes intrusively instead of allocating per-position lists
omit 625452d92 OPENNLP-1894: Hold the lexicon in a double-array trie with frequency-recoded labels
omit 04d380ce7 OPENNLP-1894: Precompute category runs, unbox the trie, and reject inexpressible dictionary values
omit 559fdd072 OPENNLP-1894: Categorize by code point, keep unknown candidates inside their category run, and validate context ids at load
omit a256b90a6 OPENNLP-1894: Document dictionary acquisition with a checksum-verifying download helper
omit c7fadc913 OPENNLP-1894: Keep lattice test sources ASCII-only via Unicode escapes, add an EUC-JP loading example
omit 05737234a OPENNLP-1894: Usage example and edge-case tests for the lattice and unigram segmenters, corrected javadoc
omit f1fada5c7 OPENNLP-1894: Frequency-driven segmentation over user-supplied lexicons
omit 781c22214 OPENNLP-1894: Character trie for lattice prefix search
omit f733fcffc OPENNLP-1894: Lattice segmentation over user-supplied mecab-format dictionaries
add f5e65680c OPENNLP-1904: Default-locale case folding corrupts lemmatizer models and output (#1206)
add 543067eea OPENNLP-1905: Default-locale case folding breaks Morfologik dictionary lookup (#1207)
add ef2610208 Bump org.apache.maven:apache-maven from 3.9.6 to 3.9.16 (#1224)
add 853028a9f Bump onnxruntime.version from 1.28.0 to 1.29.0 (#1216)
add b48862a87 Bump junit.version from 6.1.2 to 6.1.3 (#1218)
add fdffe5a22 Bump io.github.classgraph:classgraph from 4.8.186 to 4.8.192 (#1219)
add 9d224f938 OPENNLP-1906: Sentence detector abbreviation veto is quadratic in document length (#1208)
add bb41293b4 OPENNLP-1914 - Bump org.apache.maven.wrapper:maven-wrapper from 3.3.2 to 3.3.4 (#1222)
add 74464fbd7 Minor: Regenerated NOTICE File for 853028a9f46953c24eb0e1643445cc120a8040ad (#1228)
add 4df09a94e OPENNLP-1915: Fix line endings of mvnw.cmd (#1231)
add a42e50c40 OPENNLP-1913: Include Maven Wrapper configuration in source release (#1229)
add ea2ec5350 Minor: Regenerated NOTICE File for a42e50c407592db8bd05c756b98b0a32f733466e (#1234)
add 88810656a Bump io.github.classgraph:classgraph from 4.8.192 to 4.8.193 (#1241)
add ef942efd1 Minor: Regenerated NOTICE File for 88810656a5ea115ecb22ed2cd7f6fca5dbdf5acd (#1245)
add daf86d202 OPENNLP-1902: Verify cached models against their SHA-512 checksum (#1239)
add 6fd56f5f4 OPENNLP-1916: Normalizer documentation and fail loud on chain (#1235)
add b46b95706 OPENNLP-1903: Replace per-candidate Sequence copies with chain nodes in BeamSearch (#1205)
add 8f7befe44 OPENNLP-1922: Fix ReDoS / stack exhaustion in RegexNameFinderFactory EMAIL and URL patterns (#1250)
add 06eeaa425 Bump actions/setup-java from 5.7.0 to 6.0.0 (#1247)
add 723539519 Bump io.github.classgraph:classgraph from 4.8.193 to 4.8.194 (#1255)
add d6b864199 OPENNLP-1888: Document annotation container with typed offset-anchored layers (#1182)
add 06f43f0f0 OPENNLP-1923: Update logcaptor to 2.12.7 (#1253)
add e04982a48 Minor: Regenerated NOTICE File for 06f43f0f0433f746c9f26ac1b8e8e5d83fc39f39 (#1258)
add 4caec0a6c util: General installer for user-supplied third-party resources
add a624d721e util: Usage example and hardening tests for the resource installer and tar reader, corrected javadoc
add 8325cacd5 model: Document archived-resource installation with a mirror-tested example
add 407cad923 util: Address review: fold tar detection into TarStream, tighten javadoc
add 90faf3585 util: Store *.bin sources packed in ResourceInstaller
add 7c172f8df util: Address review: cite the installer mirror test and drop an em dash
add 6a911eb3e util: Pin hardened ResourceInstaller contract (failing tests)
add d859f2017 util: Bound and stage ResourceInstaller downloads
add d40f8d671 model: Document installer limits, SHA-512, and staged installation
add 32b87768d util: Pin installer boundaries, defaults, and redirect statuses
add 9b95286f2 util: Pin installer and tar hardening (failing tests)
add df5380b79 util: Bound timeouts, schemes, promotion, and tar headers
add c3a183294 model: Document installer schemes, tar formats, and the Limits builder
add 18594ef6d util: Pin tar skip and metadata expansion limits (failing tests)
add c18f23eaa util: Bound tar metadata expansion and maximum-size skips
add 73447d284 model: Clarify installer expansion limits
add e36ac5d6e util: Pin required remote checksums and vacant-target promotion (failing tests)
add 177c95181 util: Require checksums for remote sources and refuse promotion over existing files
add 9eb5c1849 util: Pin entry-count budget and ceiling startup overrides (failing tests)
add 8f24727b5 util: Bound archive entry counts and read ceiling overrides at class load
add f1b74ee07 model: Document required remote checksums, the entry ceiling, and its overrides
add b277ff462 util: Address review: extract repeated test literals, tighten entry-ceiling prose
add 65e3a3352 OPENNLP-1893: Hunspell-format affix engine over user-supplied dictionaries
add ec45639f4 OPENNLP-1893: Twofold suffix analysis through Hunspell continuation classes
add d8a2f9fa8 OPENNLP-1893: Usage, threading, and malformed-input tests for the Hunspell engine, precise javadoc
add e058e7e62 OPENNLP-1893: Document Hunspell dictionary acquisition with a license-preserving download helper
add dba8175ad OPENNLP-1893: Cut morphology like hunspell does, accept UTF-8 flags and strip-only rules, and read the parser through the whitespace seam
add bd2175d1f OPENNLP-1893: Read flags as code points, tolerate trailing morphology, and stem nothing from nothing
add 969f33723 OPENNLP-1893: Resolve numeric dictionary flags through the AF alias table
add d865ef8a8 OPENNLP-1893: Walk only the affix rules that can apply, bucketed by their boundary character
add 29341e781 OPENNLP-1893: Decompose unanalyzed words into two flagged compound parts
add 57f451691 OPENNLP-1893: Honor the blocking flags, circumfixes, and compound positioning
add a513d271c OPENNLP-1893: Add hunspell manual coverage with mirror-tested examples
add 052146aa8 OPENNLP-1893: Apply the review-convention pass: factual license prose, split null contracts, thread-safety annotations
add 5f60ed506 OPENNLP-1893: Add {@inheritDoc} to the stemmer overrides and trim empty javadoc lines
add 89c2ba1d3 OPENNLP-1893: Address review: fold the affix twins, extract tags, complete javadoc
add 6989ee3a5 OPENNLP-1893: Fail loud on result-altering unsupported affix directives
add 302b3533d OPENNLP-1893: Bound stream size and match affix conditions by code point
add 5e9e7a8b3 OPENNLP-1893: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog
add 3a8092da7 OPENNLP-1893: Sync shared DownloadUtil with the startup-overridable download ceiling
add 62714132a OPENNLP-1893: Trigger CI for the DownloadUtil sync commit
add 4e59422d3 OPENNLP-1893: Address review: unbox the boundary lookups and publish the stream ceiling
add 096379c0a OPENNLP-1893: Use a numeric character reference for the no-break space in the manual
add 86088af8c OPENNLP-1893: Reconcile the shared download test files with the sibling PR
add f1b0d3dcf OPENNLP-1893: Expose silent COMPOUNDRULE, IGNORE, KEEPCASE and ungated full-strip rules with failing tests
add db1da9889 OPENNLP-1893: Fail loud on COMPOUNDRULE, IGNORE, KEEPCASE and gate full-strip rules behind FULLSTRIP
add 18a85598e Merge branch 'OPENNLP-1893-hunspell' into OPENNLP-1909-resource-installer
add e6477c801 OPENNLP-1894: Lattice segmentation over user-supplied mecab-format dictionaries
add 728f6339a OPENNLP-1894: Character trie for lattice prefix search
add 127fc5185 OPENNLP-1894: Frequency-driven segmentation over user-supplied lexicons
add a32dd4b3d OPENNLP-1894: Usage example and edge-case tests for the lattice and unigram segmenters, corrected javadoc
add f5cc56444 OPENNLP-1894: Keep lattice test sources ASCII-only via Unicode escapes, add an EUC-JP loading example
add c0bbc0a1a OPENNLP-1894: Document dictionary acquisition with a checksum-verifying download helper
add 80a6a2c64 OPENNLP-1894: Categorize by code point, keep unknown candidates inside their category run, and validate context ids at load
add 40361dcfd OPENNLP-1894: Precompute category runs, unbox the trie, and reject inexpressible dictionary values
add 1b49eaaa1 OPENNLP-1894: Hold the lexicon in a double-array trie with frequency-recoded labels
add d554f0c4f OPENNLP-1894: Chain lattice nodes intrusively instead of allocating per-position lists
add c14f24e71 OPENNLP-1894: Document lattice CJK tokenization with a mirror-tested example
add 2665c00a7 OPENNLP-1894: Apply the review-convention pass and drop unreferenced lexicon accessors
add 1fd96ae00 OPENNLP-1894: Trim parsed lines as Unicode whitespace and document the tokenizer overrides
add 585ff0e16 OPENNLP-1894: Address review: complete javadoc, hoist constants, and fold fixture duplication
add 4c9d1871f OPENNLP-1894: Move lattice tokenizer types into opennlp-api
add d62a140b2 OPENNLP-1894: Cap archive extract budgets and harden MeCab load edges
add b024809b9 OPENNLP-1894: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog
add 200fcad97 OPENNLP-1894: Make download and extraction byte budgets overridable at startup
add 5a3451b09 OPENNLP-1894: Trigger CI for the budget-override commit
add cc55a3a76 OPENNLP-1894: Address review: cite named formats and align test conventions
add b09260476 OPENNLP-1894: Reconcile the shared download test files with the sibling PR
add 64ad31dbd OPENNLP-1894: Add a remote-gated end-to-end test over the catalog dictionaries
add 7725b7aa9 OPENNLP-1894: Bound matrix cells separately so real distributions load
add f24fbad35 OPENNLP-1894: Extract dictionary files from the archive root only
add c964bbd73 Merge branch 'OPENNLP-1894-lattice-cjk' into OPENNLP-1909-resource-installer
add d7bc43942 util: Pin converged catalog installs and mecab pax and reinstall behavior (failing tests)
add e0ca2ee04 util: Converge dictionary downloads and unpacking on ResourceInstaller
add 5bb222568 model: Document the converged download path for hunspell and mecab
add dba135358 OPENNLP-1894: Address lattice tokenizer review feedback
add b1b38039f Merge updated OPENNLP-1894 into OPENNLP-1909
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (bca31fc38)
\
N -- N -- N refs/heads/OPENNLP-1909-resource-installer (b1b38039f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/license.yml | 2 +-
.github/workflows/maven.yml | 8 +-
.github/workflows/publish-snapshots.yml | 2 +-
.mvn/wrapper/maven-wrapper.properties | 20 +-
NOTICE | 6 +-
mvnw | 50 +-
mvnw.cmd | 56 +-
.../java/opennlp/tools/document/Annotation.java | 68 +++
.../main/java/opennlp/tools/document/Document.java | 196 +++++++
.../opennlp/tools/document/DocumentAnalyzer.java | 125 ++++
.../opennlp/tools/document/DocumentAnnotator.java | 62 ++
.../opennlp/tools/document/DocumentAnnotators.java | 118 ++++
.../opennlp/tools/document/ImmutableDocument.java | 193 ++++++
.../main/java/opennlp/tools/document/LayerKey.java | 165 ++++++
.../main/java/opennlp/tools/document/Layers.java | 134 +++++
.../tools/tokenize/lattice/CategoryTable.java | 201 +++++++
.../tools/tokenize/lattice/DoubleArrayLexicon.java | 253 ++++++++
.../tools/tokenize/lattice/LatticeTokenizer.java | 2 +-
.../tools/tokenize/lattice/MecabDictionary.java | 416 +------------
.../tools/tokenize/lattice/UnigramSegmenter.java | 60 +-
.../java/opennlp/tools/util/ResourceLimits.java | 46 +-
.../main/java/opennlp/tools/util/StringUtil.java | 29 +-
.../util/normalizer/OffsetAwareNormalizer.java | 6 +-
.../tools/document/DocumentAnnotatorsTest.java | 121 ++++
.../tools/document/DocumentContractTest.java | 644 +++++++++++++++++++++
.../java/opennlp/tools/document/DocumentTest.java | 148 +++++
.../opennlp/tools/util/ResourceLimitsTest.java | 64 ++
.../opennlp/tools/util/StringUtilLocaleTest.java | 78 +++
.../tools/cmdline/DetailedFMeasureListener.java | 2 +-
.../tools/cmdline/FineGrainedReportListener.java | 32 +-
.../cmdline/FineGrainedReportListenerTest.java | 71 +++
.../src/main/java/opennlp/dl/InferenceOptions.java | 2 +-
.../src/main/java/opennlp/tools/ml/BeamSearch.java | 125 +++-
.../java/opennlp/tools/ml/model/AbstractModel.java | 5 +-
.../opennlp/tools/util/TrainingParameters.java | 9 +-
.../java/opennlp/tools/ml/BeamSearchBenchmark.java | 166 ++++++
.../SentenceDetectorMEAbbreviationBenchmark.java | 208 +++++++
.../opennlp/tools/chunker/ChunkerAnnotator.java | 145 +++++
.../tools/lemmatizer/DictionaryLemmatizer.java | 3 +-
.../tools/lemmatizer/LemmatizerAnnotator.java | 137 +++++
.../opennlp/tools/lemmatizer/LemmatizerME.java | 3 +-
.../tools/namefind/NameFinderAnnotator.java | 149 +++++
.../tools/namefind/RegexNameFinderFactory.java | 39 +-
.../java/opennlp/tools/parser/ParserAnnotator.java | 208 +++++++
.../opennlp/tools/postag/POSTaggerAnnotator.java | 118 ++++
.../sentdetect/SentenceDetectorAnnotator.java | 94 +++
.../tools/sentdetect/SentenceDetectorME.java | 213 ++++++-
.../opennlp/tools/stemmer/StemmerAnnotator.java | 107 ++++
.../opennlp/tools/tokenize/TokenizerAnnotator.java | 116 ++++
.../AggregateCharSequenceNormalizer.java | 27 +-
.../AlignedAggregateCharSequenceNormalizer.java | 4 +-
.../tools/chunker/ChunkerAnnotatorTest.java | 211 +++++++
.../tools/document/DocumentAnalyzerTest.java | 206 +++++++
.../document/DocumentPipelineExampleTest.java | 246 ++++++++
.../opennlp/tools/document/TestComponents.java | 92 +++
.../tools/lemmatizer/DictionaryLemmatizerTest.java | 33 ++
.../tools/lemmatizer/LemmatizerAnnotatorTest.java | 247 ++++++++
.../tools/lemmatizer/LemmatizerMELocaleTest.java | 72 +++
.../tools/ml/BeamSearchEquivalenceTest.java | 612 ++++++++++++++++++++
.../test/java/opennlp/tools/ml/BeamSearchTest.java | 74 +++
.../java/opennlp/tools/ml/SeededMaxentModel.java | 120 ++++
.../tools/namefind/NameFinderAnnotatorTest.java | 298 ++++++++++
.../tools/namefind/RegexNameFinderFactoryTest.java | 94 +++
.../opennlp/tools/parser/ParserAnnotatorTest.java | 158 +++++
.../tools/postag/POSTaggerAnnotatorTest.java | 231 ++++++++
.../LegacyAbbreviationSentenceDetectorME.java | 111 ++++
...tenceDetectorMEAbbreviationEquivalenceTest.java | 481 +++++++++++++++
.../tools/stemmer/StemmerAnnotatorTest.java | 97 ++++
.../tokenize/lattice/LatticeUsageExampleTest.java | 6 +-
.../tokenize/lattice/UnigramSegmenterTest.java | 29 +-
.../AggregateCharSequenceNormalizerTest.java | 65 +++
.../normalizer/AlignedNormalizerPipelineTest.java | 48 +-
.../tools/util/normalizer/TextNormalizerTest.java | 8 +-
opennlp-distr/src/main/assembly/src.xml | 7 +
opennlp-distr/src/main/readme/NOTICE | 6 +-
opennlp-docs/src/docbkx/document.xml | 338 +++++++++++
opennlp-docs/src/docbkx/normalizer.xml | 4 +-
opennlp-docs/src/docbkx/opennlp.xml | 1 +
.../lemmatizer/MorfologikLemmatizer.java | 3 +-
.../tagdict/MorfologikTagDictionary.java | 3 +-
.../opennlp/morfologik/AbstractMorfologikTest.java | 16 +-
.../opennlp/morfologik/MorfologikLocaleTest.java | 80 +++
...yWithLemma.info => dictionaryLocaleSafety.info} | 2 +-
.../src/test/resources/dictionaryLocaleSafety.txt | 2 +
.../ChunkerDetailedFMeasureListenerTest.java | 37 ++
.../java/opennlp/tools/util/StringUtilTest.java | 27 +
.../opennlp/tools/util/TrainingParametersTest.java | 36 ++
pom.xml | 8 +-
88 files changed, 8764 insertions(+), 601 deletions(-)
create mode 100644 opennlp-api/src/main/java/opennlp/tools/document/Annotation.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/document/Document.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/document/DocumentAnalyzer.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/document/DocumentAnnotator.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/document/DocumentAnnotators.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/document/ImmutableDocument.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/document/LayerKey.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/document/Layers.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/tokenize/lattice/CategoryTable.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/tokenize/lattice/DoubleArrayLexicon.java
create mode 100644 opennlp-api/src/test/java/opennlp/tools/document/DocumentAnnotatorsTest.java
create mode 100644 opennlp-api/src/test/java/opennlp/tools/document/DocumentContractTest.java
create mode 100644 opennlp-api/src/test/java/opennlp/tools/document/DocumentTest.java
create mode 100644 opennlp-api/src/test/java/opennlp/tools/util/ResourceLimitsTest.java
create mode 100644 opennlp-api/src/test/java/opennlp/tools/util/StringUtilLocaleTest.java
create mode 100644 opennlp-core/opennlp-cli/src/test/java/opennlp/tools/cmdline/FineGrainedReportListenerTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/ml/BeamSearchBenchmark.java
create mode 100644 opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/sentdetect/SentenceDetectorMEAbbreviationBenchmark.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/chunker/ChunkerAnnotator.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/lemmatizer/LemmatizerAnnotator.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/NameFinderAnnotator.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/ParserAnnotator.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/postag/POSTaggerAnnotator.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/SentenceDetectorAnnotator.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/StemmerAnnotator.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerAnnotator.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/chunker/ChunkerAnnotatorTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/document/DocumentAnalyzerTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/document/DocumentPipelineExampleTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/document/TestComponents.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/lemmatizer/LemmatizerAnnotatorTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/lemmatizer/LemmatizerMELocaleTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/ml/BeamSearchEquivalenceTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/ml/SeededMaxentModel.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/namefind/NameFinderAnnotatorTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/ParserAnnotatorTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/postag/POSTaggerAnnotatorTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentdetect/LegacyAbbreviationSentenceDetectorME.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentdetect/SentenceDetectorMEAbbreviationEquivalenceTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/StemmerAnnotatorTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/AggregateCharSequenceNormalizerTest.java
create mode 100644 opennlp-docs/src/docbkx/document.xml
create mode 100644 opennlp-extensions/opennlp-morfologik/src/test/java/opennlp/morfologik/MorfologikLocaleTest.java
copy opennlp-extensions/opennlp-morfologik/src/test/resources/{dictionaryWithLemma.info => dictionaryLocaleSafety.info} (95%)
create mode 100644 opennlp-extensions/opennlp-morfologik/src/test/resources/dictionaryLocaleSafety.txt