You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/09/17 06:50:38 UTC
(opennlp) branch OPENNLP-1951-ad-name-text-id updated (cc44f971c -> a50a54077)
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1951-ad-name-text-id
in repository https://gitbox.apache.org/repos/asf/opennlp.git
from cc44f971c Bump actions/setup-java from 6.0.0 to 6.0.1 (#1292)
add c961a00c0 OPENNLP-1955: Replace literal String.split calls with a regex-free StringUtil.split (#1302)
add 11beb47d3 OPENNLP-1956: Update classgraph to 4.8.195 (#1294)
add 393c86eed Minor: Regenerated NOTICE File for 11beb47d3cfa5fb743e0ceb1ecb48ac513974bdc (#1304)
add 7142e77c3 OPENNLP-1949: End ADSentenceStream at end of file inside an open sentence (#1298)
add a00134cb6 OPENNLP-1948: Release file handles by MarkableFileInputStreamFactory and the Leipzig reader on close and reset (#1297)
add f09b7f0a5 OPENNLP-1947: Stop TokenizerME from changing the shared WhitespaceTokenizer (#1296)
add 7def2ce9c OPENNLP-1951: Clear adaptive data at each new LIT or CIE text in ADNameSampleStream
add 5bcbc29ee OPENNLP-1950: Fix ADChunkSampleStream fails with NPE on the FlorestaVirgem corpus (#1299)
new 628cdfd2a Merge Apache main into OPENNLP-1951-ad-name-text-id
add 229e4255a OPENNLP-1952: Report a CoNLL-U multiword token range without its word lines (#1301)
new a50a54077 Merge Apache main into OPENNLP-1951-ad-name-text-id
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
NOTICE | 2 +-
.../tools/tokenize/WhitespaceTokenizer.java | 14 ++
.../opennlp/tools/util/PlainTextByLineStream.java | 9 +-
.../main/java/opennlp/tools/util/StringUtil.java | 74 ++++++++++
.../opennlp/tools/util/ext/ExtensionLoader.java | 3 +-
.../opennlp/tools/util/StringUtilSplitTest.java | 164 +++++++++++++++++++++
.../tools/cmdline/FineGrainedReportListener.java | 12 +-
.../tools/cmdline/doccat/DoccatSVMTrainerTool.java | 3 +-
.../tools/cmdline/doccat/DoccatTrainerTool.java | 3 +-
.../languagemodel/NGramLanguageModelTool.java | 3 +-
.../TokenNameFinderCrossValidatorTool.java | 3 +-
.../namefind/TokenNameFinderEvaluatorTool.java | 3 +-
.../namefind/TokenNameFinderTrainerTool.java | 3 +-
.../cmdline/sentiment/SentimentEvaluatorTool.java | 3 +-
.../tools/formats/BioNLP2004NameSampleStream.java | 2 +-
.../tools/formats/Conll02NameSampleStream.java | 2 +-
.../tools/formats/Conll03NameSampleStream.java | 2 +-
.../tools/formats/ConllXPOSSampleStream.java | 3 +-
.../tools/formats/EvalitaNameSampleStream.java | 2 +-
.../formats/GermEval2014NameSampleStream.java | 2 +-
.../tools/formats/ad/ADChunkSampleStream.java | 13 +-
.../tools/formats/ad/ADNameSampleStream.java | 27 +++-
.../opennlp/tools/formats/ad/ADSentenceStream.java | 35 ++++-
.../formats/ad/PortugueseContractionUtility.java | 4 +-
.../tools/formats/brat/BratAnnotationStream.java | 7 +-
.../formats/brat/BratNameSampleStreamFactory.java | 3 +-
.../opennlp/tools/formats/conllu/ConlluStream.java | 23 ++-
.../tools/formats/conllu/ConlluWordLine.java | 3 +-
.../leipzig/LeipzigLanguageSampleStream.java | 6 +-
.../tools/formats/masc/MascSentenceParser.java | 3 +-
.../formats/nkjp/NKJPSegmentationDocument.java | 3 +-
.../formats/ontonotes/DocumentToLineStream.java | 3 +-
.../tools/util/MarkableFileInputStream.java | 10 ++
.../tools/formats/ad/ADChunkSampleStreamTest.java | 47 ++++++
.../tools/formats/ad/ADNameSampleStreamTest.java | 72 +++++++++
.../tools/formats/ad/ADSentenceParserTest.java | 74 ++++++++++
.../tools/formats/ad/ADSentenceStreamTest.java | 126 ++++++++++++++++
.../tools/formats/conllu/ConlluStreamTest.java | 104 +++++++++++++
.../leipzig/LeipzigLanguageSampleStreamTest.java | 16 ++
.../tools/util/MarkableFileInputStreamTest.java | 88 +++++++++++
.../opennlp/tools/chunker/ChunkSampleStream.java | 3 +-
.../java/opennlp/tools/doccat/DoccatFactory.java | 3 +-
.../tools/lemmatizer/DictionaryLemmatizer.java | 6 +-
.../tools/lemmatizer/LemmaSampleStream.java | 3 +-
.../java/opennlp/tools/postag/POSDictionary.java | 2 +-
.../java/opennlp/tools/tokenize/TokenizerME.java | 13 +-
.../util/featuregen/WordClusterDictionary.java | 3 +-
.../tools/util/normalizer/EmojiAnnotations.java | 10 +-
.../tools/util/normalizer/EmojiEmoticons.java | 10 +-
.../FullCaseFoldCharSequenceNormalizer.java | 10 +-
.../java/opennlp/tools/util/wordvector/Glove.java | 3 +-
.../tools/lemmatizer/DictionaryLemmatizerTest.java | 18 +++
.../opennlp/tools/postag/POSDictionaryTest.java | 15 ++
.../opennlp/tools/tokenize/TokenizerMETest.java | 40 +++++
.../tools/tokenize/WhitespaceTokenizerTest.java | 13 ++
opennlp-distr/src/main/readme/NOTICE | 2 +-
opennlp-docs/src/docbkx/corpora.xml | 47 ++++++
opennlp-docs/src/docbkx/introduction.xml | 17 +++
opennlp-docs/src/docbkx/tokenizer.xml | 13 ++
.../opennlp/uima/namefind/AbstractNameFinder.java | 5 +-
.../tools/util/PlainTextByLineStreamTest.java | 48 ++++++
pom.xml | 2 +-
62 files changed, 1189 insertions(+), 76 deletions(-)
create mode 100644 opennlp-api/src/test/java/opennlp/tools/util/StringUtilSplitTest.java
create mode 100644 opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/ad/ADSentenceParserTest.java
create mode 100644 opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/ad/ADSentenceStreamTest.java
create mode 100644 opennlp-core/opennlp-formats/src/test/java/opennlp/tools/util/MarkableFileInputStreamTest.java
(opennlp) 02/02: Merge Apache main into OPENNLP-1951-ad-name-text-id
Posted by kr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a commit to branch OPENNLP-1951-ad-name-text-id
in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit a50a5407786cc5c4b0d6bbd037c08b2665aa210c
Merge: 628cdfd2a 229e4255a
Author: Kristian Rickert <kr...@apache.org>
AuthorDate: Thu Sep 17 02:33:34 2026 -0400
Merge Apache main into OPENNLP-1951-ad-name-text-id
.../opennlp/tools/formats/conllu/ConlluStream.java | 23 ++++-
.../tools/formats/conllu/ConlluStreamTest.java | 104 +++++++++++++++++++++
opennlp-docs/src/docbkx/corpora.xml | 13 +++
3 files changed, 136 insertions(+), 4 deletions(-)
(opennlp) 01/02: Merge Apache main into OPENNLP-1951-ad-name-text-id
Posted by kr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a commit to branch OPENNLP-1951-ad-name-text-id
in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 628cdfd2ac159ad1c6553b098e3d3e55d9be7691
Merge: 7def2ce9c 5bcbc29ee
Author: Kristian Rickert <kr...@apache.org>
AuthorDate: Thu Sep 17 02:32:57 2026 -0400
Merge Apache main into OPENNLP-1951-ad-name-text-id
# Conflicts:
# opennlp-docs/src/docbkx/corpora.xml
.../tools/formats/ad/ADChunkSampleStream.java | 13 +++-
.../opennlp/tools/formats/ad/ADSentenceStream.java | 17 ++++-
.../tools/formats/ad/ADChunkSampleStreamTest.java | 47 ++++++++++++++
.../tools/formats/ad/ADSentenceParserTest.java | 74 ++++++++++++++++++++++
opennlp-docs/src/docbkx/corpora.xml | 12 ++++
5 files changed, 161 insertions(+), 2 deletions(-)
diff --cc opennlp-docs/src/docbkx/corpora.xml
index 4681bba30,deeafe5a5..97c0f0a70
--- a/opennlp-docs/src/docbkx/corpora.xml
+++ b/opennlp-docs/src/docbkx/corpora.xml
@@@ -430,18 -430,17 +430,30 @@@ Execution time: 1,955 seconds]]
3.0.0, the empty open sentence no longer causes an endless read.
</para>
</section>
+ <section xml:id="tools.corpora.arvores-deitadas.leaf-tags">
+ <title>AD leaf tags with embedded equals signs</title>
+ <para>
+ Leaf tags may contain equals signs after the indentation, as in
+ <code>=H==CJT:num</code>. The reader takes the functional tag after the last
+ colon, so this example has the tag <code>num</code>. A chunk leaf without a
+ functional tag uses its lexeme as the tag instead of a null value. This permits
+ FlorestaVirgem chunk conversion to continue and changes the POS labels of the
+ affected tagged leaves to their functional tags, such as <code>num</code> or
+ <code>prop</code>. Re-evaluate models trained from affected samples.
+ </para>
++ </section>
+ <section xml:id="tools.corpora.arvores-deitadas.text-boundaries">
+ <title>Text boundaries in AD name samples</title>
+ <para>
+ During a pass over LIT or CIE input, consecutive sentences with the same text
+ name share a text id. When the name changes, the first name sample of the next
+ text sets <code>clearAdaptiveData</code>, so name-finder context does not carry
+ across text boundaries. Resetting the stream clears this boundary state, so the
+ first sample of a new pass also clears adaptive data. Amazonia input continues
+ to use its numeric text id.
+ This changes training and evaluation samples for LIT and CIE corpora; evaluate
+ models with representative data from those corpora.
+ </para>
</section>
<section xml:id="tools.corpora.arvores-deitadas.evaluation">
<title>Training and Evaluation</title>