You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by "krickert (via GitHub)" <gi...@apache.org> on 2026/07/16 10:51:43 UTC
[PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
krickert opened a new pull request, #1177:
URL: https://github.com/apache/opennlp/pull/1177
This change adds an emoji annotation layer that classic components can opt into, built in three strictly separated tiers so no third-party data ever enters the project:
- **Bundled facts**: a project-authored attribute-row table (`codepoints;attribute;value;source;notes`) covering 40 symbols with four attributes each, loaded by `EmojiAnnotations` with per-value provenance and an audit test. The file carries no third-party data; notes cite UCD character names.
- **Derived facts**: `EmojiFlags` decodes ISO 3166 codes from flag emoji sequences (regional-indicator pairs and tag sequences) as a pure algorithm over the code points, with no data file at all.
- **Joined facts**: `EmojiAnnotationJoin` resolves ISO codes against a user-installed gazetteer at load or run time. Foreign identifiers are never baked into the bundled files, so dataset version churn cannot stale the distribution.
The annotation surface is `EmojiAnnotator`, a parallel accessor beside `Term`, deliberately not a set of `Dimension` constants (rationale in the javadoc). Wiring is opt-in only: a feature-generator XML factory for NameFinder, a `DoccatFactory` array entry, and the `SentimentFactory` seam. Default behavior of every component is unchanged; nothing activates without explicit configuration.
Out of scope for this iteration, tracked on the ticket: manual section, skin-tone/ZWJ sequence reduction, and a loader for user-downloaded sentiment rankings (the project ships no sentiment data; the bundled table's coarse sentiment column is project-authored).
Verification: the full opennlp-runtime suite passes on current main (1449 tests), including the 39 emoji-specific tests (record store audit, flag decoding, annotator surface, and the opt-in wiring for all three components).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1177:
URL: https://github.com/apache/opennlp/pull/1177#issuecomment-5033443083
All six addressed in 6d66ef7. Both generators now validate their arguments with the same fail-loud messages as the rest of the layer (pinned by tests), the two package-private data helpers carry full javadoc, and annotate() itself now short-circuits on the non-annotatable fast path via the existing guard, which also removed the duplicate check in collectFeatures. Good catch on the short circuit; it saves the lookup and the flag decode for every ordinary token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #1177:
URL: https://github.com/apache/opennlp/pull/1177#issuecomment-5025685334
> **Automated Claude Opus review. Human review will follow.**
**Overview.** Adds an emoji annotation layer (`EmojiAnnotation`/`EmojiAnnotations`/`EmojiAnnotator` + derived `EmojiFlags`, join hook `EmojiAnnotationJoin`) with a bundled 40-symbol data file, and opt-in wiring into NameFinder (feature-gen factory), Doccat (`FeatureGenerator`), and Sentiment (factory + context generator). Defaults are untouched and tests pin that. The layering and fail-loud loading are solid; the findings are mostly the usual categories: distr licensing, sibling consistency, commentary volume, constants, and duplication.
## Blocking
- `opennlp-distr/src/main/readme/LICENSE:615` — the binary distribution's LICENSE enumerates the bundled Unicode files (`confusables.txt`, `CaseFolding.txt`, `WordBreakProperty.txt`, …) but was not updated for `emoji-annotations.txt`, while root `LICENSE` and `src/license/NOTICE.template` were. All three levels must cover the new bundled data — add the same Unicode-derived-values entry there (and verify the distr NOTICE picks up the template change).
- `opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java:120` — `extractFeatures` throws IAE on `null` text, but the sibling `BagOfWordsFeatureGenerator` in the same package uses `Objects.requireNonNull` (NPE) for the same contract. Partial alignment is worse than none: either put the `null` contract (+ `@throws`) on the `FeatureGenerator` interface and align all implementations, or drop the check here. As written, swapping generators changes the exception type. The `@throws IllegalArgumentException` is also missing on the override's Javadoc.
- Commentary discipline — the "annotations are not `Dimension` constants" design essay appears **three times** (`EmojiAnnotation.java:35`, `EmojiAnnotations.java:44`, `EmojiAnnotator.java:29`, the last ~20 lines long) plus a fourth prose version in `normalizer.xml` and a fifth in the PR body. Keep one condensed 2–3 line version on `EmojiAnnotator` ("Annotations are per-symbol metadata, not text transforms; they are a parallel surface beside `Term`, see OPENNLP-1870.") and delete the rest. Same for the triplicated inline comment "Every annotatable symbol starts beyond ASCII (audited in EmojiAnnotationsTest)" (`EmojiFeatureGenerator.java:125`, `EmojiSentimentContextGenerator.java:57`, `EmojiAnnotationFeatureGenerator.java:60`) — extract the guard into one shared helper (e.g. package-private on `EmojiAnnotator`) and comment it once. A reader in five years does not need the migration rationale in three class headers.
- `opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java:156,246` — please declare `";"`, `"#"`, and `" "` as constants in this class; the sibling `EmojiEmoticons` in the same package already defines `FIELD_SEPARATOR`, `COMMENT_PREFIX`, and `MAPPING_CODE_POINT_SEPARATOR` with Javadoc for exactly this format. Either reuse or mirror them; also `stripped.split(" ")` runs inside the per-line loop.
- Feature vocabulary inconsistency — Doccat/Sentiment emit `emojiSentiment=` / `emojiEntityType=` (`EmojiFeatureGenerator.java:45-48`, `EmojiSentimentContextGenerator.java:44-47`, identical duplicated constant sets), while the NameFinder generator emits `emoji.sentiment=` / `emoji.type=` (`EmojiAnnotationFeatureGenerator.java:48-51`). These names freeze into trained models at release; decide one naming scheme now (or document why they differ) and hoist the shared prefixes to a single location instead of three copies.
- Javadoc — `EmojiSentimentFactory.java:31` and `EmojiAnnotationFeatureGeneratorFactory.java:36`: public constructors without Javadoc. The overrides `getContext` (`EmojiSentimentContextGenerator.java:52`), `extractFeatures`, `createFeatures`, `create()` should carry `{@inheritDoc}` plus at most one sentence.
## Minor
- `EmojiAnnotator.java:118` — a flag token is decoded twice: `EmojiFlags.isFlag(symbol)` (lenient decode) then `isoRegion(symbol).orElseThrow()` (strict decode). Expose the lenient decode package-private (returning `null`/`Optional`) and call it once; this is the per-token hot path.
- `EmojiSentimentContextGenerator.java:53` — the base `getContext(String[])` simply returns `text`; don't re-derive "every token is a feature" with a manual loop. Use `Collections.addAll(context, super.getContext(text))` so the subclass can't drift from the default.
- `EmojiSentimentContextGenerator.java:49` — hard-codes `new EmojiAnnotator()`; the other two generators expose an `(EmojiAnnotator)` constructor for a join-configured annotator. Add the same seam here, or the join layer is unreachable from sentiment.
- `EmojiFlags.java:112` — `isRegionalIndicator(int)` is `public` with no production caller outside the class. New public API is frozen at release; make it private until a real caller exists.
- `EmojiFlags.java:122` — for input `🇩` + `"E"` the error message says "got a lone indicator", which is wrong for that case; the length check conflates "nothing follows" with "a BMP char follows". Reword or split the check.
- `EmojiAnnotations.java:66` — lazy double-checked `volatile` load, while sibling `EmojiEmoticons` uses an eager `static final INSTANCE` holder. Align the loading pattern within the package, or say why lazy is needed here.
- Tests: `cp(int...)` is copy-pasted into **five** test classes — extract a shared test utility. The malformed-input batteries (`EmojiFlagsTest.malformedRegionalIndicatorSequencesFailLoud`, `malformedTagSequencesFailLoud`, `EmojiAnnotationsTest.parseFailsLoud*`) should be `@ParameterizedTest` so the failing input is identifiable. `assertTrue(!…)` → `assertFalse` in `EmojiAnnotationsTest` (`notes().isEmpty()`) and `EmojiAnnotatorTest` (`source().isEmpty()`). `EmojiAnnotation.Value` null `value`/`source` are untested (only empty).
- `emoji-annotations.txt:16-55` — ~40 lines of design rationale in the data-file header; trim to format + provenance + coverage claim, the rest belongs on OPENNLP-1870.
- Serialization: nothing here is `Serializable` and the doccat generator is manifest-re-instantiated via its no-arg constructor — verified, no serialVersionUID work needed.
- docbkx: indentation matches the surrounding sections and the claims (defaults unchanged, manifest re-creation) check out against the code.
## Process
- Defaults are untouched and pinned by tests, so no eval build is strictly required — but since three components gain new opt-in feature paths, a quick eval sanity run before merge is still cheap insurance.
- The feature-name freeze (blocking item above) is the one thing that cannot be fixed post-release; settle it before merge.
- If there are other emoji-layer PRs open for the ticket's follow-ups, please apply the `cp()` test-utility and constants findings there likewise.
PR title is fine as release-notes language.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1177:
URL: https://github.com/apache/opennlp/pull/1177#issuecomment-5031381660
Eval started. I'm going to flip on a couple drafts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "mawiesne (via GitHub)" <gi...@apache.org>.
mawiesne commented on PR #1177:
URL: https://github.com/apache/opennlp/pull/1177#issuecomment-5031092086
Nice, let's see what eval build tells us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on code in PR #1177:
URL: https://github.com/apache/opennlp/pull/1177#discussion_r3621744515
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.sentiment;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import opennlp.tools.util.normalizer.EmojiAnnotator;
+
+/**
+ * A {@link SentimentContextGenerator} that adds emoji annotation features to the default
+ * token context: for every annotated token the project-authored coarse sentiment score
+ * ({@code emojiSentiment=2}), the entity type ({@code emojiType=HEART}), the category
+ * ({@code emojiCategory=SMILEYS_AND_EMOTION}), and for flags the region ({@code emojiRegion=DE}).
+ * A crying-face or heart pictograph thereby becomes direct evidence for the sentiment model
+ * instead of an opaque token.
+ *
+ * <p>Strictly opt-in through the {@link SentimentFactory} seam: train with
+ * {@link EmojiSentimentFactory} (or any factory whose
+ * {@link SentimentFactory#createContextGenerator()} returns this class) and the same context is
+ * regenerated at prediction time; the default factory and existing models are unchanged.</p>
+ *
+ * @see EmojiSentimentFactory
+ * @see EmojiAnnotator
+ */
+public class EmojiSentimentContextGenerator extends SentimentContextGenerator {
+
+ private final EmojiAnnotator annotator;
+
+ /**
+ * Instantiates a context generator over the bundled and derived annotation layers.
+ */
+ public EmojiSentimentContextGenerator() {
+ this(new EmojiAnnotator());
+ }
+
+ /**
+ * Instantiates a context generator over a configured annotator, for example one with a
+ * gazetteer join.
+ *
+ * @param annotator The annotator to use. Must not be {@code null}.
+ * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ */
+ public EmojiSentimentContextGenerator(EmojiAnnotator annotator) {
+ if (annotator == null) {
+ throw new IllegalArgumentException("Annotator must not be null");
+ }
+ this.annotator = annotator;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Appends the emoji annotation features of every annotated token to the default token
+ * context.</p>
+ */
+ @Override
+ public String[] getContext(String[] text) {
+ final List<String> context = new ArrayList<>(text.length);
Review Comment:
done
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util.featuregen;
+
+import java.util.List;
+
+import opennlp.tools.util.normalizer.EmojiAnnotator;
+
+/**
+ * Generates features from the emoji annotation layer for the token at the given index: the
+ * project-authored coarse sentiment score ({@code emojiSentiment=2}), the entity type
+ * ({@code emojiType=HEART}), the document category ({@code emojiCategory=SMILEYS_AND_EMOTION}),
+ * and for flags the ISO 3166 region ({@code emojiRegion=DE}). For the name finder the entity
+ * type behaves like gazetteer evidence: a flag token carries {@code emojiType=FLAG} and its
+ * region without any dictionary.
+ *
+ * <p>Strictly opt-in: this generator only runs when a feature generation descriptor names
+ * {@link EmojiAnnotationFeatureGeneratorFactory}, so default descriptors and existing models are
+ * unchanged. A token that is not an annotated symbol contributes no features.</p>
+ *
+ * <p>Unlike most feature generators this one is stateless and thread-safe (when its annotator's
+ * join is): it keeps no adaptive data.</p>
+ *
+ * @see EmojiAnnotator
+ */
+public class EmojiAnnotationFeatureGenerator implements AdaptiveFeatureGenerator {
+
+ private final EmojiAnnotator annotator;
+
+ /**
+ * Instantiates a generator over the bundled and derived annotation layers.
+ */
+ public EmojiAnnotationFeatureGenerator() {
+ this(new EmojiAnnotator());
+ }
+
+ /**
+ * Instantiates a generator over a configured annotator, for example one with a gazetteer join.
+ *
+ * @param annotator The annotator to use. Must not be {@code null}.
+ * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ */
+ public EmojiAnnotationFeatureGenerator(EmojiAnnotator annotator) {
+ if (annotator == null) {
+ throw new IllegalArgumentException("Annotator must not be null");
+ }
+ this.annotator = annotator;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Adds one feature per present annotation attribute of the token at {@code index}.</p>
+ */
+ @Override
+ public void createFeatures(List<String> features, String[] tokens, int index,
+ String[] previousOutcomes) {
+ annotator.collectFeatures(tokens[index], features);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1177:
URL: https://github.com/apache/opennlp/pull/1177#issuecomment-5029350750
All findings addressed as of 33285b4. Point by point:
**Blocking**
- *distr LICENSE*: added the `emoji-annotations.txt` entry; the distr NOTICE comes from the already-updated template.
- *Null contract*: kept fail-loud IAE and added the missing `@throws`, rather than downgrading to `requireNonNull`.
- *Commentary*: done, including the suggested phrasing. One condensed note on `EmojiAnnotator`, copies deleted, the beyond-ASCII guard extracted into one shared helper.
- *Constants*: mirrored the `EmojiEmoticons` constants; the split no longer compiles a pattern per line.
- *Feature vocabulary*: fixed structurally rather than by renaming. Emission now lives in one method, `EmojiAnnotator.collectFeatures(token, sink)`; the generators are one-line delegations and the prefixes are private, so nothing is left to drift. Unified dot-free names: `emojiSentiment=`, `emojiType=`, `emojiCategory=`, `emojiRegion=`. Done now while 3.0 is unreleased since it shrinks the public surface.
- *Javadoc*: factory constructors documented, overrides carry `{@inheritDoc}` plus one sentence.
**Minor**
All applied: single lenient flag decode on the hot path, `Collections.addAll` over `super.getContext`, the `(EmojiAnnotator)` constructor seam on the sentiment generator, `isRegionalIndicator` made private (its direct test folded into the `isoRegion`/`isFlag` coverage), the lone-indicator message split into accurate cases, eager holder instead of DCL, shared `cp()` test utility, parameterized malformed-input batteries, `assertFalse` swaps, `Value` null tests, and the data-file header trimmed to format plus provenance.
**Process**
Build green I'll run eval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "mawiesne (via GitHub)" <gi...@apache.org>.
mawiesne merged PR #1177:
URL: https://github.com/apache/opennlp/pull/1177
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on code in PR #1177:
URL: https://github.com/apache/opennlp/pull/1177#discussion_r3621745680
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * The bundled-facts layer of the emoji annotation record store: license-clean, provenance-tagged
+ * attributes intrinsic to a pictograph (name, coarse sentiment, entity type, document category),
+ * loaded once from the project-authored {@code emoji-annotations.txt} resource. Each row of that
+ * file is one attribute of one symbol ({@code codepoints ; attribute ; value ; source ; notes}),
+ * so every value carries its own provenance and adding an attribute later is new rows plus loader
+ * support instead of a file-format break. The loader fails loud on an unknown attribute or a
+ * malformed row: the data and the code move together, so an unrecognized attribute is corruption,
+ * not extensibility.
+ *
+ * <p>Data licensing: the name values are the CLDR short names and the entity-type/category values
+ * are derived from the group and subgroup headers of the upstream {@code emoji-test.txt}
+ * (UTS #51, Unicode License V3, see the NOTICE file); the sentiment scores are original
+ * project judgments tagged {@code UNSPECIFIED}. No third-party sentiment data set is copied; in
+ * particular the Emoji Sentiment Ranking (CC BY-SA) is not used in any form.</p>
+ *
+ * <p>{@link EmojiAnnotator} is the accessor. Lookups strip U+FE0F (emoji presentation selector)
+ * because it does not change identity; bundled rows are keyed without it. Flag emoji have no
+ * bundled rows: region is derived, and gazetteer ids are never baked in.</p>
+ */
+public final class EmojiAnnotations {
+
+ private static final String RESOURCE = "emoji-annotations.txt";
+
+ /** Starts a comment line in {@code emoji-annotations.txt}. */
+ private static final String COMMENT_PREFIX = "#";
+
+ /**
+ * Field separator in {@code emoji-annotations.txt}
+ * ({@code codepoints ; attribute ; value ; source ; notes}).
+ */
+ private static final String FIELD_SEPARATOR = ";";
+
+ /**
+ * Separates hex code points inside the code point field. The bundled table format uses ASCII
+ * space ({@code U+0020}), not a general whitespace class.
+ */
+ private static final String MAPPING_CODE_POINT_SEPARATOR = " ";
+
+ // The records keyed by code point sequence, loaded once when this class initializes.
+ private static final Map<String, EmojiAnnotation> ANNOTATIONS = load();
+
+ private EmojiAnnotations() {
+ }
+
+ /**
+ * Returns the bundled annotation record of one emoji.
+ *
+ * @param symbol The code point sequence of one symbol, for example one {@link Term#original()}
+ * token. U+FE0F presentation selectors are ignored. Must not be {@code null}.
+ * @return The record, or empty when the bundled data does not annotate the symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the bundled data resource is missing.
+ * @throws UncheckedIOException if the bundled data resource cannot be read.
+ */
+ public static Optional<EmojiAnnotation> lookup(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ final String key = stripPresentationSelector(symbol);
+ if (key.isEmpty()) {
+ return Optional.empty();
+ }
+ return Optional.ofNullable(ANNOTATIONS.get(key));
+ }
+
+ // Removes every U+FE0F VARIATION SELECTOR-16; allocation-free when none is present.
+ // Package-visible so EmojiAnnotator keys derived-only records the same way.
+ static String stripPresentationSelector(CharSequence symbol) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #1177:
URL: https://github.com/apache/opennlp/pull/1177#discussion_r3620674053
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.sentiment;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import opennlp.tools.util.normalizer.EmojiAnnotator;
+
+/**
+ * A {@link SentimentContextGenerator} that adds emoji annotation features to the default
+ * token context: for every annotated token the project-authored coarse sentiment score
+ * ({@code emojiSentiment=2}), the entity type ({@code emojiType=HEART}), the category
+ * ({@code emojiCategory=SMILEYS_AND_EMOTION}), and for flags the region ({@code emojiRegion=DE}).
+ * A crying-face or heart pictograph thereby becomes direct evidence for the sentiment model
+ * instead of an opaque token.
+ *
+ * <p>Strictly opt-in through the {@link SentimentFactory} seam: train with
+ * {@link EmojiSentimentFactory} (or any factory whose
+ * {@link SentimentFactory#createContextGenerator()} returns this class) and the same context is
+ * regenerated at prediction time; the default factory and existing models are unchanged.</p>
+ *
+ * @see EmojiSentimentFactory
+ * @see EmojiAnnotator
+ */
+public class EmojiSentimentContextGenerator extends SentimentContextGenerator {
+
+ private final EmojiAnnotator annotator;
+
+ /**
+ * Instantiates a context generator over the bundled and derived annotation layers.
+ */
+ public EmojiSentimentContextGenerator() {
+ this(new EmojiAnnotator());
+ }
+
+ /**
+ * Instantiates a context generator over a configured annotator, for example one with a
+ * gazetteer join.
+ *
+ * @param annotator The annotator to use. Must not be {@code null}.
+ * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ */
+ public EmojiSentimentContextGenerator(EmojiAnnotator annotator) {
+ if (annotator == null) {
+ throw new IllegalArgumentException("Annotator must not be null");
+ }
+ this.annotator = annotator;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Appends the emoji annotation features of every annotated token to the default token
+ * context.</p>
+ */
+ @Override
+ public String[] getContext(String[] text) {
+ final List<String> context = new ArrayList<>(text.length);
Review Comment:
possible NPE? plz arg check?
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Assembles one {@link EmojiAnnotation} per symbol from bundled facts
+ * ({@link EmojiAnnotations}), derived facts ({@link EmojiFlags}), and optional joined facts
+ * ({@link EmojiAnnotationJoin}). {@link #annotate(Term)} keys on {@link Term#original()};
+ * non-emoji tokens and degenerate flag-shaped text return empty. Instances are immutable and
+ * thread-safe when the join is.
+ *
+ * <p>Annotations are per-symbol metadata, not text transforms; a parallel surface beside
+ * {@link Term} rather than {@link Dimension} constants. See OPENNLP-1870.</p>
+ */
+public final class EmojiAnnotator {
+
+ /** Feature-name prefix for the coarse sentiment score, for example {@code emojiSentiment=2}. */
+ private static final String FEATURE_SENTIMENT_PREFIX = "emojiSentiment=";
+
+ /** Feature-name prefix for the coarse entity type, for example {@code emojiType=HEART}. */
+ private static final String FEATURE_TYPE_PREFIX = "emojiType=";
+
+ /**
+ * Feature-name prefix for the document-category hint, for example
+ * {@code emojiCategory=SMILEYS_AND_EMOTION}.
+ */
+ private static final String FEATURE_CATEGORY_PREFIX = "emojiCategory=";
+
+ /** Feature-name prefix for a flag's ISO 3166 region, for example {@code emojiRegion=DE}. */
+ private static final String FEATURE_REGION_PREFIX = "emojiRegion=";
+
+ // Provenance tags of the derived facts; the mechanisms are defined by UTS #51.
+ private static final String FLAG_SEQUENCE = "UTS51:flag-sequence";
+ private static final String TAG_SEQUENCE = "UTS51:tag-sequence";
+ private static final String DERIVED_NOTE = "decoded from the code point sequence";
+
+ // The first non-ASCII code unit; every annotatable symbol starts at or beyond it.
+ private static final char FIRST_NON_ASCII = 0x80;
+
+ private final EmojiAnnotationJoin join;
+
+ /**
+ * Creates an annotator over the bundled and derived layers only.
+ */
+ public EmojiAnnotator() {
+ this.join = null;
+ }
+
+ /**
+ * Creates an annotator that also resolves joined facts through {@code join}.
+ *
+ * @param join The joined-facts hook, called while a record is assembled. Must not be
+ * {@code null}.
+ * @throws IllegalArgumentException if {@code join} is {@code null}.
+ */
+ public EmojiAnnotator(EmojiAnnotationJoin join) {
+ if (join == null) {
+ throw new IllegalArgumentException("Join must not be null");
+ }
+ this.join = join;
+ }
+
+ /**
+ * Annotates one term, keyed on its {@link Term#original() original} text (see the class note on
+ * why the original layer is the one annotations describe).
+ *
+ * @param term The term to annotate. Must not be {@code null}.
+ * @return The assembled record, or empty when the term is not an annotated symbol.
+ * @throws IllegalArgumentException if {@code term} is {@code null}.
+ * @throws IllegalStateException if the configured join violates its contract (returns
+ * {@code null} or a key colliding with an existing attribute).
+ */
+ public Optional<EmojiAnnotation> annotate(Term term) {
+ if (term == null) {
+ throw new IllegalArgumentException("Term must not be null");
+ }
+ return annotate(term.original());
+ }
+
+ /**
+ * Annotates one symbol.
+ *
+ * @param symbol The code point sequence of one symbol (one token). U+FE0F presentation
+ * selectors are ignored. Must not be {@code null}.
+ * @return The assembled record, or empty when {@code symbol} is not an annotated symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the configured join violates its contract (returns
+ * {@code null} or a key colliding with an existing attribute).
+ */
+ public Optional<EmojiAnnotation> annotate(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ final EmojiAnnotation bundled = EmojiAnnotations.lookup(symbol).orElse(null);
Review Comment:
if symbol is empty we can short circuit and direclty return Optional.empty no? (save some method invocations etc)
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util.featuregen;
+
+import java.util.List;
+
+import opennlp.tools.util.normalizer.EmojiAnnotator;
+
+/**
+ * Generates features from the emoji annotation layer for the token at the given index: the
+ * project-authored coarse sentiment score ({@code emojiSentiment=2}), the entity type
+ * ({@code emojiType=HEART}), the document category ({@code emojiCategory=SMILEYS_AND_EMOTION}),
+ * and for flags the ISO 3166 region ({@code emojiRegion=DE}). For the name finder the entity
+ * type behaves like gazetteer evidence: a flag token carries {@code emojiType=FLAG} and its
+ * region without any dictionary.
+ *
+ * <p>Strictly opt-in: this generator only runs when a feature generation descriptor names
+ * {@link EmojiAnnotationFeatureGeneratorFactory}, so default descriptors and existing models are
+ * unchanged. A token that is not an annotated symbol contributes no features.</p>
+ *
+ * <p>Unlike most feature generators this one is stateless and thread-safe (when its annotator's
+ * join is): it keeps no adaptive data.</p>
+ *
+ * @see EmojiAnnotator
+ */
+public class EmojiAnnotationFeatureGenerator implements AdaptiveFeatureGenerator {
+
+ private final EmojiAnnotator annotator;
+
+ /**
+ * Instantiates a generator over the bundled and derived annotation layers.
+ */
+ public EmojiAnnotationFeatureGenerator() {
+ this(new EmojiAnnotator());
+ }
+
+ /**
+ * Instantiates a generator over a configured annotator, for example one with a gazetteer join.
+ *
+ * @param annotator The annotator to use. Must not be {@code null}.
+ * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ */
+ public EmojiAnnotationFeatureGenerator(EmojiAnnotator annotator) {
+ if (annotator == null) {
+ throw new IllegalArgumentException("Annotator must not be null");
+ }
+ this.annotator = annotator;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Adds one feature per present annotation attribute of the token at {@code index}.</p>
+ */
+ @Override
+ public void createFeatures(List<String> features, String[] tokens, int index,
+ String[] previousOutcomes) {
+ annotator.collectFeatures(tokens[index], features);
Review Comment:
possible NPE? arg val?
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Assembles one {@link EmojiAnnotation} per symbol from bundled facts
+ * ({@link EmojiAnnotations}), derived facts ({@link EmojiFlags}), and optional joined facts
+ * ({@link EmojiAnnotationJoin}). {@link #annotate(Term)} keys on {@link Term#original()};
+ * non-emoji tokens and degenerate flag-shaped text return empty. Instances are immutable and
+ * thread-safe when the join is.
+ *
+ * <p>Annotations are per-symbol metadata, not text transforms; a parallel surface beside
+ * {@link Term} rather than {@link Dimension} constants. See OPENNLP-1870.</p>
+ */
+public final class EmojiAnnotator {
+
+ /** Feature-name prefix for the coarse sentiment score, for example {@code emojiSentiment=2}. */
+ private static final String FEATURE_SENTIMENT_PREFIX = "emojiSentiment=";
+
+ /** Feature-name prefix for the coarse entity type, for example {@code emojiType=HEART}. */
+ private static final String FEATURE_TYPE_PREFIX = "emojiType=";
+
+ /**
+ * Feature-name prefix for the document-category hint, for example
+ * {@code emojiCategory=SMILEYS_AND_EMOTION}.
+ */
+ private static final String FEATURE_CATEGORY_PREFIX = "emojiCategory=";
+
+ /** Feature-name prefix for a flag's ISO 3166 region, for example {@code emojiRegion=DE}. */
+ private static final String FEATURE_REGION_PREFIX = "emojiRegion=";
+
+ // Provenance tags of the derived facts; the mechanisms are defined by UTS #51.
+ private static final String FLAG_SEQUENCE = "UTS51:flag-sequence";
+ private static final String TAG_SEQUENCE = "UTS51:tag-sequence";
+ private static final String DERIVED_NOTE = "decoded from the code point sequence";
+
+ // The first non-ASCII code unit; every annotatable symbol starts at or beyond it.
+ private static final char FIRST_NON_ASCII = 0x80;
+
+ private final EmojiAnnotationJoin join;
+
+ /**
+ * Creates an annotator over the bundled and derived layers only.
+ */
+ public EmojiAnnotator() {
+ this.join = null;
+ }
+
+ /**
+ * Creates an annotator that also resolves joined facts through {@code join}.
+ *
+ * @param join The joined-facts hook, called while a record is assembled. Must not be
+ * {@code null}.
+ * @throws IllegalArgumentException if {@code join} is {@code null}.
+ */
+ public EmojiAnnotator(EmojiAnnotationJoin join) {
+ if (join == null) {
+ throw new IllegalArgumentException("Join must not be null");
+ }
+ this.join = join;
+ }
+
+ /**
+ * Annotates one term, keyed on its {@link Term#original() original} text (see the class note on
+ * why the original layer is the one annotations describe).
+ *
+ * @param term The term to annotate. Must not be {@code null}.
+ * @return The assembled record, or empty when the term is not an annotated symbol.
+ * @throws IllegalArgumentException if {@code term} is {@code null}.
+ * @throws IllegalStateException if the configured join violates its contract (returns
+ * {@code null} or a key colliding with an existing attribute).
+ */
+ public Optional<EmojiAnnotation> annotate(Term term) {
+ if (term == null) {
+ throw new IllegalArgumentException("Term must not be null");
+ }
+ return annotate(term.original());
+ }
+
+ /**
+ * Annotates one symbol.
+ *
+ * @param symbol The code point sequence of one symbol (one token). U+FE0F presentation
+ * selectors are ignored. Must not be {@code null}.
+ * @return The assembled record, or empty when {@code symbol} is not an annotated symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the configured join violates its contract (returns
+ * {@code null} or a key colliding with an existing attribute).
+ */
+ public Optional<EmojiAnnotation> annotate(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ final EmojiAnnotation bundled = EmojiAnnotations.lookup(symbol).orElse(null);
Review Comment:
i.e. using isAnnotatableToken(CharSequence token) ?
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * The bundled-facts layer of the emoji annotation record store: license-clean, provenance-tagged
+ * attributes intrinsic to a pictograph (name, coarse sentiment, entity type, document category),
+ * loaded once from the project-authored {@code emoji-annotations.txt} resource. Each row of that
+ * file is one attribute of one symbol ({@code codepoints ; attribute ; value ; source ; notes}),
+ * so every value carries its own provenance and adding an attribute later is new rows plus loader
+ * support instead of a file-format break. The loader fails loud on an unknown attribute or a
+ * malformed row: the data and the code move together, so an unrecognized attribute is corruption,
+ * not extensibility.
+ *
+ * <p>Data licensing: the name values are the CLDR short names and the entity-type/category values
+ * are derived from the group and subgroup headers of the upstream {@code emoji-test.txt}
+ * (UTS #51, Unicode License V3, see the NOTICE file); the sentiment scores are original
+ * project judgments tagged {@code UNSPECIFIED}. No third-party sentiment data set is copied; in
+ * particular the Emoji Sentiment Ranking (CC BY-SA) is not used in any form.</p>
+ *
+ * <p>{@link EmojiAnnotator} is the accessor. Lookups strip U+FE0F (emoji presentation selector)
+ * because it does not change identity; bundled rows are keyed without it. Flag emoji have no
+ * bundled rows: region is derived, and gazetteer ids are never baked in.</p>
+ */
+public final class EmojiAnnotations {
+
+ private static final String RESOURCE = "emoji-annotations.txt";
+
+ /** Starts a comment line in {@code emoji-annotations.txt}. */
+ private static final String COMMENT_PREFIX = "#";
+
+ /**
+ * Field separator in {@code emoji-annotations.txt}
+ * ({@code codepoints ; attribute ; value ; source ; notes}).
+ */
+ private static final String FIELD_SEPARATOR = ";";
+
+ /**
+ * Separates hex code points inside the code point field. The bundled table format uses ASCII
+ * space ({@code U+0020}), not a general whitespace class.
+ */
+ private static final String MAPPING_CODE_POINT_SEPARATOR = " ";
+
+ // The records keyed by code point sequence, loaded once when this class initializes.
+ private static final Map<String, EmojiAnnotation> ANNOTATIONS = load();
+
+ private EmojiAnnotations() {
+ }
+
+ /**
+ * Returns the bundled annotation record of one emoji.
+ *
+ * @param symbol The code point sequence of one symbol, for example one {@link Term#original()}
+ * token. U+FE0F presentation selectors are ignored. Must not be {@code null}.
+ * @return The record, or empty when the bundled data does not annotate the symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the bundled data resource is missing.
+ * @throws UncheckedIOException if the bundled data resource cannot be read.
+ */
+ public static Optional<EmojiAnnotation> lookup(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ final String key = stripPresentationSelector(symbol);
+ if (key.isEmpty()) {
+ return Optional.empty();
+ }
+ return Optional.ofNullable(ANNOTATIONS.get(key));
+ }
+
+ // Removes every U+FE0F VARIATION SELECTOR-16; allocation-free when none is present.
+ // Package-visible so EmojiAnnotator keys derived-only records the same way.
+ static String stripPresentationSelector(CharSequence symbol) {
Review Comment:
Javadoc?
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * The bundled-facts layer of the emoji annotation record store: license-clean, provenance-tagged
+ * attributes intrinsic to a pictograph (name, coarse sentiment, entity type, document category),
+ * loaded once from the project-authored {@code emoji-annotations.txt} resource. Each row of that
+ * file is one attribute of one symbol ({@code codepoints ; attribute ; value ; source ; notes}),
+ * so every value carries its own provenance and adding an attribute later is new rows plus loader
+ * support instead of a file-format break. The loader fails loud on an unknown attribute or a
+ * malformed row: the data and the code move together, so an unrecognized attribute is corruption,
+ * not extensibility.
+ *
+ * <p>Data licensing: the name values are the CLDR short names and the entity-type/category values
+ * are derived from the group and subgroup headers of the upstream {@code emoji-test.txt}
+ * (UTS #51, Unicode License V3, see the NOTICE file); the sentiment scores are original
+ * project judgments tagged {@code UNSPECIFIED}. No third-party sentiment data set is copied; in
+ * particular the Emoji Sentiment Ranking (CC BY-SA) is not used in any form.</p>
+ *
+ * <p>{@link EmojiAnnotator} is the accessor. Lookups strip U+FE0F (emoji presentation selector)
+ * because it does not change identity; bundled rows are keyed without it. Flag emoji have no
+ * bundled rows: region is derived, and gazetteer ids are never baked in.</p>
+ */
+public final class EmojiAnnotations {
+
+ private static final String RESOURCE = "emoji-annotations.txt";
+
+ /** Starts a comment line in {@code emoji-annotations.txt}. */
+ private static final String COMMENT_PREFIX = "#";
+
+ /**
+ * Field separator in {@code emoji-annotations.txt}
+ * ({@code codepoints ; attribute ; value ; source ; notes}).
+ */
+ private static final String FIELD_SEPARATOR = ";";
+
+ /**
+ * Separates hex code points inside the code point field. The bundled table format uses ASCII
+ * space ({@code U+0020}), not a general whitespace class.
+ */
+ private static final String MAPPING_CODE_POINT_SEPARATOR = " ";
+
+ // The records keyed by code point sequence, loaded once when this class initializes.
+ private static final Map<String, EmojiAnnotation> ANNOTATIONS = load();
+
+ private EmojiAnnotations() {
+ }
+
+ /**
+ * Returns the bundled annotation record of one emoji.
+ *
+ * @param symbol The code point sequence of one symbol, for example one {@link Term#original()}
+ * token. U+FE0F presentation selectors are ignored. Must not be {@code null}.
+ * @return The record, or empty when the bundled data does not annotate the symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the bundled data resource is missing.
+ * @throws UncheckedIOException if the bundled data resource cannot be read.
+ */
+ public static Optional<EmojiAnnotation> lookup(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ final String key = stripPresentationSelector(symbol);
+ if (key.isEmpty()) {
+ return Optional.empty();
+ }
+ return Optional.ofNullable(ANNOTATIONS.get(key));
+ }
+
+ // Removes every U+FE0F VARIATION SELECTOR-16; allocation-free when none is present.
+ // Package-visible so EmojiAnnotator keys derived-only records the same way.
+ static String stripPresentationSelector(CharSequence symbol) {
+ final int length = symbol.length();
+ int i = 0;
+ while (i < length && symbol.charAt(i) != 0xFE0F) {
+ i++;
+ }
+ if (i == length) {
+ return symbol.toString();
+ }
+ final StringBuilder stripped = new StringBuilder(length - 1);
+ stripped.append(symbol, 0, i);
+ for (int k = i + 1; k < length; k++) {
+ final char c = symbol.charAt(k);
+ if (c != 0xFE0F) {
+ stripped.append(c);
+ }
+ }
+ return stripped.toString();
+ }
+
+ private static Map<String, EmojiAnnotation> load() {
+ try (InputStream in = EmojiAnnotations.class.getResourceAsStream(RESOURCE)) {
+ if (in == null) {
+ throw new IllegalStateException("Missing emoji annotation data resource: " + RESOURCE);
+ }
+ return parse(in);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to read emoji annotation data resource " + RESOURCE, e);
+ }
+ }
+
+ // Package-private so the malformed-data handling can be exercised without the bundled resource.
+ // Parses rows of "codepoints ; attribute ; value ; source ; notes" with space-separated
+ // hexadecimal code points; '#' starts a comment line. The notes column is the fifth and final
+ // field, so it may contain ';'.
+ static Map<String, EmojiAnnotation> parse(InputStream in) throws IOException {
Review Comment:
JavaDoc=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and Sentiment (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on code in PR #1177:
URL: https://github.com/apache/opennlp/pull/1177#discussion_r3621742298
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Assembles one {@link EmojiAnnotation} per symbol from bundled facts
+ * ({@link EmojiAnnotations}), derived facts ({@link EmojiFlags}), and optional joined facts
+ * ({@link EmojiAnnotationJoin}). {@link #annotate(Term)} keys on {@link Term#original()};
+ * non-emoji tokens and degenerate flag-shaped text return empty. Instances are immutable and
+ * thread-safe when the join is.
+ *
+ * <p>Annotations are per-symbol metadata, not text transforms; a parallel surface beside
+ * {@link Term} rather than {@link Dimension} constants. See OPENNLP-1870.</p>
+ */
+public final class EmojiAnnotator {
+
+ /** Feature-name prefix for the coarse sentiment score, for example {@code emojiSentiment=2}. */
+ private static final String FEATURE_SENTIMENT_PREFIX = "emojiSentiment=";
+
+ /** Feature-name prefix for the coarse entity type, for example {@code emojiType=HEART}. */
+ private static final String FEATURE_TYPE_PREFIX = "emojiType=";
+
+ /**
+ * Feature-name prefix for the document-category hint, for example
+ * {@code emojiCategory=SMILEYS_AND_EMOTION}.
+ */
+ private static final String FEATURE_CATEGORY_PREFIX = "emojiCategory=";
+
+ /** Feature-name prefix for a flag's ISO 3166 region, for example {@code emojiRegion=DE}. */
+ private static final String FEATURE_REGION_PREFIX = "emojiRegion=";
+
+ // Provenance tags of the derived facts; the mechanisms are defined by UTS #51.
+ private static final String FLAG_SEQUENCE = "UTS51:flag-sequence";
+ private static final String TAG_SEQUENCE = "UTS51:tag-sequence";
+ private static final String DERIVED_NOTE = "decoded from the code point sequence";
+
+ // The first non-ASCII code unit; every annotatable symbol starts at or beyond it.
+ private static final char FIRST_NON_ASCII = 0x80;
+
+ private final EmojiAnnotationJoin join;
+
+ /**
+ * Creates an annotator over the bundled and derived layers only.
+ */
+ public EmojiAnnotator() {
+ this.join = null;
+ }
+
+ /**
+ * Creates an annotator that also resolves joined facts through {@code join}.
+ *
+ * @param join The joined-facts hook, called while a record is assembled. Must not be
+ * {@code null}.
+ * @throws IllegalArgumentException if {@code join} is {@code null}.
+ */
+ public EmojiAnnotator(EmojiAnnotationJoin join) {
+ if (join == null) {
+ throw new IllegalArgumentException("Join must not be null");
+ }
+ this.join = join;
+ }
+
+ /**
+ * Annotates one term, keyed on its {@link Term#original() original} text (see the class note on
+ * why the original layer is the one annotations describe).
+ *
+ * @param term The term to annotate. Must not be {@code null}.
+ * @return The assembled record, or empty when the term is not an annotated symbol.
+ * @throws IllegalArgumentException if {@code term} is {@code null}.
+ * @throws IllegalStateException if the configured join violates its contract (returns
+ * {@code null} or a key colliding with an existing attribute).
+ */
+ public Optional<EmojiAnnotation> annotate(Term term) {
+ if (term == null) {
+ throw new IllegalArgumentException("Term must not be null");
+ }
+ return annotate(term.original());
+ }
+
+ /**
+ * Annotates one symbol.
+ *
+ * @param symbol The code point sequence of one symbol (one token). U+FE0F presentation
+ * selectors are ignored. Must not be {@code null}.
+ * @return The assembled record, or empty when {@code symbol} is not an annotated symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the configured join violates its contract (returns
+ * {@code null} or a key colliding with an existing attribute).
+ */
+ public Optional<EmojiAnnotation> annotate(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ final EmojiAnnotation bundled = EmojiAnnotations.lookup(symbol).orElse(null);
Review Comment:
fixed per suggestion
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * The bundled-facts layer of the emoji annotation record store: license-clean, provenance-tagged
+ * attributes intrinsic to a pictograph (name, coarse sentiment, entity type, document category),
+ * loaded once from the project-authored {@code emoji-annotations.txt} resource. Each row of that
+ * file is one attribute of one symbol ({@code codepoints ; attribute ; value ; source ; notes}),
+ * so every value carries its own provenance and adding an attribute later is new rows plus loader
+ * support instead of a file-format break. The loader fails loud on an unknown attribute or a
+ * malformed row: the data and the code move together, so an unrecognized attribute is corruption,
+ * not extensibility.
+ *
+ * <p>Data licensing: the name values are the CLDR short names and the entity-type/category values
+ * are derived from the group and subgroup headers of the upstream {@code emoji-test.txt}
+ * (UTS #51, Unicode License V3, see the NOTICE file); the sentiment scores are original
+ * project judgments tagged {@code UNSPECIFIED}. No third-party sentiment data set is copied; in
+ * particular the Emoji Sentiment Ranking (CC BY-SA) is not used in any form.</p>
+ *
+ * <p>{@link EmojiAnnotator} is the accessor. Lookups strip U+FE0F (emoji presentation selector)
+ * because it does not change identity; bundled rows are keyed without it. Flag emoji have no
+ * bundled rows: region is derived, and gazetteer ids are never baked in.</p>
+ */
+public final class EmojiAnnotations {
+
+ private static final String RESOURCE = "emoji-annotations.txt";
+
+ /** Starts a comment line in {@code emoji-annotations.txt}. */
+ private static final String COMMENT_PREFIX = "#";
+
+ /**
+ * Field separator in {@code emoji-annotations.txt}
+ * ({@code codepoints ; attribute ; value ; source ; notes}).
+ */
+ private static final String FIELD_SEPARATOR = ";";
+
+ /**
+ * Separates hex code points inside the code point field. The bundled table format uses ASCII
+ * space ({@code U+0020}), not a general whitespace class.
+ */
+ private static final String MAPPING_CODE_POINT_SEPARATOR = " ";
+
+ // The records keyed by code point sequence, loaded once when this class initializes.
+ private static final Map<String, EmojiAnnotation> ANNOTATIONS = load();
+
+ private EmojiAnnotations() {
+ }
+
+ /**
+ * Returns the bundled annotation record of one emoji.
+ *
+ * @param symbol The code point sequence of one symbol, for example one {@link Term#original()}
+ * token. U+FE0F presentation selectors are ignored. Must not be {@code null}.
+ * @return The record, or empty when the bundled data does not annotate the symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the bundled data resource is missing.
+ * @throws UncheckedIOException if the bundled data resource cannot be read.
+ */
+ public static Optional<EmojiAnnotation> lookup(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ final String key = stripPresentationSelector(symbol);
+ if (key.isEmpty()) {
+ return Optional.empty();
+ }
+ return Optional.ofNullable(ANNOTATIONS.get(key));
+ }
+
+ // Removes every U+FE0F VARIATION SELECTOR-16; allocation-free when none is present.
+ // Package-visible so EmojiAnnotator keys derived-only records the same way.
+ static String stripPresentationSelector(CharSequence symbol) {
+ final int length = symbol.length();
+ int i = 0;
+ while (i < length && symbol.charAt(i) != 0xFE0F) {
+ i++;
+ }
+ if (i == length) {
+ return symbol.toString();
+ }
+ final StringBuilder stripped = new StringBuilder(length - 1);
+ stripped.append(symbol, 0, i);
+ for (int k = i + 1; k < length; k++) {
+ final char c = symbol.charAt(k);
+ if (c != 0xFE0F) {
+ stripped.append(c);
+ }
+ }
+ return stripped.toString();
+ }
+
+ private static Map<String, EmojiAnnotation> load() {
+ try (InputStream in = EmojiAnnotations.class.getResourceAsStream(RESOURCE)) {
+ if (in == null) {
+ throw new IllegalStateException("Missing emoji annotation data resource: " + RESOURCE);
+ }
+ return parse(in);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to read emoji annotation data resource " + RESOURCE, e);
+ }
+ }
+
+ // Package-private so the malformed-data handling can be exercised without the bundled resource.
+ // Parses rows of "codepoints ; attribute ; value ; source ; notes" with space-separated
+ // hexadecimal code points; '#' starts a comment line. The notes column is the fifth and final
+ // field, so it may contain ';'.
+ static Map<String, EmojiAnnotation> parse(InputStream in) throws IOException {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org