You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/07/16 10:51:18 UTC
(opennlp) 01/05: OPENNLP-1870: Emoji annotation record store with bundled facts and audit
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a commit to branch OPENNLP-1870
in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit b7fbcd9bb9000287e2cecefe7a6b9a40be37056d
Author: Kristian Rickert <kr...@gmail.com>
AuthorDate: Mon Jul 6 22:00:41 2026 -0400
OPENNLP-1870: Emoji annotation record store with bundled facts and audit
Adds the bundled-facts layer of the emoji annotation record store: a
provenance-tagged, project-authored attribute table (emoji-annotations.txt)
loaded by EmojiAnnotations into per-symbol EmojiAnnotation records.
- Attribute-row format (codepoints ; attribute ; value ; source ; notes), so
adding an attribute later is new rows plus loader support instead of a file
format break, and every value carries its own provenance.
- Attributes: name (CLDR short name, CLDR:annotation), sentiment (project
judgment on a coarse -2..2 scale, UNSPECIFIED), entityType and category
(assigned from the emoji-test.txt subgroup/group headers, UCD:emoji-test).
No third-party sentiment data set is copied; in particular the CC BY-SA
Emoji Sentiment Ranking is not used in any form.
- The loader fails loud on an unknown attribute or malformed row
(IllegalArgumentException naming the resource and line): the data and the
code move together, so an unrecognized attribute is corruption, not
extensibility. Values are validated into their typed domains at load.
- Lookups strip U+FE0F, since presentation selection does not change a
symbol's identity.
- Audit tests pin coverage (every EMOJI-direction source of
emoji-emoticons.txt has a complete four-attribute record), per-attribute
provenance tags, typed domains, and the record/row counts (40/160).
- LICENSE/NOTICE/NOTICE.template attribution for the Unicode-derived values
(CLDR short names, emoji-test.txt group/subgroup classifications) in the
otherwise project-authored table.
Flag emoji intentionally have no bundled rows; their region decodes from the
code point sequence itself in the derived-facts layer (next commit), and
gazetteer identifiers are never baked into bundled data.
Part of the OPENNLP-1852 epic. Stacks on OPENNLP-1869.
---
LICENSE | 8 +-
NOTICE | 9 +
.../tools/util/normalizer/EmojiAnnotation.java | 185 ++++++++++++++
.../tools/util/normalizer/EmojiAnnotations.java | 254 +++++++++++++++++++
.../tools/util/normalizer/EmojiCategory.java | 62 +++++
.../tools/util/normalizer/EmojiEntityType.java | 57 +++++
.../tools/util/normalizer/emoji-annotations.txt | 223 +++++++++++++++++
.../util/normalizer/EmojiAnnotationsTest.java | 269 +++++++++++++++++++++
src/license/NOTICE.template | 9 +
9 files changed, 1074 insertions(+), 2 deletions(-)
diff --git a/LICENSE b/LICENSE
index 1b92a4dc6..e77ddad5f 100644
--- a/LICENSE
+++ b/LICENSE
@@ -375,9 +375,13 @@ The following license applies to the bundled Unicode data files in
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/tokenize/uax29
(WordBreakProperty.txt, ExtendedPictographic.txt),
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer
-(confusables.txt, CaseFolding.txt), and
+(confusables.txt, CaseFolding.txt),
opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/tokenize/uax29
-(WordBreakTest.txt):
+(WordBreakTest.txt), and to the Unicode-derived values (the CLDR short names
+and the emoji-test.txt group/subgroup classifications) in the otherwise
+project-authored
+opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
+(see NOTICE):
UNICODE LICENSE V3
diff --git a/NOTICE b/NOTICE
index bca534b1f..c503a5f8f 100644
--- a/NOTICE
+++ b/NOTICE
@@ -123,6 +123,15 @@ top of each bundled file. These files are distributed under the Unicode License
V3, the full text of which is reproduced in the LICENSE file accompanying this
distribution.
+The project-authored annotation table
+opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
+is licensed under the Apache License 2.0, but its name values (the CLDR short
+names) and its entity-type/category derivations are transcribed from the
+upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51,
+version 17.0), distributed under the Unicode License V3 reproduced in the
+LICENSE file. Its sentiment scores are original project judgments; no
+third-party sentiment data set is included.
+
Copyright (c) 1991-2025 Unicode, Inc. All rights reserved.
============================================================================
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
new file mode 100644
index 000000000..b19ce8a4e
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+
+/**
+ * The annotation record of one emoji: an extensible per-symbol store of attribute values, each
+ * carrying its own provenance. A record is assembled from up to three strictly separated layers:
+ * bundled facts from {@code emoji-annotations.txt} (see {@link EmojiAnnotations}), derived facts
+ * computed from the code point sequence itself, and joined facts resolved at run time against
+ * user-installed data. The typed accessors ({@link #name()}, {@link #sentiment()},
+ * {@link #entityType()}, {@link #category()}) are projections of the {@link #attributes()} map,
+ * so an attribute added later is new rows plus an accessor, not a shape change.
+ *
+ * <p>Every attribute is optional: a consumer that only wants one dimension pays nothing for the
+ * rest, and a record never fabricates a value it has no source for. Instances are immutable and
+ * thread-safe.</p>
+ *
+ * <p>These annotations are per-symbol <em>metadata</em>, not text transforms, which is why they
+ * are not {@link Dimension} constants; see {@link EmojiAnnotations} for the design note.</p>
+ *
+ * @param symbol The annotated code point sequence, without the U+FE0F presentation selector.
+ * @param attributes The attribute values keyed by attribute name ({@link #NAME},
+ * {@link #SENTIMENT}, {@link #ENTITY_TYPE}, {@link #CATEGORY}).
+ */
+public record EmojiAnnotation(String symbol, Map<String, Value> attributes) {
+
+ /** The attribute key of the human-readable name (the CLDR short name). */
+ public static final String NAME = "name";
+
+ /** The attribute key of the project-authored coarse sentiment score, an integer in -2..2. */
+ public static final String SENTIMENT = "sentiment";
+
+ /** The attribute key of the coarse {@link EmojiEntityType entity type}. */
+ public static final String ENTITY_TYPE = "entityType";
+
+ /** The attribute key of the {@link EmojiCategory document-category hint}. */
+ public static final String CATEGORY = "category";
+
+ /**
+ * One attribute value with its provenance, mirroring one row of the bundled data file.
+ *
+ * @param value The attribute value. Must not be {@code null} or empty.
+ * @param source The provenance tag, for example {@code CLDR:annotation}, {@code UCD:emoji-test},
+ * or {@code UNSPECIFIED} (the explicit marker for a project judgment). Must not be
+ * {@code null} or empty.
+ * @param notes Free-text notes, for example the upstream subgroup a value was derived from.
+ * Must not be {@code null}; may be empty.
+ */
+ public record Value(String value, String source, String notes) {
+
+ public Value {
+ if (value == null || value.isEmpty()) {
+ throw new IllegalArgumentException("Value must not be null or empty");
+ }
+ if (source == null || source.isEmpty()) {
+ throw new IllegalArgumentException("Source must not be null or empty");
+ }
+ if (notes == null) {
+ throw new IllegalArgumentException("Notes must not be null");
+ }
+ }
+ }
+
+ /**
+ * Creates a record; normally records come from {@link EmojiAnnotations#lookup(CharSequence)}.
+ *
+ * @param symbol The annotated code point sequence. Must not be {@code null} or empty.
+ * @param attributes The attribute values keyed by attribute name. Must not be {@code null} or
+ * contain {@code null} keys or values; it is defensively copied.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null} or empty, or if
+ * {@code attributes} is {@code null} or contains a {@code null} key or value.
+ */
+ public EmojiAnnotation {
+ if (symbol == null || symbol.isEmpty()) {
+ throw new IllegalArgumentException("Symbol must not be null or empty");
+ }
+ if (attributes == null) {
+ throw new IllegalArgumentException("Attributes must not be null");
+ }
+ for (final Map.Entry<String, Value> entry : attributes.entrySet()) {
+ if (entry.getKey() == null || entry.getValue() == null) {
+ throw new IllegalArgumentException(
+ "Attributes must not contain a null key or value, got: " + attributes);
+ }
+ }
+ attributes = Map.copyOf(attributes);
+ }
+
+ /**
+ * Returns the value of one attribute with its provenance.
+ *
+ * @param name The attribute name, for example {@link #NAME}. Must not be {@code null}.
+ * @return The value, or empty when this record does not carry the attribute.
+ * @throws IllegalArgumentException if {@code name} is {@code null}.
+ */
+ public Optional<Value> attribute(String name) {
+ if (name == null) {
+ throw new IllegalArgumentException("Name must not be null");
+ }
+ return Optional.ofNullable(attributes.get(name));
+ }
+
+ /**
+ * {@return the human-readable name (the CLDR short name), or empty when not annotated}
+ */
+ public Optional<String> name() {
+ final Value value = attributes.get(NAME);
+ return value == null ? Optional.empty() : Optional.of(value.value());
+ }
+
+ /**
+ * {@return the project-authored coarse sentiment score in -2..2, or empty when not annotated}
+ *
+ * @throws IllegalStateException if the stored value is not an integer, which cannot happen for
+ * records loaded from the bundled data (the loader validates it).
+ */
+ public OptionalInt sentiment() {
+ final Value value = attributes.get(SENTIMENT);
+ if (value == null) {
+ return OptionalInt.empty();
+ }
+ try {
+ return OptionalInt.of(Integer.parseInt(value.value()));
+ } catch (NumberFormatException e) {
+ throw new IllegalStateException("Sentiment value '" + value.value() + "' of symbol '"
+ + symbol + "' is not an integer", e);
+ }
+ }
+
+ /**
+ * {@return the coarse entity type, or empty when not annotated}
+ *
+ * @throws IllegalStateException if the stored value is not an {@link EmojiEntityType} constant,
+ * which cannot happen for records loaded from the bundled data (the loader validates it).
+ */
+ public Optional<EmojiEntityType> entityType() {
+ final Value value = attributes.get(ENTITY_TYPE);
+ if (value == null) {
+ return Optional.empty();
+ }
+ try {
+ return Optional.of(EmojiEntityType.valueOf(value.value()));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalStateException("Entity type value '" + value.value() + "' of symbol '"
+ + symbol + "' is not an EmojiEntityType constant", e);
+ }
+ }
+
+ /**
+ * {@return the document-category hint, or empty when not annotated}
+ *
+ * @throws IllegalStateException if the stored value is not an {@link EmojiCategory} constant,
+ * which cannot happen for records loaded from the bundled data (the loader validates it).
+ */
+ public Optional<EmojiCategory> category() {
+ final Value value = attributes.get(CATEGORY);
+ if (value == null) {
+ return Optional.empty();
+ }
+ try {
+ return Optional.of(EmojiCategory.valueOf(value.value()));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalStateException("Category value '" + value.value() + "' of symbol '"
+ + symbol + "' is not an EmojiCategory constant", e);
+ }
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
new file mode 100644
index 000000000..a191f803f
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * The bundled-facts layer of the emoji annotation record store: license-clean, provenance-tagged
+ * attributes intrinsic to a pictograph (name, coarse sentiment, entity type, document category),
+ * loaded once from the project-authored {@code emoji-annotations.txt} resource. Each row of that
+ * file is one attribute of one symbol ({@code codepoints ; attribute ; value ; source ; notes}),
+ * so every value carries its own provenance and adding an attribute later is new rows plus loader
+ * support instead of a file-format break. The loader fails loud on an unknown attribute or a
+ * malformed row: the data and the code move together, so an unrecognized attribute is corruption,
+ * not extensibility.
+ *
+ * <p>Data licensing: the name values are the CLDR short names and the entity-type/category values
+ * are derived from the group and subgroup headers of the upstream {@code emoji-test.txt}
+ * (UTS #51, Unicode License V3, see the NOTICE file); the sentiment scores are original
+ * project judgments tagged {@code UNSPECIFIED}. No third-party sentiment data set is copied; in
+ * particular the Emoji Sentiment Ranking (CC BY-SA) is not used in any form.</p>
+ *
+ * <p>Design note: these annotations are surfaced beside the {@link Term} model rather than as new
+ * {@link Dimension} constants. A {@code Dimension} is a character-level text <em>transform</em>
+ * whose result is another string layer of the token (each layer feeds the next, and
+ * {@link Term#peel()}/{@link Term#normalized()} walk that stack), while an annotation is typed,
+ * per-symbol <em>metadata</em> about the original pictograph: it has no place in the transform
+ * pipeline, would not compose with the layers below it, and would force typed facts through a
+ * stringly {@code Map<Dimension, String>}. The parallel accessor surface is
+ * {@code EmojiAnnotator}, which consumes a {@code Term} and returns the record.</p>
+ *
+ * <p>Lookups strip U+FE0F VARIATION SELECTOR-16 (the emoji presentation selector) from the queried
+ * sequence, because presentation selection does not change a symbol's identity; the bundled rows
+ * are keyed without it. Flag emoji intentionally have no bundled rows: their region decodes from
+ * the code point sequence itself (the derived-facts layer), and gazetteer identifiers are never
+ * baked into bundled data.</p>
+ */
+public final class EmojiAnnotations {
+
+ private static final String RESOURCE = "emoji-annotations.txt";
+
+ // The records keyed by code point sequence, loaded lazily on first use and cached. Volatile so
+ // the fully built, immutable map is safely published to every thread that observes it non-null.
+ private static volatile Map<String, EmojiAnnotation> annotations;
+
+ private EmojiAnnotations() {
+ }
+
+ /**
+ * Returns the bundled annotation record of one emoji.
+ *
+ * @param symbol The code point sequence of one symbol, for example one {@link Term#original()}
+ * token. U+FE0F presentation selectors are ignored. Must not be {@code null}.
+ * @return The record, or empty when the bundled data does not annotate the symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the bundled data resource is missing.
+ * @throws UncheckedIOException if the bundled data resource cannot be read.
+ */
+ public static Optional<EmojiAnnotation> lookup(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ final String key = stripPresentationSelector(symbol);
+ if (key.isEmpty()) {
+ return Optional.empty();
+ }
+ return Optional.ofNullable(annotations().get(key));
+ }
+
+ // Removes every U+FE0F VARIATION SELECTOR-16; allocation-free when none is present.
+ private static String stripPresentationSelector(CharSequence symbol) {
+ final int length = symbol.length();
+ int i = 0;
+ while (i < length && symbol.charAt(i) != 0xFE0F) {
+ i++;
+ }
+ if (i == length) {
+ return symbol.toString();
+ }
+ final StringBuilder stripped = new StringBuilder(length - 1);
+ stripped.append(symbol, 0, i);
+ for (int k = i + 1; k < length; k++) {
+ final char c = symbol.charAt(k);
+ if (c != 0xFE0F) {
+ stripped.append(c);
+ }
+ }
+ return stripped.toString();
+ }
+
+ private static Map<String, EmojiAnnotation> annotations() {
+ Map<String, EmojiAnnotation> map = annotations;
+ if (map == null) {
+ synchronized (EmojiAnnotations.class) {
+ map = annotations;
+ if (map == null) {
+ map = load();
+ annotations = map;
+ }
+ }
+ }
+ return map;
+ }
+
+ private static Map<String, EmojiAnnotation> load() {
+ try (InputStream in = EmojiAnnotations.class.getResourceAsStream(RESOURCE)) {
+ if (in == null) {
+ throw new IllegalStateException("Missing emoji annotation data resource: " + RESOURCE);
+ }
+ return parse(in);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to read emoji annotation data resource " + RESOURCE, e);
+ }
+ }
+
+ // Package-private so the malformed-data handling can be exercised without the bundled resource.
+ // Parses rows of "codepoints ; attribute ; value ; source ; notes" with space-separated
+ // hexadecimal code points; '#' starts a comment line. The notes column is the fifth and final
+ // field, so it may contain ';'.
+ static Map<String, EmojiAnnotation> parse(InputStream in) throws IOException {
+ final Map<String, Map<String, EmojiAnnotation.Value>> rows = new HashMap<>();
+ try (BufferedReader reader =
+ new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8))) {
+ String line;
+ int lineNumber = 0;
+ while ((line = reader.readLine()) != null) {
+ lineNumber++;
+ final String content = line.strip();
+ if (content.isEmpty() || content.startsWith("#")) {
+ continue;
+ }
+ // Bounded split: only the first four separators are structural.
+ final String[] fields = content.split(";", 5);
+ if (fields.length != 5) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": expected 5 fields, got " + fields.length
+ + " in: " + content);
+ }
+ final String symbol = decode(fields[0], lineNumber, content);
+ final String attribute = fields[1].strip();
+ final String value = fields[2].strip();
+ final String source = fields[3].strip();
+ final String notes = fields[4].strip();
+ validate(attribute, value, source, lineNumber, content);
+ final Map<String, EmojiAnnotation.Value> record =
+ rows.computeIfAbsent(symbol, k -> new HashMap<>());
+ if (record.putIfAbsent(attribute, new EmojiAnnotation.Value(value, source, notes)) != null) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": duplicate attribute '" + attribute
+ + "' in: " + content);
+ }
+ }
+ }
+ final Map<String, EmojiAnnotation> records = new HashMap<>(rows.size());
+ for (final Map.Entry<String, Map<String, EmojiAnnotation.Value>> entry : rows.entrySet()) {
+ records.put(entry.getKey(), new EmojiAnnotation(entry.getKey(), entry.getValue()));
+ }
+ return Map.copyOf(records);
+ }
+
+ // Fails loud on an unknown attribute and on a value the attribute's type does not admit, so a
+ // corrupted or drifted data file cannot load quietly.
+ private static void validate(String attribute, String value, String source,
+ int lineNumber, String content) {
+ if (value.isEmpty()) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": empty value in: " + content);
+ }
+ if (source.isEmpty()) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": empty source in: " + content);
+ }
+ switch (attribute) {
+ case EmojiAnnotation.NAME:
+ break;
+ case EmojiAnnotation.SENTIMENT:
+ final int score;
+ try {
+ score = Integer.parseInt(value);
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": sentiment value '" + value
+ + "' is not an integer in: " + content, e);
+ }
+ if (score < -2 || score > 2) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": sentiment value " + score
+ + " is outside -2..2 in: " + content);
+ }
+ break;
+ case EmojiAnnotation.ENTITY_TYPE:
+ try {
+ EmojiEntityType.valueOf(value);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": unrecognized entityType value '" + value
+ + "' in: " + content, e);
+ }
+ break;
+ case EmojiAnnotation.CATEGORY:
+ try {
+ EmojiCategory.valueOf(value);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": unrecognized category value '" + value
+ + "' in: " + content, e);
+ }
+ break;
+ default:
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": unknown attribute '" + attribute + "' in: " + content);
+ }
+ }
+
+ private static String decode(String hexCodePoints, int lineNumber, String content) {
+ final String stripped = hexCodePoints.strip();
+ if (stripped.isEmpty()) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": empty code point sequence in: " + content);
+ }
+ try {
+ final StringBuilder decoded = new StringBuilder();
+ for (final String hex : stripped.split(" ")) {
+ decoded.appendCodePoint(Integer.parseInt(hex, 16));
+ }
+ return decoded.toString();
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE
+ + " at line " + lineNumber + ": " + content, e);
+ }
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java
new file mode 100644
index 000000000..cfedf4018
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+/**
+ * The document-category hint of an annotated emoji, one constant per group header of the upstream
+ * Unicode {@code emoji-test.txt} (Emoji Keyboard/Display Test Data for
+ * <a href="https://www.unicode.org/reports/tr51/">UTS #51</a>). The group level of that file
+ * is already a document-category-shaped taxonomy, so this enum mirrors it one to one rather than
+ * inventing a competing one; the constant a bundled record carries is provenance-tagged
+ * {@code UCD:emoji-test} in {@code emoji-annotations.txt}.
+ *
+ * <p>All upstream groups are represented, including those the bundled data does not populate yet,
+ * so growing the data never needs an enum change. See {@link EmojiEntityType} for the finer,
+ * subgroup-derived projection of the same upstream table.</p>
+ */
+public enum EmojiCategory {
+
+ /** Upstream group {@code Smileys & Emotion}: faces, hearts, and other emotion symbols. */
+ SMILEYS_AND_EMOTION,
+
+ /** Upstream group {@code People & Body}: people, body parts, hand gestures, roles. */
+ PEOPLE_AND_BODY,
+
+ /** Upstream group {@code Component}: skin tone and hair components, not free-standing symbols. */
+ COMPONENT,
+
+ /** Upstream group {@code Animals & Nature}: animals, plants, and nature symbols. */
+ ANIMALS_AND_NATURE,
+
+ /** Upstream group {@code Food & Drink}: food, drink, and dishware. */
+ FOOD_AND_DRINK,
+
+ /** Upstream group {@code Travel & Places}: places, buildings, transport, sky and weather. */
+ TRAVEL_AND_PLACES,
+
+ /** Upstream group {@code Activities}: events, sports, games, and arts. */
+ ACTIVITIES,
+
+ /** Upstream group {@code Objects}: clothing, tools, instruments, and other objects. */
+ OBJECTS,
+
+ /** Upstream group {@code Symbols}: signs, arrows, punctuation-like and abstract symbols. */
+ SYMBOLS,
+
+ /** Upstream group {@code Flags}: flag emoji; assigned by the derived layer, not by bundled rows. */
+ FLAGS
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java
new file mode 100644
index 000000000..5c43c7840
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+/**
+ * The coarse entity type of an annotated emoji, a small project-authored taxonomy assigned from
+ * the subgroup headers of the upstream Unicode {@code emoji-test.txt} (Emoji Keyboard/Display Test
+ * Data for <a href="https://www.unicode.org/reports/tr51/">UTS #51</a>): for example the
+ * {@code face-smiling} and {@code face-concerned} subgroups map to {@link #FACE} and
+ * {@code transport-ground} maps to {@link #VEHICLE}. The subgroup a bundled record was derived
+ * from is cited in the notes column of {@code emoji-annotations.txt}, provenance-tagged
+ * {@code UCD:emoji-test}.
+ *
+ * <p>The set is deliberately coarse and grows with the bundled data; a new constant is a
+ * compatible addition. {@link #FLAG} is assigned by the derived layer (regional-indicator and tag
+ * sequence decoding), never by a bundled row, so flag coverage does not depend on data
+ * rows.</p>
+ *
+ * @see EmojiCategory
+ */
+public enum EmojiEntityType {
+
+ /** A face pictograph, from the {@code face-*} subgroups. */
+ FACE,
+
+ /** A heart pictograph, from the {@code heart} subgroup. */
+ HEART,
+
+ /** An animal, from the {@code animal-*} subgroups. */
+ ANIMAL,
+
+ /** Food, from the {@code food-*} subgroups. */
+ FOOD,
+
+ /** A vehicle, from the {@code transport-*} subgroups. */
+ VEHICLE,
+
+ /** A landmark or building, from the {@code place-*} subgroups. */
+ LANDMARK,
+
+ /** A flag; assigned by the derived layer from the code point sequence itself. */
+ FLAG
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
new file mode 100644
index 000000000..d95a83f14
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
@@ -0,0 +1,223 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Bundled emoji annotation facts, authored by this project. This is the bundled-facts layer of
+# the emoji annotation record store (loaded by EmojiAnnotations): one row per attribute of one
+# pictograph, so adding an attribute later is new rows plus loader support instead of a file
+# format break, and every value carries its own provenance. Row format:
+#
+# codepoints ; attribute ; value ; source ; notes
+#
+# codepoints is the space-separated hexadecimal code point sequence of the pictograph, keyed
+# without U+FE0F VARIATION SELECTOR-16 (the emoji presentation selector); lookups strip it,
+# because presentation selection does not change a symbol's identity.
+#
+# Attributes and their provenance:
+# name The CLDR short name as printed in the upstream emoji-test.txt (Emoji
+# Keyboard/Display Test Data for UTS #51, version 17.0); source CLDR:annotation.
+# sentiment A project-authored coarse polarity score, an integer in -2..2; original work,
+# source UNSPECIFIED, the explicit marker for a project judgment. No third-party
+# sentiment data set is copied here; in particular the Emoji Sentiment Ranking
+# (CC BY-SA) is NOT used in any form.
+# entityType A coarse entity type (an EmojiEntityType constant), assigned by this project
+# from the subgroup header of the upstream emoji-test.txt; source UCD:emoji-test,
+# the subgroup is cited in the notes.
+# category A document-category hint (an EmojiCategory constant), the group header of the
+# upstream emoji-test.txt; source UCD:emoji-test, the group is cited in the notes.
+#
+# The loader fails loud on an unknown attribute or a malformed row: the data and the code move
+# together, so an unrecognized attribute is corruption, not extensibility.
+#
+# Coverage claim (audited by EmojiAnnotationsTest): every pictograph that emoji-emoticons.txt
+# folds (every EMOJI-direction source, presentation selector stripped) has a record here, and
+# every record carries all four attributes.
+#
+# Flag emoji deliberately have no rows: their region decodes algorithmically from the code point
+# sequence (the derived-facts layer), and gazetteer identifiers are never baked into bundled data
+# (they churn, the gazetteer is an optional download, and baked identifiers would couple this
+# file's version to the gazetteer's version; the joined-facts layer resolves them at run time).
+#
+# Code points, names, and group/subgroup headers checked against emoji-test.txt version 17.0.
+#
+# A line starting with '#' is a comment; data rows carry no inline comments. The notes column is
+# the fifth and final field, so it may contain ';'.
+#
+# Smileys & Emotion
+1F600 ; name ; grinning face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F600 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F600 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F600 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F603 ; name ; grinning face with big eyes ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F603 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F603 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F603 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F604 ; name ; grinning face with smiling eyes ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F604 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F604 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F604 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F601 ; name ; beaming face with smiling eyes ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F601 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F601 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F601 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F642 ; name ; slightly smiling face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F642 ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F642 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F642 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F609 ; name ; winking face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F609 ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F609 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F609 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F60A ; name ; smiling face with smiling eyes ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F60A ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F60A ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F60A ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+263A ; name ; smiling face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+263A ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+263A ; entityType ; FACE ; UCD:emoji-test ; subgroup face-affection
+263A ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F617 ; name ; kissing face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F617 ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F617 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-affection
+1F617 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F618 ; name ; face blowing a kiss ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F618 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F618 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-affection
+1F618 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F61B ; name ; face with tongue ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F61B ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F61B ; entityType ; FACE ; UCD:emoji-test ; subgroup face-tongue
+1F61B ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F61C ; name ; winking face with tongue ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F61C ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F61C ; entityType ; FACE ; UCD:emoji-test ; subgroup face-tongue
+1F61C ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F61D ; name ; squinting face with tongue ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F61D ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F61D ; entityType ; FACE ; UCD:emoji-test ; subgroup face-tongue
+1F61D ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F610 ; name ; neutral face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F610 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F610 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-neutral-skeptical
+1F610 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F611 ; name ; expressionless face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F611 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F611 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-neutral-skeptical
+1F611 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F615 ; name ; confused face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F615 ; sentiment ; -1 ; UNSPECIFIED ; project judgment
+1F615 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F615 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F641 ; name ; slightly frowning face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F641 ; sentiment ; -1 ; UNSPECIFIED ; project judgment
+1F641 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F641 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+2639 ; name ; frowning face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+2639 ; sentiment ; -1 ; UNSPECIFIED ; project judgment
+2639 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+2639 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F62E ; name ; face with open mouth ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F62E ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F62E ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F62E ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F632 ; name ; astonished face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F632 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F632 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F632 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F622 ; name ; crying face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F622 ; sentiment ; -2 ; UNSPECIFIED ; project judgment
+1F622 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F622 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F62D ; name ; loudly crying face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F62D ; sentiment ; -2 ; UNSPECIFIED ; project judgment
+1F62D ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F62D ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+2764 ; name ; red heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+2764 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+2764 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+2764 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F9E1 ; name ; orange heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F9E1 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F9E1 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F9E1 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F49B ; name ; yellow heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F49B ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F49B ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F49B ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F49A ; name ; green heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F49A ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F49A ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F49A ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F499 ; name ; blue heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F499 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F499 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F499 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F49C ; name ; purple heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F49C ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F49C ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F49C ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F90E ; name ; brown heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F90E ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F90E ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F90E ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F5A4 ; name ; black heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F5A4 ; sentiment ; 0 ; UNSPECIFIED ; project judgment, polarity ambiguous in running text
+1F5A4 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F5A4 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F90D ; name ; white heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F90D ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F90D ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F90D ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+1F494 ; name ; broken heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F494 ; sentiment ; -2 ; UNSPECIFIED ; project judgment
+1F494 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F494 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion
+#
+# Animals & Nature
+1F436 ; name ; dog face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F436 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F436 ; entityType ; ANIMAL ; UCD:emoji-test ; subgroup animal-mammal
+1F436 ; category ; ANIMALS_AND_NATURE ; UCD:emoji-test ; group Animals & Nature
+1F431 ; name ; cat face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F431 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F431 ; entityType ; ANIMAL ; UCD:emoji-test ; subgroup animal-mammal
+1F431 ; category ; ANIMALS_AND_NATURE ; UCD:emoji-test ; group Animals & Nature
+#
+# Food & Drink
+1F355 ; name ; pizza ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F355 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F355 ; entityType ; FOOD ; UCD:emoji-test ; subgroup food-prepared
+1F355 ; category ; FOOD_AND_DRINK ; UCD:emoji-test ; group Food & Drink
+1F34E ; name ; red apple ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F34E ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F34E ; entityType ; FOOD ; UCD:emoji-test ; subgroup food-fruit
+1F34E ; category ; FOOD_AND_DRINK ; UCD:emoji-test ; group Food & Drink
+#
+# Travel & Places
+1F697 ; name ; automobile ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F697 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F697 ; entityType ; VEHICLE ; UCD:emoji-test ; subgroup transport-ground
+1F697 ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places
+2708 ; name ; airplane ; CLDR:annotation ; emoji-test.txt v17.0 short name
+2708 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+2708 ; entityType ; VEHICLE ; UCD:emoji-test ; subgroup transport-air
+2708 ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places
+1F5FD ; name ; Statue of Liberty ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F5FD ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F5FD ; entityType ; LANDMARK ; UCD:emoji-test ; subgroup place-building
+1F5FD ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places
+1F5FC ; name ; Tokyo tower ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F5FC ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F5FC ; entityType ; LANDMARK ; UCD:emoji-test ; subgroup place-building
+1F5FC ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java
new file mode 100644
index 000000000..27c76d735
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class EmojiAnnotationsTest {
+
+ private static final String[] ATTRIBUTES = {EmojiAnnotation.NAME, EmojiAnnotation.SENTIMENT,
+ EmojiAnnotation.ENTITY_TYPE, EmojiAnnotation.CATEGORY};
+
+ private static String cp(int codePoint) {
+ return new String(Character.toChars(codePoint));
+ }
+
+ private static Map<String, EmojiAnnotation> bundled() throws IOException {
+ try (InputStream in = EmojiAnnotations.class.getResourceAsStream("emoji-annotations.txt")) {
+ return EmojiAnnotations.parse(in);
+ }
+ }
+
+ private static Map<String, EmojiAnnotation> parse(String data) throws IOException {
+ return EmojiAnnotations.parse(new ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8)));
+ }
+
+ // --- audits of the bundled data ---
+
+ @Test
+ void everyBundledRecordCarriesAllFourAttributes() throws IOException {
+ // The coverage claim in the data file header: a record is complete, never a partial row set,
+ // so a consumer can rely on any of the four dimensions once a symbol is annotated at all.
+ for (final EmojiAnnotation annotation : bundled().values()) {
+ for (final String attribute : ATTRIBUTES) {
+ assertTrue(annotation.attribute(attribute).isPresent(),
+ "Missing attribute '" + attribute + "' for symbol: " + annotation.symbol());
+ }
+ }
+ }
+
+ @Test
+ void everyBundledValueCarriesItsPinnedProvenance() throws IOException {
+ // The licensing audit: names come from the CLDR short names, entity type and category from
+ // the emoji-test.txt group/subgroup headers, and sentiment is ALWAYS a project judgment
+ // (UNSPECIFIED). A sentiment row with any other source would mean third-party sentiment data
+ // (for example the CC BY-SA Emoji Sentiment Ranking) leaked into the bundled file.
+ final Map<String, String> expectedSource = Map.of(
+ EmojiAnnotation.NAME, "CLDR:annotation",
+ EmojiAnnotation.SENTIMENT, "UNSPECIFIED",
+ EmojiAnnotation.ENTITY_TYPE, "UCD:emoji-test",
+ EmojiAnnotation.CATEGORY, "UCD:emoji-test");
+ for (final EmojiAnnotation annotation : bundled().values()) {
+ for (final Map.Entry<String, EmojiAnnotation.Value> entry :
+ annotation.attributes().entrySet()) {
+ final EmojiAnnotation.Value value = entry.getValue();
+ assertEquals(expectedSource.get(entry.getKey()), value.source(),
+ "Unexpected source for " + annotation.symbol() + " " + entry.getKey());
+ assertTrue(!value.notes().isEmpty(),
+ "Empty notes for " + annotation.symbol() + " " + entry.getKey());
+ }
+ }
+ }
+
+ @Test
+ void everyEmojiFoldSourceIsAnnotated() throws IOException {
+ // The cross-file coverage claim: every pictograph emoji-emoticons.txt can fold has a record
+ // here (presentation selector stripped), so the fold layer and the annotation layer never
+ // disagree about which symbols the project understands.
+ final Map<String, EmojiAnnotation> annotations = bundled();
+ final EmojiEmoticons.Tables tables;
+ try (InputStream in = EmojiEmoticons.class.getResourceAsStream("emoji-emoticons.txt")) {
+ tables = EmojiEmoticons.parse(in);
+ }
+ for (final List<EmojiEmoticons.Mapping> candidates : tables.emojiToEmoticon().values()) {
+ for (final EmojiEmoticons.Mapping mapping : candidates) {
+ final String key = mapping.source().replace(cp(0xFE0F), "");
+ assertTrue(annotations.containsKey(key),
+ "Emoji fold source has no annotation record: " + mapping);
+ }
+ }
+ }
+
+ @Test
+ void bundledRecordAndRowCountsAreAudited() throws IOException {
+ // Locks the bundled counts so a data edit trips this test for a conscious bump, the same
+ // discipline as the emoji-emoticons.txt and CaseFolding.txt audits.
+ final Map<String, EmojiAnnotation> annotations = bundled();
+ assertEquals(40, annotations.size());
+ int rowCount = 0;
+ for (final EmojiAnnotation annotation : annotations.values()) {
+ rowCount += annotation.attributes().size();
+ }
+ assertEquals(160, rowCount);
+ }
+
+ @Test
+ void bundledValuesAreWithinTheirTypedDomains() throws IOException {
+ // The typed accessors must never throw for bundled data: sentiment parses into -2..2 and the
+ // enum-backed attributes resolve to constants. Also asserts values are printable ASCII, the
+ // same discipline as the data file itself.
+ for (final EmojiAnnotation annotation : bundled().values()) {
+ final int sentiment = annotation.sentiment().orElseThrow();
+ assertTrue(sentiment >= -2 && sentiment <= 2,
+ "Sentiment outside -2..2 for " + annotation.symbol());
+ annotation.entityType().orElseThrow();
+ annotation.category().orElseThrow();
+ final String name = annotation.name().orElseThrow();
+ name.chars().forEach(c ->
+ assertTrue(c >= 0x20 && c < 0x7F, "Non-ASCII-printable name: " + name));
+ }
+ }
+
+ // --- lookup behavior ---
+
+ @Test
+ void lookupResolvesATypedRecord() {
+ final EmojiAnnotation annotation = EmojiAnnotations.lookup(cp(0x1F642)).orElseThrow();
+ assertEquals(cp(0x1F642), annotation.symbol());
+ assertEquals("slightly smiling face", annotation.name().orElseThrow());
+ assertEquals(1, annotation.sentiment().orElseThrow());
+ assertEquals(EmojiEntityType.FACE, annotation.entityType().orElseThrow());
+ assertEquals(EmojiCategory.SMILEYS_AND_EMOTION, annotation.category().orElseThrow());
+ }
+
+ @Test
+ void lookupStripsThePresentationSelector() {
+ // U+2764 U+FE0F (emoji presentation) and bare U+2764 are the same symbol; the bundled rows
+ // are keyed without the selector.
+ final EmojiAnnotation withSelector =
+ EmojiAnnotations.lookup(cp(0x2764) + cp(0xFE0F)).orElseThrow();
+ final EmojiAnnotation bare = EmojiAnnotations.lookup(cp(0x2764)).orElseThrow();
+ assertEquals(bare, withSelector);
+ assertEquals("red heart", withSelector.name().orElseThrow());
+ }
+
+ @Test
+ void lookupOfUnannotatedTextIsEmpty() {
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup("word"));
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup(":-)"));
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup(""));
+ // A bare presentation selector strips to an empty key.
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup(cp(0xFE0F)));
+ // Flags are handled by the derived layer, never by bundled rows.
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup(cp(0x1F1E9) + cp(0x1F1EA)));
+ }
+
+ @Test
+ void lookupFailsLoudOnNull() {
+ assertThrows(IllegalArgumentException.class, () -> EmojiAnnotations.lookup(null));
+ }
+
+ // --- fail-loud parsing ---
+
+ @Test
+ void parseFailsLoudOnWrongFieldCount() {
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; name ; slightly smiling face ; CLDR:annotation\n"));
+ }
+
+ @Test
+ void parseFailsLoudOnUnknownAttribute() {
+ // An unknown attribute is corruption, not extensibility: the data and the code move together.
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; shortName ; slightly smiling face ; CLDR:annotation ; n\n"));
+ }
+
+ @Test
+ void parseFailsLoudOnMalformedHex() {
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F64X ; name ; slightly smiling face ; CLDR:annotation ; n\n"));
+ }
+
+ @Test
+ void parseFailsLoudOnEmptyFields() {
+ assertThrows(IllegalArgumentException.class,
+ () -> parse(" ; name ; slightly smiling face ; CLDR:annotation ; n\n"));
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; name ; ; CLDR:annotation ; n\n"));
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; name ; slightly smiling face ; ; n\n"));
+ }
+
+ @Test
+ void parseFailsLoudOnSentimentOutsideTheCoarseScale() {
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; sentiment ; 3 ; UNSPECIFIED ; n\n"));
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; sentiment ; -3 ; UNSPECIFIED ; n\n"));
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; sentiment ; positive ; UNSPECIFIED ; n\n"));
+ }
+
+ @Test
+ void parseFailsLoudOnUnrecognizedEnumValues() {
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; entityType ; SMILEY ; UCD:emoji-test ; n\n"));
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; category ; EMOTION ; UCD:emoji-test ; n\n"));
+ }
+
+ @Test
+ void parseFailsLoudOnDuplicateAttributeRows() {
+ assertThrows(IllegalArgumentException.class,
+ () -> parse("1F642 ; sentiment ; 1 ; UNSPECIFIED ; first\n"
+ + "1F642 ; sentiment ; 2 ; UNSPECIFIED ; duplicate\n"));
+ }
+
+ @Test
+ void parseSkipsCommentAndBlankLinesAndKeepsSemicolonsInNotes() throws IOException {
+ final Map<String, EmojiAnnotation> parsed = parse("# comment\n\n \n"
+ + "1F642 ; name ; slightly smiling face ; CLDR:annotation ; note; with semicolon\n");
+ assertEquals(1, parsed.size());
+ assertEquals("note; with semicolon",
+ parsed.get(cp(0x1F642)).attribute(EmojiAnnotation.NAME).orElseThrow().notes());
+ }
+
+ // --- record validation (records are also user-constructable, for tests and custom joins) ---
+
+ @Test
+ void recordConstructionFailsLoudOnBadArguments() {
+ final Map<String, EmojiAnnotation.Value> attributes = Map.of(EmojiAnnotation.NAME,
+ new EmojiAnnotation.Value("slightly smiling face", "CLDR:annotation", ""));
+ assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation(null, attributes));
+ assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation("", attributes));
+ assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation(cp(0x1F642), null));
+ assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation.Value("", "s", ""));
+ assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation.Value("v", "", ""));
+ assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation.Value("v", "s", null));
+ }
+
+ @Test
+ void typedAccessorsFailLoudOnValuesOutsideTheirDomain() {
+ // Only reachable through hand-built records; the loader validates bundled data. A quietly
+ // swallowed conversion here would surface as a silently missing feature downstream.
+ final EmojiAnnotation annotation = new EmojiAnnotation(cp(0x1F642), Map.of(
+ EmojiAnnotation.SENTIMENT, new EmojiAnnotation.Value("happy", "UNSPECIFIED", ""),
+ EmojiAnnotation.ENTITY_TYPE, new EmojiAnnotation.Value("SMILEY", "UNSPECIFIED", ""),
+ EmojiAnnotation.CATEGORY, new EmojiAnnotation.Value("EMOTION", "UNSPECIFIED", "")));
+ assertThrows(IllegalStateException.class, annotation::sentiment);
+ assertThrows(IllegalStateException.class, annotation::entityType);
+ assertThrows(IllegalStateException.class, annotation::category);
+ assertThrows(IllegalArgumentException.class, () -> annotation.attribute(null));
+ }
+}
diff --git a/src/license/NOTICE.template b/src/license/NOTICE.template
index 593720c09..da75e2a6a 100644
--- a/src/license/NOTICE.template
+++ b/src/license/NOTICE.template
@@ -123,6 +123,15 @@ top of each bundled file. These files are distributed under the Unicode License
V3, the full text of which is reproduced in the LICENSE file accompanying this
distribution.
+The project-authored annotation table
+opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
+is licensed under the Apache License 2.0, but its name values (the CLDR short
+names) and its entity-type/category derivations are transcribed from the
+upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51,
+version 17.0), distributed under the Unicode License V3 reproduced in the
+LICENSE file. Its sentiment scores are original project judgments; no
+third-party sentiment data set is included.
+
Copyright (c) 1991-2025 Unicode, Inc. All rights reserved.
============================================================================
\ No newline at end of file