You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by rm...@apache.org on 2026/09/28 15:43:28 UTC
(lucene) branch branch_10x updated (bfbbd514931 -> c2e4612e051)
This is an automated email from the ASF dual-hosted git repository.
rmuir pushed a change to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
from bfbbd514931 Say how the staged query vectors are read (#16682)
new 405116dd0e1 Add ThaiCharFilter, ThaiNormalizationFilter, and ThaiNormalizer (#16716) (#16717)
new 8234d17b577 Curate Thai stopwords to prevent over-filtering of compound words (#5323, #14730) (#16718)
new ed0288d7990 Add ThaiRepeatFilter to handle Thai Maiyamok (ๆ) reduplication (#16719) (#16720)
new 2ba75780cd6 Support user dictionary in ThaiTokenizer and ThaiTokenizerFactory (#16721) (#16722)
new f05c0f3521c Allow configurable buffer size in SegmentingTokenizerBase and recognize whitespace as safe boundary in ThaiTokenizer (#10153) (#16727)
new c2e4612e051 docs: move Thai improvements to 10.6
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
lucene/CHANGES.txt | 13 ++
lucene/analysis/common/src/java/module-info.java | 5 +-
.../apache/lucene/analysis/th/ThaiAnalyzer.java | 19 +-
.../apache/lucene/analysis/th/ThaiCharFilter.java | 211 +++++++++++++++++++++
.../ThaiCharFilterFactory.java} | 28 +--
.../ThaiNormalizationFilter.java} | 23 ++-
.../ThaiNormalizationFilterFactory.java} | 25 ++-
.../apache/lucene/analysis/th/ThaiNormalizer.java | 175 +++++++++++++++++
.../lucene/analysis/th/ThaiRepeatFilter.java | 166 ++++++++++++++++
.../ThaiRepeatFilterFactory.java} | 27 +--
.../apache/lucene/analysis/th/ThaiTokenizer.java | 116 ++++++++++-
.../lucene/analysis/th/ThaiTokenizerFactory.java | 23 ++-
.../analysis/util/SegmentingTokenizerBase.java | 31 ++-
.../org.apache.lucene.analysis.CharFilterFactory | 1 +
.../org.apache.lucene.analysis.TokenFilterFactory | 2 +
.../org/apache/lucene/analysis/th/stopwords.txt | 42 +---
.../lucene/analysis/th/TestThaiAnalyzer.java | 42 ++++
.../lucene/analysis/th/TestThaiCharFilter.java | 133 +++++++++++++
.../TestThaiCharFilterFactory.java} | 16 +-
.../TestThaiNormalizationFilterFactory.java} | 19 +-
.../lucene/analysis/th/TestThaiNormalizer.java | 110 +++++++++++
.../lucene/analysis/th/TestThaiRepeatFilter.java | 105 ++++++++++
.../TestThaiRepeatFilterFactory.java} | 16 +-
.../lucene/analysis/th/TestThaiTokenizer.java | 113 +++++++++++
.../analysis/th/TestThaiTokenizerFactory.java | 10 +
.../lucene/analysis/th/customThaiDictionary.txt | 3 +
.../analysis/util/TestSegmentingTokenizerBase.java | 20 +-
27 files changed, 1372 insertions(+), 122 deletions(-)
create mode 100644 lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiCharFilter.java
copy lucene/analysis/common/src/java/org/apache/lucene/analysis/{fa/PersianCharFilterFactory.java => th/ThaiCharFilterFactory.java} (66%)
copy lucene/analysis/common/src/java/org/apache/lucene/analysis/{hi/HindiStemFilter.java => th/ThaiNormalizationFilter.java} (73%)
copy lucene/analysis/common/src/java/org/apache/lucene/analysis/{core/CaseFoldingFilterFactory.java => th/ThaiNormalizationFilterFactory.java} (68%)
create mode 100644 lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizer.java
create mode 100644 lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiRepeatFilter.java
copy lucene/analysis/common/src/java/org/apache/lucene/analysis/{te/TeluguStemFilterFactory.java => th/ThaiRepeatFilterFactory.java} (66%)
create mode 100644 lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiCharFilter.java
copy lucene/analysis/common/src/test/org/apache/lucene/analysis/{miscellaneous/TestScandinavianFoldingFilterFactory.java => th/TestThaiCharFilterFactory.java} (71%)
copy lucene/analysis/common/src/test/org/apache/lucene/analysis/{es/TestSpanishPluralStemFilterFactory.java => th/TestThaiNormalizationFilterFactory.java} (72%)
create mode 100644 lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiNormalizer.java
create mode 100644 lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiRepeatFilter.java
copy lucene/analysis/common/src/test/org/apache/lucene/analysis/{ckb/TestSoraniStemFilterFactory.java => th/TestThaiRepeatFilterFactory.java} (76%)
create mode 100644 lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizer.java
create mode 100644 lucene/analysis/common/src/test/org/apache/lucene/analysis/th/customThaiDictionary.txt
(lucene) 03/06: Add ThaiRepeatFilter to handle Thai Maiyamok (ๆ) reduplication (#16719) (#16720)
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmuir pushed a commit to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit ed0288d7990901a2af8cf4b154e8d9f79a469349
Author: Kamthorn Krairaksa <ka...@users.noreply.github.com>
AuthorDate: Mon Sep 28 04:54:53 2026 +0700
Add ThaiRepeatFilter to handle Thai Maiyamok (ๆ) reduplication (#16719) (#16720)
* Add ThaiRepeatFilter to handle Thai Maiyamok (ๆ) reduplication (#16719)
* Fix OffsetAttribute handling in ThaiRepeatFilter and add randomized offset tests
- Ensure both the primary term and repeated Maiyamok tokens receive precise, non-overlapping start/end offsets.
- For attached Maiyamok (e.g. 'เร็วๆ'), set primary token endOffset to boundary before 'ๆ' and set repeated token offsets to match the corresponding 'ๆ' character(s).
- For multiple standalone Maiyamok (e.g. 'ๆๆ'), partition token offsets evenly across each repetition.
- Update TestThaiRepeatFilter with exact offset assertions and add randomized testing with checkRandomData.
* Update CHANGES.txt for #16720
* Apply spotless formatting to ThaiRepeatFilter and tests
* Fix: register ThaiRepeatFilterFactory in module-info.java provides block for JPMS discovery
---
lucene/analysis/common/src/java/module-info.java | 1 +
.../apache/lucene/analysis/th/ThaiAnalyzer.java | 3 +-
.../lucene/analysis/th/ThaiRepeatFilter.java | 166 +++++++++++++++++++++
.../analysis/th/ThaiRepeatFilterFactory.java | 59 ++++++++
.../org.apache.lucene.analysis.TokenFilterFactory | 1 +
.../lucene/analysis/th/TestThaiAnalyzer.java | 7 +
.../lucene/analysis/th/TestThaiRepeatFilter.java | 105 +++++++++++++
.../analysis/th/TestThaiRepeatFilterFactory.java | 47 ++++++
8 files changed, 388 insertions(+), 1 deletion(-)
diff --git a/lucene/analysis/common/src/java/module-info.java b/lucene/analysis/common/src/java/module-info.java
index 4f262a8ab7b..8969d656206 100644
--- a/lucene/analysis/common/src/java/module-info.java
+++ b/lucene/analysis/common/src/java/module-info.java
@@ -264,6 +264,7 @@ module org.apache.lucene.analysis.common {
org.apache.lucene.analysis.te.TeluguNormalizationFilterFactory,
org.apache.lucene.analysis.te.TeluguStemFilterFactory,
org.apache.lucene.analysis.th.ThaiNormalizationFilterFactory,
+ org.apache.lucene.analysis.th.ThaiRepeatFilterFactory,
org.apache.lucene.analysis.tr.TurkishLowerCaseFilterFactory,
org.apache.lucene.analysis.util.ElisionFilterFactory;
provides org.apache.lucene.analysis.TokenizerFactory with
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiAnalyzer.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiAnalyzer.java
index 715237810af..8cbaa177a95 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiAnalyzer.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiAnalyzer.java
@@ -95,7 +95,7 @@ public final class ThaiAnalyzer extends StopwordAnalyzerBase {
*
* @return {@link org.apache.lucene.analysis.Analyzer.TokenStreamComponents} built from a {@link
* ThaiTokenizer} filtered with {@link LowerCaseFilter}, {@link DecimalDigitFilter}, {@link
- * ThaiNormalizationFilter} and {@link StopFilter}
+ * ThaiNormalizationFilter}, {@link ThaiRepeatFilter} and {@link StopFilter}
*/
@Override
protected TokenStreamComponents createComponents(String fieldName) {
@@ -103,6 +103,7 @@ public final class ThaiAnalyzer extends StopwordAnalyzerBase {
TokenStream result = new LowerCaseFilter(source);
result = new DecimalDigitFilter(result);
result = new ThaiNormalizationFilter(result);
+ result = new ThaiRepeatFilter(result);
result = new StopFilter(result, stopwords);
return new TokenStreamComponents(source, result);
}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiRepeatFilter.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiRepeatFilter.java
new file mode 100644
index 00000000000..1ef37ff70ae
--- /dev/null
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiRepeatFilter.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.TokenFilter;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
+import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
+import org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute;
+import org.apache.lucene.util.CharsRefBuilder;
+
+/**
+ * A {@link TokenFilter} that handles Thai Maiyamok (ๆ, U+0E46), which denotes word repetition
+ * (reduplication).
+ *
+ * <p>When a standalone Maiyamok token is encountered, it is replaced with the preceding term. When
+ * a token with trailing Maiyamok is encountered, trailing Maiyamok characters are stripped and
+ * duplicate token(s) are emitted.
+ */
+public final class ThaiRepeatFilter extends TokenFilter {
+
+ /** Thai Maiyamok character (ๆ) */
+ public static final char MAIYAMOK = '\u0E46';
+
+ private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
+ private final OffsetAttribute offsetAtt = addAttribute(OffsetAttribute.class);
+ private final PositionIncrementAttribute posIncAtt =
+ addAttribute(PositionIncrementAttribute.class);
+
+ private final CharsRefBuilder lastTerm = new CharsRefBuilder();
+ private int pendingRepeats = 0;
+ private int repeatIndex = 0;
+ private int[] repeatStarts = new int[4];
+ private int[] repeatEnds = new int[4];
+ private State savedState;
+
+ /**
+ * Creates a new ThaiRepeatFilter.
+ *
+ * @param input the input {@link TokenStream}
+ */
+ public ThaiRepeatFilter(TokenStream input) {
+ super(input);
+ }
+
+ @Override
+ public boolean incrementToken() throws IOException {
+ if (pendingRepeats > 0) {
+ restoreState(savedState);
+ termAtt.copyBuffer(lastTerm.chars(), 0, lastTerm.length());
+ posIncAtt.setPositionIncrement(1);
+ offsetAtt.setOffset(repeatStarts[repeatIndex], repeatEnds[repeatIndex]);
+ repeatIndex++;
+ pendingRepeats--;
+ return true;
+ }
+
+ while (input.incrementToken()) {
+ char[] buffer = termAtt.buffer();
+ int len = termAtt.length();
+ int origStart = offsetAtt.startOffset();
+ int origEnd = offsetAtt.endOffset();
+
+ // Case 1: Token consists entirely of Maiyamok characters (e.g. "ๆ" or "ๆๆ")
+ if (isAllMaiyamok(buffer, len)) {
+ if (lastTerm.length() == 0) {
+ // No preceding token (dangling Maiyamok at stream start); discard
+ continue;
+ }
+ int count = len;
+ termAtt.copyBuffer(lastTerm.chars(), 0, lastTerm.length());
+ if (count > 1) {
+ int firstEnd = Math.min(origEnd, origStart + 1);
+ offsetAtt.setOffset(origStart, firstEnd);
+ ensureRepeatCapacity(count - 1);
+ for (int i = 0; i < count - 1; i++) {
+ int s = Math.min(origEnd, origStart + 1 + i);
+ int e = (i == count - 2) ? origEnd : Math.min(origEnd, s + 1);
+ repeatStarts[i] = s;
+ repeatEnds[i] = e;
+ }
+ savedState = captureState();
+ pendingRepeats = count - 1;
+ repeatIndex = 0;
+ }
+ return true;
+ }
+
+ // Case 2: Token has trailing Maiyamok attached (e.g. "เร็วๆ")
+ int maiyamokCount = 0;
+ while (len - 1 - maiyamokCount >= 0 && buffer[len - 1 - maiyamokCount] == MAIYAMOK) {
+ maiyamokCount++;
+ }
+ if (maiyamokCount > 0) {
+ int baseLen = len - maiyamokCount;
+ termAtt.setLength(baseLen);
+ lastTerm.copyChars(termAtt.buffer(), 0, termAtt.length());
+
+ int baseEnd = Math.max(origStart, origEnd - maiyamokCount);
+ offsetAtt.setOffset(origStart, baseEnd);
+
+ ensureRepeatCapacity(maiyamokCount);
+ for (int i = 0; i < maiyamokCount; i++) {
+ int s = Math.min(origEnd, baseEnd + i);
+ int e = (i == maiyamokCount - 1) ? origEnd : Math.min(origEnd, s + 1);
+ repeatStarts[i] = s;
+ repeatEnds[i] = e;
+ }
+
+ savedState = captureState();
+ pendingRepeats = maiyamokCount;
+ repeatIndex = 0;
+ return true;
+ }
+
+ // Normal token without Maiyamok
+ lastTerm.copyChars(buffer, 0, len);
+ return true;
+ }
+
+ return false;
+ }
+
+ private void ensureRepeatCapacity(int count) {
+ if (count > repeatStarts.length) {
+ repeatStarts = new int[count];
+ repeatEnds = new int[count];
+ }
+ }
+
+ private static boolean isAllMaiyamok(char[] buffer, int len) {
+ if (len == 0) {
+ return false;
+ }
+ for (int i = 0; i < len; i++) {
+ if (buffer[i] != MAIYAMOK) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public void reset() throws IOException {
+ super.reset();
+ lastTerm.clear();
+ pendingRepeats = 0;
+ repeatIndex = 0;
+ savedState = null;
+ }
+}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiRepeatFilterFactory.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiRepeatFilterFactory.java
new file mode 100644
index 00000000000..aab00e0deea
--- /dev/null
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiRepeatFilterFactory.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.util.Map;
+import org.apache.lucene.analysis.TokenFilterFactory;
+import org.apache.lucene.analysis.TokenStream;
+
+/**
+ * Factory for {@link ThaiRepeatFilter}.
+ *
+ * <pre><code class="language-xml">
+ * <fieldType name="text_th" class="solr.TextField" positionIncrementGap="100">
+ * <analyzer>
+ * <tokenizer class="solr.ThaiTokenizerFactory"/>
+ * <filter class="solr.ThaiRepeatFilterFactory"/>
+ * </analyzer>
+ * </fieldType></code></pre>
+ *
+ * @since 11.0.0
+ * @lucene.spi {@value #NAME}
+ */
+public class ThaiRepeatFilterFactory extends TokenFilterFactory {
+
+ /** SPI name */
+ public static final String NAME = "thaiRepeat";
+
+ /** Creates a new ThaiRepeatFilterFactory */
+ public ThaiRepeatFilterFactory(Map<String, String> args) {
+ super(args);
+ if (!args.isEmpty()) {
+ throw new IllegalArgumentException("Unknown parameters: " + args);
+ }
+ }
+
+ /** Default ctor for compatibility with SPI */
+ public ThaiRepeatFilterFactory() {
+ throw defaultCtorException();
+ }
+
+ @Override
+ public TokenStream create(TokenStream input) {
+ return new ThaiRepeatFilter(input);
+ }
+}
diff --git a/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.TokenFilterFactory b/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.TokenFilterFactory
index b5b276803bf..8952a837f93 100644
--- a/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.TokenFilterFactory
+++ b/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.TokenFilterFactory
@@ -125,5 +125,6 @@ org.apache.lucene.analysis.core.FlattenGraphFilterFactory
org.apache.lucene.analysis.te.TeluguNormalizationFilterFactory
org.apache.lucene.analysis.te.TeluguStemFilterFactory
org.apache.lucene.analysis.th.ThaiNormalizationFilterFactory
+org.apache.lucene.analysis.th.ThaiRepeatFilterFactory
org.apache.lucene.analysis.tr.TurkishLowerCaseFilterFactory
org.apache.lucene.analysis.util.ElisionFilterFactory
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java
index 58565d23293..c4aeeda47bc 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java
@@ -147,6 +147,13 @@ public class TestThaiAnalyzer extends BaseTokenStreamTestCase {
analyzer.close();
}
+ public void testMaiyamok() throws Exception {
+ Analyzer analyzer = new ThaiAnalyzer(CharArraySet.EMPTY_SET);
+ assertAnalyzesTo(analyzer, "วิ่งเร็วๆ", new String[] {"วิ่ง", "เร็ว", "เร็ว"});
+ assertAnalyzesTo(analyzer, "พูดมากๆ นะ", new String[] {"พูด", "มาก", "มาก", "นะ"});
+ analyzer.close();
+ }
+
/** LUCENE-4253, #14730: Test that common content words in compounds are not over-filtered */
public void testCompoundAndContentWordsNotOverFiltered() throws Exception {
Analyzer analyzer = new ThaiAnalyzer();
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiRepeatFilter.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiRepeatFilter.java
new file mode 100644
index 00000000000..758f3d98280
--- /dev/null
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiRepeatFilter.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.IOException;
+import java.io.StringReader;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.Tokenizer;
+import org.apache.lucene.tests.analysis.BaseTokenStreamTestCase;
+import org.apache.lucene.tests.analysis.MockTokenizer;
+
+/** Test case for {@link ThaiRepeatFilter}. */
+public class TestThaiRepeatFilter extends BaseTokenStreamTestCase {
+
+ public void testStandaloneMaiyamok() throws IOException {
+ // Standalone token "ๆ" after "เร็ว"
+ TokenStream ts = whitespaceMockTokenizer("เร็ว ๆ");
+ ts = new ThaiRepeatFilter(ts);
+ assertTokenStreamContents(
+ ts, new String[] {"เร็ว", "เร็ว"}, new int[] {0, 5}, new int[] {4, 6});
+ }
+
+ public void testAttachedMaiyamok() throws IOException {
+ // Attached "ๆ" at the end of word "เร็วๆ"
+ TokenStream ts = whitespaceMockTokenizer("เร็วๆ");
+ ts = new ThaiRepeatFilter(ts);
+ assertTokenStreamContents(
+ ts, new String[] {"เร็ว", "เร็ว"}, new int[] {0, 4}, new int[] {4, 5});
+ }
+
+ public void testMultipleMaiyamok() throws IOException {
+ // Double Maiyamok "มากๆๆ"
+ TokenStream ts = whitespaceMockTokenizer("มากๆๆ");
+ ts = new ThaiRepeatFilter(ts);
+ assertTokenStreamContents(
+ ts, new String[] {"มาก", "มาก", "มาก"}, new int[] {0, 3, 4}, new int[] {3, 4, 5});
+
+ // Separate tokens "มาก ๆ ๆ"
+ ts = whitespaceMockTokenizer("มาก ๆ ๆ");
+ ts = new ThaiRepeatFilter(ts);
+ assertTokenStreamContents(
+ ts, new String[] {"มาก", "มาก", "มาก"}, new int[] {0, 4, 6}, new int[] {3, 5, 7});
+ }
+
+ public void testInSentence() throws IOException {
+ TokenStream ts = whitespaceMockTokenizer("เด็ก ๆ กำลัง วิ่ง เล่น");
+ ts = new ThaiRepeatFilter(ts);
+ assertTokenStreamContents(
+ ts,
+ new String[] {"เด็ก", "เด็ก", "กำลัง", "วิ่ง", "เล่น"},
+ new int[] {0, 5, 7, 13, 18},
+ new int[] {4, 6, 12, 17, 22});
+ }
+
+ public void testLeadingDanglingMaiyamok() throws IOException {
+ // Maiyamok at stream start with no preceding token
+ TokenStream ts = whitespaceMockTokenizer("ๆ ครับ");
+ ts = new ThaiRepeatFilter(ts);
+ assertTokenStreamContents(ts, new String[] {"ครับ"});
+ }
+
+ public void testEmpty() throws IOException {
+ TokenStream ts = whitespaceMockTokenizer("");
+ ts = new ThaiRepeatFilter(ts);
+ assertTokenStreamContents(ts, new String[] {});
+ }
+
+ public void testWithThaiTokenizer() throws IOException {
+ assumeTrue(
+ "JRE does not support Thai dictionary-based BreakIterator", ThaiTokenizer.DBBI_AVAILABLE);
+ Tokenizer tokenizer = new ThaiTokenizer();
+ tokenizer.setReader(new StringReader("วิ่งเร็วๆ"));
+ TokenStream ts = new ThaiRepeatFilter(tokenizer);
+ assertTokenStreamContents(
+ ts, new String[] {"วิ่ง", "เร็ว", "เร็ว"}, new int[] {0, 4, 8}, new int[] {4, 8, 9});
+ }
+
+ public void testRandomStrings() throws IOException {
+ Analyzer a =
+ new Analyzer() {
+ @Override
+ protected TokenStreamComponents createComponents(String fieldName) {
+ Tokenizer tokenizer = new MockTokenizer(MockTokenizer.WHITESPACE, false);
+ return new TokenStreamComponents(tokenizer, new ThaiRepeatFilter(tokenizer));
+ }
+ };
+ checkRandomData(random(), a, 200 * RANDOM_MULTIPLIER);
+ a.close();
+ }
+}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiRepeatFilterFactory.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiRepeatFilterFactory.java
new file mode 100644
index 00000000000..dc925398bfc
--- /dev/null
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiRepeatFilterFactory.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.Reader;
+import java.io.StringReader;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.Tokenizer;
+import org.apache.lucene.tests.analysis.BaseTokenStreamFactoryTestCase;
+import org.apache.lucene.tests.analysis.MockTokenizer;
+
+/** Simple tests to ensure the Thai repeat factory is working. */
+public class TestThaiRepeatFilterFactory extends BaseTokenStreamFactoryTestCase {
+
+ public void testRepeat() throws Exception {
+ Reader reader = new StringReader("เร็ว ๆ");
+ TokenStream stream = new MockTokenizer(MockTokenizer.WHITESPACE, false);
+ ((Tokenizer) stream).setReader(reader);
+ stream = tokenFilterFactory("ThaiRepeat").create(stream);
+ assertTokenStreamContents(stream, new String[] {"เร็ว", "เร็ว"});
+ }
+
+ /** Test that bogus arguments result in exception */
+ public void testBogusArguments() throws Exception {
+ IllegalArgumentException expected =
+ expectThrows(
+ IllegalArgumentException.class,
+ () -> {
+ tokenFilterFactory("ThaiRepeat", "bogusArg", "bogusValue");
+ });
+ assertTrue(expected.getMessage().contains("Unknown parameters"));
+ }
+}
(lucene) 04/06: Support user dictionary in ThaiTokenizer and ThaiTokenizerFactory (#16721) (#16722)
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmuir pushed a commit to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 2ba75780cd6d46fb29bd3c5e9e8ed8580b25812c
Author: Kamthorn Krairaksa <ka...@users.noreply.github.com>
AuthorDate: Mon Sep 28 07:58:58 2026 +0700
Support user dictionary in ThaiTokenizer and ThaiTokenizerFactory (#16721) (#16722)
* Support user dictionary in ThaiTokenizer and ThaiTokenizerFactory (#16721)
* Update CHANGES.txt for #16722
* Apply spotless formatting to ThaiTokenizer and TestThaiTokenizerFactory
* Simplify user dictionary min/max word length calculation per review suggestion
---
.../apache/lucene/analysis/th/ThaiTokenizer.java | 76 +++++++++++++++++++++-
.../lucene/analysis/th/ThaiTokenizerFactory.java | 21 +++++-
.../lucene/analysis/th/TestThaiTokenizer.java | 66 +++++++++++++++++++
.../analysis/th/TestThaiTokenizerFactory.java | 10 +++
.../lucene/analysis/th/customThaiDictionary.txt | 3 +
5 files changed, 171 insertions(+), 5 deletions(-)
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java
index 50674f1f657..7e76cfde780 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java
@@ -18,6 +18,7 @@ package org.apache.lucene.analysis.th;
import java.text.BreakIterator;
import java.util.Locale;
+import org.apache.lucene.analysis.CharArraySet;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
import org.apache.lucene.analysis.util.CharArrayIterator;
@@ -25,7 +26,11 @@ import org.apache.lucene.analysis.util.SegmentingTokenizerBase;
import org.apache.lucene.util.AttributeFactory;
/**
- * Tokenizer that use {@link BreakIterator} to tokenize Thai text.
+ * Tokenizer that uses {@link BreakIterator} to tokenize Thai text.
+ *
+ * <p>Supports an optional user dictionary ({@link CharArraySet}) for custom or domain-specific
+ * words. When a user dictionary word matches, it takes precedence over default segmentation
+ * boundaries.
*
* <p>WARNING: this tokenizer may not be supported by all JREs. It is known to work with Sun/Oracle
* and Harmony JREs. If your application needs to be fully portable, consider using ICUTokenizer
@@ -53,6 +58,10 @@ public class ThaiTokenizer extends SegmentingTokenizerBase {
private final BreakIterator wordBreaker;
private final CharArrayIterator wrapper = CharArrayIterator.newWordInstance();
+ private final CharArraySet userDictionary;
+ private final int minDictWordLen;
+ private final int maxDictWordLen;
+
int sentenceStart;
int sentenceEnd;
@@ -61,17 +70,51 @@ public class ThaiTokenizer extends SegmentingTokenizerBase {
/** Creates a new ThaiTokenizer */
public ThaiTokenizer() {
- this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY);
+ this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, null);
+ }
+
+ /**
+ * Creates a new ThaiTokenizer with a user dictionary.
+ *
+ * @param userDictionary custom dictionary of words, or null for default dictionary only
+ */
+ public ThaiTokenizer(CharArraySet userDictionary) {
+ this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, userDictionary);
}
/** Creates a new ThaiTokenizer, supplying the AttributeFactory */
public ThaiTokenizer(AttributeFactory factory) {
+ this(factory, null);
+ }
+
+ /**
+ * Creates a new ThaiTokenizer, supplying the AttributeFactory and user dictionary.
+ *
+ * @param factory AttributeFactory to use
+ * @param userDictionary custom dictionary of words, or null for default dictionary only
+ */
+ public ThaiTokenizer(AttributeFactory factory, CharArraySet userDictionary) {
super(factory, (BreakIterator) sentenceProto.clone());
if (!DBBI_AVAILABLE) {
throw new UnsupportedOperationException(
"This JRE does not have support for Thai segmentation");
}
wordBreaker = (BreakIterator) proto.clone();
+ this.userDictionary = userDictionary;
+ if (userDictionary != null && !userDictionary.isEmpty()) {
+ int minLen = Integer.MAX_VALUE;
+ int maxLen = 0;
+ for (Object obj : userDictionary) {
+ int length = ((char[]) obj).length;
+ minLen = Math.min(minLen, length);
+ maxLen = Math.max(maxLen, length);
+ }
+ this.minDictWordLen = minLen;
+ this.maxDictWordLen = maxLen;
+ } else {
+ this.minDictWordLen = 0;
+ this.maxDictWordLen = 0;
+ }
}
@Override
@@ -102,10 +145,39 @@ public class ThaiTokenizer extends SegmentingTokenizerBase {
return false; // BreakIterator exhausted
}
+ boolean reanchor = false;
+ if (userDictionary != null && !userDictionary.isEmpty()) {
+ int remaining = sentenceEnd - (sentenceStart + start);
+ int maxLen = Math.min(maxDictWordLen, remaining);
+ for (int l = maxLen; l >= minDictWordLen; l--) {
+ if (userDictionary.contains(buffer, sentenceStart + start, l)) {
+ int customEnd = start + l;
+ if (customEnd != end) {
+ end = customEnd;
+ if (end >= sentenceEnd - sentenceStart) {
+ wordBreaker.last();
+ } else if (wordBreaker.isBoundary(end)) {
+ wordBreaker.following(end - 1);
+ } else {
+ reanchor = true;
+ }
+ }
+ break;
+ }
+ }
+ }
+
clearAttributes();
termAtt.copyBuffer(buffer, sentenceStart + start, end - start);
offsetAtt.setOffset(
correctOffset(offset + sentenceStart + start), correctOffset(offset + sentenceStart + end));
+
+ if (reanchor) {
+ sentenceStart += end;
+ wrapper.setText(buffer, sentenceStart, sentenceEnd - sentenceStart);
+ wordBreaker.setText(wrapper);
+ }
+
return true;
}
}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizerFactory.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizerFactory.java
index bf0b7f8f937..b973db20bb4 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizerFactory.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizerFactory.java
@@ -16,10 +16,14 @@
*/
package org.apache.lucene.analysis.th;
+import java.io.IOException;
import java.util.Map;
+import org.apache.lucene.analysis.CharArraySet;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.TokenizerFactory;
import org.apache.lucene.util.AttributeFactory;
+import org.apache.lucene.util.ResourceLoader;
+import org.apache.lucene.util.ResourceLoaderAware;
/**
* Factory for {@link ThaiTokenizer}.
@@ -27,21 +31,25 @@ import org.apache.lucene.util.AttributeFactory;
* <pre class="prettyprint">
* <fieldType name="text_thai" class="solr.TextField" positionIncrementGap="100">
* <analyzer>
- * <tokenizer class="solr.ThaiTokenizerFactory"/>
+ * <tokenizer class="solr.ThaiTokenizerFactory" dictionary="custom_words.txt"/>
* </analyzer>
* </fieldType></pre>
*
* @since 4.10.0
* @lucene.spi {@value #NAME}
*/
-public class ThaiTokenizerFactory extends TokenizerFactory {
+public class ThaiTokenizerFactory extends TokenizerFactory implements ResourceLoaderAware {
/** SPI name */
public static final String NAME = "thai";
+ private final String dictFile;
+ private CharArraySet dictionary;
+
/** Creates a new ThaiTokenizerFactory */
public ThaiTokenizerFactory(Map<String, String> args) {
super(args);
+ dictFile = get(args, "dictionary");
if (!args.isEmpty()) {
throw new IllegalArgumentException("Unknown parameters: " + args);
}
@@ -52,8 +60,15 @@ public class ThaiTokenizerFactory extends TokenizerFactory {
throw defaultCtorException();
}
+ @Override
+ public void inform(ResourceLoader loader) throws IOException {
+ if (dictFile != null) {
+ dictionary = getWordSet(loader, dictFile, false);
+ }
+ }
+
@Override
public Tokenizer create(AttributeFactory factory) {
- return new ThaiTokenizer(factory);
+ return new ThaiTokenizer(factory, dictionary);
}
}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizer.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizer.java
new file mode 100644
index 00000000000..264aee2598d
--- /dev/null
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizer.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.IOException;
+import java.io.StringReader;
+import java.util.List;
+import org.apache.lucene.analysis.CharArraySet;
+import org.apache.lucene.analysis.Tokenizer;
+import org.apache.lucene.tests.analysis.BaseTokenStreamTestCase;
+
+/** Test case for {@link ThaiTokenizer}. */
+public class TestThaiTokenizer extends BaseTokenStreamTestCase {
+
+ @Override
+ public void setUp() throws Exception {
+ super.setUp();
+ assumeTrue(
+ "JRE does not support Thai dictionary-based BreakIterator", ThaiTokenizer.DBBI_AVAILABLE);
+ }
+
+ public void testDefaultSegmentation() throws IOException {
+ Tokenizer tokenizer = new ThaiTokenizer();
+ tokenizer.setReader(new StringReader("ภาษาไทย"));
+ assertTokenStreamContents(tokenizer, new String[] {"ภาษา", "ไทย"});
+ }
+
+ public void testUserDictionary() throws IOException {
+ CharArraySet userDict = new CharArraySet(List.of("พารากอน", "คนขับรถ"), false);
+ Tokenizer tokenizer = new ThaiTokenizer(userDict);
+ tokenizer.setReader(new StringReader("ไปพารากอนกัน"));
+ assertTokenStreamContents(tokenizer, new String[] {"ไป", "พารากอน", "กัน"});
+ }
+
+ public void testMultipleUserDictionaryTermsAndOffsets() throws IOException {
+ CharArraySet userDict = new CharArraySet(List.of("พารากอน", "คนขับรถ"), false);
+ Tokenizer tokenizer = new ThaiTokenizer(userDict);
+ tokenizer.setReader(new StringReader("เขาเป็นคนขับรถไปพารากอน"));
+ assertTokenStreamContents(
+ tokenizer,
+ new String[] {"เขา", "เป็น", "คนขับรถ", "ไป", "พารากอน"},
+ new int[] {0, 3, 7, 14, 16},
+ new int[] {3, 7, 14, 16, 23});
+ }
+
+ public void testEmptyUserDictionary() throws IOException {
+ CharArraySet userDict = new CharArraySet(0, false);
+ Tokenizer tokenizer = new ThaiTokenizer(userDict);
+ tokenizer.setReader(new StringReader("ภาษาไทย"));
+ assertTokenStreamContents(tokenizer, new String[] {"ภาษา", "ไทย"});
+ }
+}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizerFactory.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizerFactory.java
index e77a749564f..21c13c7d36b 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizerFactory.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizerFactory.java
@@ -32,6 +32,16 @@ public class TestThaiTokenizerFactory extends BaseTokenStreamFactoryTestCase {
tokenizer, new String[] {"การ", "ที่", "ได้", "ต้อง", "แสดง", "ว่า", "งาน", "ดี"});
}
+ public void testCustomDictionary() throws Exception {
+ assumeTrue(
+ "JRE does not support Thai dictionary-based BreakIterator", ThaiTokenizer.DBBI_AVAILABLE);
+ Tokenizer tokenizer =
+ tokenizerFactory("Thai", "dictionary", "customThaiDictionary.txt")
+ .create(newAttributeFactory());
+ tokenizer.setReader(new StringReader("ไปพารากอน"));
+ assertTokenStreamContents(tokenizer, new String[] {"ไป", "พารากอน"});
+ }
+
/** Test that bogus arguments result in exception */
public void testBogusArguments() throws Exception {
assumeTrue(
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/customThaiDictionary.txt b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/customThaiDictionary.txt
new file mode 100644
index 00000000000..d042b89e41e
--- /dev/null
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/customThaiDictionary.txt
@@ -0,0 +1,3 @@
+# Custom Thai Dictionary
+พารากอน
+คนขับรถ
(lucene) 02/06: Curate Thai stopwords to prevent over-filtering of compound words (#5323, #14730) (#16718)
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmuir pushed a commit to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 8234d17b577fd4d365613121a0c6e189580542b9
Author: Kamthorn Krairaksa <ka...@users.noreply.github.com>
AuthorDate: Mon Sep 28 04:34:36 2026 +0700
Curate Thai stopwords to prevent over-filtering of compound words (#5323, #14730) (#16718)
* Curate Thai stopwords to prevent over-filtering of compound words (LUCENE-4253, #14730)
* Update CHANGES.txt for #16718
* Apply spotless formatting to TestThaiAnalyzer
---
.../org/apache/lucene/analysis/th/stopwords.txt | 42 ++++------------------
.../lucene/analysis/th/TestThaiAnalyzer.java | 16 +++++++++
2 files changed, 23 insertions(+), 35 deletions(-)
diff --git a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/th/stopwords.txt b/lucene/analysis/common/src/resources/org/apache/lucene/analysis/th/stopwords.txt
index 07f0fabe692..50d44237ecd 100644
--- a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/th/stopwords.txt
+++ b/lucene/analysis/common/src/resources/org/apache/lucene/analysis/th/stopwords.txt
@@ -1,44 +1,35 @@
-# Thai stopwords from:
-# "Opinion Detection in Thai Political News Columns
-# Based on Subjectivity Analysis"
-# Khampol Sukhum, Supot Nitsuwat, and Choochart Haruechaiyasak
+# Thai stopwords
+# Originally derived from Sukhum et al. (2010) "Opinion Detection in Thai Political News Columns"
+# Curated for general information retrieval to prevent over-filtering of compound words,
+# common verbs, and nouns (LUCENE-4253, #14730).
ไว้
ไม่
-ไป
-ได้
ให้
ใน
โดย
+ได้
แห่ง
แล้ว
และ
-แรก
-แบบ
แต่
เอง
-เห็น
เลย
-เริ่ม
เรา
เมื่อ
เพื่อ
เพราะ
เป็นการ
เป็น
-เปิดเผย
-เปิด
เนื่องจาก
เดียวกัน
เดียว
เช่น
เฉพาะ
เคย
-เข้า
เขา
อีก
อาจ
อะไร
-ออก
อย่าง
อยู่
อยาก
@@ -49,38 +40,26 @@
หรือ
หนึ่ง
ส่วน
-ส่ง
สุด
สําหรับ
+สำหรับ
ว่า
-วัน
-ลง
-ร่วม
-ราย
-รับ
ระหว่าง
รวม
ยัง
มี
มาก
-มา
พร้อม
-พบ
-ผ่าน
-ผล
บาง
น่า
นี้
-นํา
นั้น
-นัก
นอกจาก
ทุก
ที่สุด
ที่
ทําให้
-ทํา
-ทาง
+ทำให้
ทั้งนี้
ทั้ง
ถ้า
@@ -92,23 +71,17 @@
ต่อ
ตาม
ตั้งแต่
-ตั้ง
-ด้าน
ด้วย
ดัง
ซึ่ง
-ช่วง
จึง
จาก
-จัด
จะ
คือ
ความ
ครั้ง
คง
-ขึ้น
ของ
-ขอ
ขณะ
ก่อน
ก็
@@ -116,4 +89,3 @@
กับ
กัน
กว่า
-กล่าว
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java
index 9bca83c8f7b..58565d23293 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java
@@ -147,6 +147,22 @@ public class TestThaiAnalyzer extends BaseTokenStreamTestCase {
analyzer.close();
}
+ /** LUCENE-4253, #14730: Test that common content words in compounds are not over-filtered */
+ public void testCompoundAndContentWordsNotOverFiltered() throws Exception {
+ Analyzer analyzer = new ThaiAnalyzer();
+ // Issue #14730: "ที่ผ่านมา" should retain "ผ่าน" and "มา"
+ assertAnalyzesTo(analyzer, "ที่ผ่านมา", new String[] {"ผ่าน", "มา"});
+ // "เปิดตัวสินค้า" should retain "เปิด"
+ assertAnalyzesTo(analyzer, "เปิดตัวสินค้า", new String[] {"เปิด", "ตัว", "สินค้า"});
+ // "ส่งออก" should retain both "ส่ง" and "ออก"
+ assertAnalyzesTo(analyzer, "ส่งออก", new String[] {"ส่ง", "ออก"});
+ // "วันหยุด" should retain "วัน"
+ assertAnalyzesTo(analyzer, "วันหยุด", new String[] {"วัน", "หยุด"});
+ // "ผลการเรียน" should retain "ผล" and "เรียน"
+ assertAnalyzesTo(analyzer, "ผลการเรียน", new String[] {"ผล", "เรียน"});
+ analyzer.close();
+ }
+
public void testNormalization() throws Exception {
Analyzer analyzer = new ThaiAnalyzer(CharArraySet.EMPTY_SET);
// Double Sara E -> Sara Ae
(lucene) 05/06: Allow configurable buffer size in SegmentingTokenizerBase and recognize whitespace as safe boundary in ThaiTokenizer (#10153) (#16727)
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmuir pushed a commit to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit f05c0f3521cf9f01960d466409c916db98113280
Author: Kamthorn Krairaksa <ka...@users.noreply.github.com>
AuthorDate: Mon Sep 28 08:38:34 2026 +0700
Allow configurable buffer size in SegmentingTokenizerBase and recognize whitespace as safe boundary in ThaiTokenizer (#10153) (#16727)
---
.../apache/lucene/analysis/th/ThaiTokenizer.java | 40 +++++++++++++++++-
.../lucene/analysis/th/ThaiTokenizerFactory.java | 4 +-
.../analysis/util/SegmentingTokenizerBase.java | 31 ++++++++++++--
.../lucene/analysis/th/TestThaiTokenizer.java | 47 ++++++++++++++++++++++
.../analysis/util/TestSegmentingTokenizerBase.java | 20 ++++++++-
5 files changed, 136 insertions(+), 6 deletions(-)
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java
index 7e76cfde780..1436981e38b 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java
@@ -55,6 +55,9 @@ public class ThaiTokenizer extends SegmentingTokenizerBase {
/** used for breaking the text into sentences */
private static final BreakIterator sentenceProto = BreakIterator.getSentenceInstance(Locale.ROOT);
+ /** Default buffer size in characters for Thai sentence segmentation */
+ public static final int DEFAULT_BUFFER_SIZE = BUFFERMAX;
+
private final BreakIterator wordBreaker;
private final CharArrayIterator wrapper = CharArrayIterator.newWordInstance();
@@ -82,6 +85,25 @@ public class ThaiTokenizer extends SegmentingTokenizerBase {
this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, userDictionary);
}
+ /**
+ * Creates a new ThaiTokenizer with a custom buffer size.
+ *
+ * @param bufferSize internal buffer size in characters
+ */
+ public ThaiTokenizer(int bufferSize) {
+ this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, null, bufferSize);
+ }
+
+ /**
+ * Creates a new ThaiTokenizer with a user dictionary and custom buffer size.
+ *
+ * @param userDictionary custom dictionary of words, or null for default dictionary only
+ * @param bufferSize internal buffer size in characters
+ */
+ public ThaiTokenizer(CharArraySet userDictionary, int bufferSize) {
+ this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, userDictionary, bufferSize);
+ }
+
/** Creates a new ThaiTokenizer, supplying the AttributeFactory */
public ThaiTokenizer(AttributeFactory factory) {
this(factory, null);
@@ -94,7 +116,18 @@ public class ThaiTokenizer extends SegmentingTokenizerBase {
* @param userDictionary custom dictionary of words, or null for default dictionary only
*/
public ThaiTokenizer(AttributeFactory factory, CharArraySet userDictionary) {
- super(factory, (BreakIterator) sentenceProto.clone());
+ this(factory, userDictionary, BUFFERMAX);
+ }
+
+ /**
+ * Creates a new ThaiTokenizer, supplying the AttributeFactory, user dictionary, and buffer size.
+ *
+ * @param factory AttributeFactory to use
+ * @param userDictionary custom dictionary of words, or null for default dictionary only
+ * @param bufferSize internal buffer size in characters
+ */
+ public ThaiTokenizer(AttributeFactory factory, CharArraySet userDictionary, int bufferSize) {
+ super(factory, (BreakIterator) sentenceProto.clone(), bufferSize);
if (!DBBI_AVAILABLE) {
throw new UnsupportedOperationException(
"This JRE does not have support for Thai segmentation");
@@ -117,6 +150,11 @@ public class ThaiTokenizer extends SegmentingTokenizerBase {
}
}
+ @Override
+ protected boolean isSafeEnd(char ch) {
+ return super.isSafeEnd(ch) || Character.isWhitespace(ch);
+ }
+
@Override
protected void setNextSentence(int sentenceStart, int sentenceEnd) {
this.sentenceStart = sentenceStart;
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizerFactory.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizerFactory.java
index b973db20bb4..dcceeb544e8 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizerFactory.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizerFactory.java
@@ -44,12 +44,14 @@ public class ThaiTokenizerFactory extends TokenizerFactory implements ResourceLo
public static final String NAME = "thai";
private final String dictFile;
+ private final int bufferSize;
private CharArraySet dictionary;
/** Creates a new ThaiTokenizerFactory */
public ThaiTokenizerFactory(Map<String, String> args) {
super(args);
dictFile = get(args, "dictionary");
+ bufferSize = getInt(args, "bufferSize", ThaiTokenizer.DEFAULT_BUFFER_SIZE);
if (!args.isEmpty()) {
throw new IllegalArgumentException("Unknown parameters: " + args);
}
@@ -69,6 +71,6 @@ public class ThaiTokenizerFactory extends TokenizerFactory implements ResourceLo
@Override
public Tokenizer create(AttributeFactory factory) {
- return new ThaiTokenizer(factory, dictionary);
+ return new ThaiTokenizer(factory, dictionary, bufferSize);
}
}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/util/SegmentingTokenizerBase.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/util/SegmentingTokenizerBase.java
index 9131ec3e456..227304606b0 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/util/SegmentingTokenizerBase.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/util/SegmentingTokenizerBase.java
@@ -36,8 +36,8 @@ import org.apache.lucene.util.AttributeFactory;
* @lucene.experimental
*/
public abstract class SegmentingTokenizerBase extends Tokenizer {
- protected static final int BUFFERMAX = 1024;
- protected final char[] buffer = new char[BUFFERMAX];
+ public static final int BUFFERMAX = 1024;
+ protected final char[] buffer;
/** true length of text in the buffer */
private int length = 0;
@@ -60,12 +60,37 @@ public abstract class SegmentingTokenizerBase extends Tokenizer {
* newly created or cloned one should always be provided to this constructor.
*/
public SegmentingTokenizerBase(BreakIterator iterator) {
- this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, iterator);
+ this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, iterator, BUFFERMAX);
+ }
+
+ /**
+ * Construct a new SegmenterBase, using the provided BreakIterator and buffer size.
+ *
+ * @param iterator BreakIterator for sentence segmentation
+ * @param bufferSize internal buffer size in characters
+ */
+ public SegmentingTokenizerBase(BreakIterator iterator, int bufferSize) {
+ this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, iterator, bufferSize);
}
/** Construct a new SegmenterBase, also supplying the AttributeFactory */
public SegmentingTokenizerBase(AttributeFactory factory, BreakIterator iterator) {
+ this(factory, iterator, BUFFERMAX);
+ }
+
+ /**
+ * Construct a new SegmenterBase, supplying the AttributeFactory, BreakIterator, and buffer size.
+ *
+ * @param factory AttributeFactory to use
+ * @param iterator BreakIterator for sentence segmentation
+ * @param bufferSize internal buffer size in characters
+ */
+ public SegmentingTokenizerBase(AttributeFactory factory, BreakIterator iterator, int bufferSize) {
super(factory);
+ if (bufferSize <= 0) {
+ throw new IllegalArgumentException("bufferSize must be > 0");
+ }
+ this.buffer = new char[bufferSize];
this.iterator = iterator;
}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizer.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizer.java
index 264aee2598d..56780dac821 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizer.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiTokenizer.java
@@ -63,4 +63,51 @@ public class TestThaiTokenizer extends BaseTokenStreamTestCase {
tokenizer.setReader(new StringReader("ภาษาไทย"));
assertTokenStreamContents(tokenizer, new String[] {"ภาษา", "ไทย"});
}
+
+ public void testLongTextAcrossBufferBoundary() throws IOException {
+ // Generate long Thai text (> 1024 chars) with spaces as clause boundaries
+ StringBuilder sb = new StringBuilder();
+ for (int i = 0; i < 110; i++) {
+ sb.append("ภาษาไทย "); // 9 chars per repetition, 110 * 9 = 990 chars
+ }
+ // Now at ~990 chars; add words spanning across index 1024
+ sb.append("มหาวิทยาลัย "); // 12 chars -> total ~1002
+ sb.append("กรุงเทพมหานคร "); // 14 chars -> total ~1016
+ sb.append("ประเทศไทย"); // 9 chars -> total ~1025, crossing 1024 boundary
+
+ Tokenizer tokenizer = new ThaiTokenizer();
+ tokenizer.setReader(new StringReader(sb.toString()));
+ tokenizer.reset();
+ boolean foundCrossingWord = false;
+ org.apache.lucene.analysis.tokenattributes.CharTermAttribute termAtt =
+ tokenizer.addAttribute(org.apache.lucene.analysis.tokenattributes.CharTermAttribute.class);
+ while (tokenizer.incrementToken()) {
+ String term = termAtt.toString();
+ if ("ประเทศ".equals(term)) {
+ foundCrossingWord = true;
+ }
+ assertFalse("Word should not be truncated into fragment 'ประ'", "ประ".equals(term));
+ assertFalse("Word should not be truncated into fragment 'เทศ'", "เทศ".equals(term));
+ }
+ tokenizer.end();
+ tokenizer.close();
+ assertTrue("Should find intact term 'ประเทศ'", foundCrossingWord);
+ }
+
+ public void testCustomBufferSize() throws IOException {
+ // Tiny buffer size of 32 chars
+ Tokenizer tokenizer = new ThaiTokenizer(32);
+ tokenizer.setReader(new StringReader("การทดสอบ ภาษาไทย มหาวิทยาลัย"));
+ assertTokenStreamContents(
+ tokenizer, new String[] {"การ", "ทดสอบ", "ภาษา", "ไทย", "มหาวิทยาลัย"});
+ }
+
+ public void testFactoryWithBufferSize() throws IOException {
+ java.util.Map<String, String> args = new java.util.HashMap<>();
+ args.put("bufferSize", "2048");
+ ThaiTokenizerFactory factory = new ThaiTokenizerFactory(args);
+ Tokenizer tokenizer = factory.create();
+ tokenizer.setReader(new StringReader("ภาษาไทย"));
+ assertTokenStreamContents(tokenizer, new String[] {"ภาษา", "ไทย"});
+ }
}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/util/TestSegmentingTokenizerBase.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/util/TestSegmentingTokenizerBase.java
index 20af3bd0e08..3befb1024f3 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/util/TestSegmentingTokenizerBase.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/util/TestSegmentingTokenizerBase.java
@@ -149,6 +149,20 @@ public class TestSegmentingTokenizerBase extends BaseTokenStreamTestCase {
checkRandomData(random(), sentenceAndWord, 200 * RANDOM_MULTIPLIER);
}
+ /** Test customizable buffer size with tiny buffer */
+ public void testCustomBufferSize() throws IOException {
+ Analyzer a =
+ new Analyzer() {
+ @Override
+ protected TokenStreamComponents createComponents(String fieldName) {
+ return new TokenStreamComponents(new WholeSentenceTokenizer(16));
+ }
+ };
+ assertAnalyzesTo(
+ a, "Sentence one.\nSentence two.", new String[] {"Sentence one.\n", "Sentence two."});
+ a.close();
+ }
+
// some tokenizers for testing
/** silly tokenizer that just returns whole sentences as tokens */
@@ -160,7 +174,11 @@ public class TestSegmentingTokenizerBase extends BaseTokenStreamTestCase {
private OffsetAttribute offsetAtt = addAttribute(OffsetAttribute.class);
public WholeSentenceTokenizer() {
- super(newAttributeFactory(), BreakIterator.getSentenceInstance(Locale.ROOT));
+ this(BUFFERMAX);
+ }
+
+ public WholeSentenceTokenizer(int bufferSize) {
+ super(newAttributeFactory(), BreakIterator.getSentenceInstance(Locale.ROOT), bufferSize);
}
@Override
(lucene) 01/06: Add ThaiCharFilter, ThaiNormalizationFilter, and ThaiNormalizer (#16716) (#16717)
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmuir pushed a commit to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 405116dd0e10a4921d8269bfaa7b74c69992f3e6
Author: Kamthorn Krairaksa <ka...@users.noreply.github.com>
AuthorDate: Mon Sep 28 04:09:27 2026 +0700
Add ThaiCharFilter, ThaiNormalizationFilter, and ThaiNormalizer (#16716) (#16717)
* Add ThaiNormalizationFilter and ThaiNormalizer to Thai analysis pipeline
* Add ThaiCharFilter and ThaiCharFilterFactory for pre-tokenization normalization (#16716)
- Implements ThaiCharFilter extending BaseCharFilter to normalize character anomalies before tokenization.
- Crucial for Thai: typographic anomalies (such as double Sara E 'เเ' or separated Sara Am) cause JDK BreakIterator to fail boundary detection and incorrectly merge adjacent words. Pre-tokenization char filtering restores correct word boundaries.
- Replaces double Sara E ('เ'+'เ' -> 'แ'), recomposes Sara Am, removes Zero-Width characters, deduplicates marks, and correctly records offset corrections via addOffCorrectMap.
- Adds ThaiCharFilterFactory (SPI: 'thai') registered in CharFilterFactory service.
- Wraps Reader with ThaiCharFilter in ThaiAnalyzer (initReader and initReaderForNormalization).
- Adds TestThaiCharFilter and TestThaiCharFilterFactory.
* Update CHANGES.txt for #16717
* Apply spotless formatting to Thai CharFilter and Normalizer
* Fix: register ThaiCharFilterFactory and ThaiNormalizationFilterFactory in module-info.java for JPMS discovery
---
lucene/analysis/common/src/java/module-info.java | 4 +-
.../apache/lucene/analysis/th/ThaiAnalyzer.java | 18 +-
.../apache/lucene/analysis/th/ThaiCharFilter.java | 211 +++++++++++++++++++++
.../lucene/analysis/th/ThaiCharFilterFactory.java | 64 +++++++
.../analysis/th/ThaiNormalizationFilter.java | 50 +++++
.../th/ThaiNormalizationFilterFactory.java | 64 +++++++
.../apache/lucene/analysis/th/ThaiNormalizer.java | 175 +++++++++++++++++
.../org.apache.lucene.analysis.CharFilterFactory | 1 +
.../org.apache.lucene.analysis.TokenFilterFactory | 1 +
.../lucene/analysis/th/TestThaiAnalyzer.java | 19 ++
.../lucene/analysis/th/TestThaiCharFilter.java | 133 +++++++++++++
.../analysis/th/TestThaiCharFilterFactory.java | 43 +++++
.../th/TestThaiNormalizationFilterFactory.java | 49 +++++
.../lucene/analysis/th/TestThaiNormalizer.java | 110 +++++++++++
14 files changed, 938 insertions(+), 4 deletions(-)
diff --git a/lucene/analysis/common/src/java/module-info.java b/lucene/analysis/common/src/java/module-info.java
index b61549e41a6..4f262a8ab7b 100644
--- a/lucene/analysis/common/src/java/module-info.java
+++ b/lucene/analysis/common/src/java/module-info.java
@@ -149,7 +149,8 @@ module org.apache.lucene.analysis.common {
org.apache.lucene.analysis.charfilter.MappingCharFilterFactory,
org.apache.lucene.analysis.cjk.CJKWidthCharFilterFactory,
org.apache.lucene.analysis.fa.PersianCharFilterFactory,
- org.apache.lucene.analysis.pattern.PatternReplaceCharFilterFactory;
+ org.apache.lucene.analysis.pattern.PatternReplaceCharFilterFactory,
+ org.apache.lucene.analysis.th.ThaiCharFilterFactory;
provides org.apache.lucene.analysis.TokenFilterFactory with
org.apache.lucene.analysis.tr.ApostropheFilterFactory,
org.apache.lucene.analysis.ar.ArabicNormalizationFilterFactory,
@@ -262,6 +263,7 @@ module org.apache.lucene.analysis.common {
org.apache.lucene.analysis.core.FlattenGraphFilterFactory,
org.apache.lucene.analysis.te.TeluguNormalizationFilterFactory,
org.apache.lucene.analysis.te.TeluguStemFilterFactory,
+ org.apache.lucene.analysis.th.ThaiNormalizationFilterFactory,
org.apache.lucene.analysis.tr.TurkishLowerCaseFilterFactory,
org.apache.lucene.analysis.util.ElisionFilterFactory;
provides org.apache.lucene.analysis.TokenizerFactory with
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiAnalyzer.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiAnalyzer.java
index 12595982f9a..715237810af 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiAnalyzer.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiAnalyzer.java
@@ -94,15 +94,15 @@ public final class ThaiAnalyzer extends StopwordAnalyzerBase {
* the text in the provided {@link Reader}.
*
* @return {@link org.apache.lucene.analysis.Analyzer.TokenStreamComponents} built from a {@link
- * ThaiTokenizer} filtered with {@link LowerCaseFilter}, {@link DecimalDigitFilter} and {@link
- * StopFilter}
+ * ThaiTokenizer} filtered with {@link LowerCaseFilter}, {@link DecimalDigitFilter}, {@link
+ * ThaiNormalizationFilter} and {@link StopFilter}
*/
@Override
protected TokenStreamComponents createComponents(String fieldName) {
final Tokenizer source = new ThaiTokenizer();
TokenStream result = new LowerCaseFilter(source);
result = new DecimalDigitFilter(result);
-
+ result = new ThaiNormalizationFilter(result);
result = new StopFilter(result, stopwords);
return new TokenStreamComponents(source, result);
}
@@ -111,6 +111,18 @@ public final class ThaiAnalyzer extends StopwordAnalyzerBase {
protected TokenStream normalize(String fieldName, TokenStream in) {
TokenStream result = new LowerCaseFilter(in);
result = new DecimalDigitFilter(result);
+ result = new ThaiNormalizationFilter(result);
return result;
}
+
+ /** Wraps the Reader with {@link ThaiCharFilter} */
+ @Override
+ protected Reader initReader(String fieldName, Reader reader) {
+ return new ThaiCharFilter(reader);
+ }
+
+ @Override
+ protected Reader initReaderForNormalization(String fieldName, Reader reader) {
+ return new ThaiCharFilter(reader);
+ }
}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiCharFilter.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiCharFilter.java
new file mode 100644
index 00000000000..6577a6b1d3e
--- /dev/null
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiCharFilter.java
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.IOException;
+import java.io.Reader;
+import org.apache.lucene.analysis.charfilter.BaseCharFilter;
+
+/**
+ * A {@link org.apache.lucene.analysis.CharFilter} that normalizes Thai text at the character stream
+ * level before tokenization.
+ *
+ * <p>Applying normalization before tokenization is critical for Thai, because typographic anomalies
+ * (such as two consecutive Sara E characters instead of Sara Ae, or zero-width characters) prevent
+ * the dictionary-based {@link java.text.BreakIterator} from locating word boundaries, causing
+ * multiple adjacent words to be mistakenly merged into one token.
+ *
+ * <p>Normalizations performed:
+ *
+ * <ul>
+ * <li>Removes Zero-Width characters (U+200B ZWSP, U+200C ZWNJ, U+200D ZWJ, U+FEFF BOM)
+ * <li>Replaces double Sara E (เ + เ, U+0E40 U+0E40) with Sara Ae (แ, U+0E41)
+ * <li>Recomposes decomposed Sara Am (U+0E4D + U+0E32 → U+0E33, and U+0E4D + tone + U+0E32
+ * → tone + U+0E33)
+ * <li>Deduplicates consecutive repeated diacritics / tone marks (e.g. ดีี → ดี)
+ * <li>Reorders misplaced tone mark before an above/below vowel to canonical order
+ * <li>Replaces Lakkhangyao (ๅ, U+0E45) with Sara Aa (า, U+0E32) unless preceded by Ru (ฤ) or Lu
+ * (ฦ)
+ * </ul>
+ *
+ * <p>All character deletions and contractions correctly update the character offset map using
+ * {@link #addOffCorrectMap(int, int)}.
+ *
+ * @since 11.0.0
+ */
+public class ThaiCharFilter extends BaseCharFilter {
+
+ private final char[] inputBuffer = new char[8];
+ private int inBufLen = 0;
+ private boolean eof = false;
+
+ private int outputOff = 0;
+ private int cumulativeDiff = 0;
+ private char prevChar = 0;
+ private char pendingOutputChar = 0;
+
+ /**
+ * Creates a new ThaiCharFilter wrapping the given {@link Reader}.
+ *
+ * @param in the input reader
+ */
+ public ThaiCharFilter(Reader in) {
+ super(in);
+ }
+
+ @Override
+ public int read() throws IOException {
+ if (pendingOutputChar != 0) {
+ char c = pendingOutputChar;
+ pendingOutputChar = 0;
+ prevChar = c;
+ outputOff++;
+ return c;
+ }
+
+ while (true) {
+ fillBuffer();
+
+ if (inBufLen == 0) {
+ return -1;
+ }
+
+ char c0 = inputBuffer[0];
+
+ // 1. Zero-width character removal (U+200B, U+200C, U+200D, U+FEFF)
+ if (c0 == '\u200B' || c0 == '\u200C' || c0 == '\u200D' || c0 == '\uFEFF') {
+ shiftBuffer(1);
+ cumulativeDiff++;
+ addOffCorrectMap(outputOff, cumulativeDiff);
+ continue;
+ }
+
+ // 2. Deduplicate consecutive identical diacritics / tone marks
+ if (isThaiDiacritic(c0) && c0 == prevChar) {
+ shiftBuffer(1);
+ cumulativeDiff++;
+ addOffCorrectMap(outputOff, cumulativeDiff);
+ continue;
+ }
+
+ // 3. Double Sara E (เ + เ -> แ)
+ if (c0 == '\u0E40' && inBufLen >= 2 && inputBuffer[1] == '\u0E40') {
+ shiftBuffer(2);
+ cumulativeDiff++;
+ addOffCorrectMap(outputOff + 1, cumulativeDiff);
+ prevChar = '\u0E41';
+ outputOff++;
+ return '\u0E41';
+ }
+
+ // 4. Sara Am recomposition
+ if (c0 == '\u0E4D') {
+ // Subcase 4A: Nikhahit + Sara Aa (ํ + า -> ำ)
+ if (inBufLen >= 2 && inputBuffer[1] == '\u0E32') {
+ shiftBuffer(2);
+ cumulativeDiff++;
+ addOffCorrectMap(outputOff + 1, cumulativeDiff);
+ prevChar = '\u0E33';
+ outputOff++;
+ return '\u0E33';
+ }
+ // Subcase 4B: Nikhahit + Tone + Sara Aa (ํ + tone + า -> tone + ำ)
+ if (inBufLen >= 3 && isThaiTone(inputBuffer[1]) && inputBuffer[2] == '\u0E32') {
+ char tone = inputBuffer[1];
+ shiftBuffer(3);
+ pendingOutputChar = '\u0E33';
+ cumulativeDiff++;
+ addOffCorrectMap(outputOff + 2, cumulativeDiff);
+ prevChar = tone;
+ outputOff++;
+ return tone;
+ }
+ }
+
+ // 5. Misplaced tone mark before above/below vowel: swap to canonical order
+ if (isThaiTone(c0) && inBufLen >= 2 && isThaiAboveBelowVowel(inputBuffer[1])) {
+ char vowel = inputBuffer[1];
+ char tone = c0;
+ inputBuffer[0] = vowel;
+ inputBuffer[1] = tone;
+ // Continue loop to process the swapped vowel at index 0
+ continue;
+ }
+
+ // 6. Lakkhangyao (ๅ, U+0E45) -> Sara Aa (า, U+0E32) unless preceded by Ru (ฤ) or Lu (ฦ)
+ if (c0 == '\u0E45') {
+ if (prevChar != '\u0E24' && prevChar != '\u0E26') {
+ c0 = '\u0E32';
+ }
+ }
+
+ // Default: emit character c0
+ shiftBuffer(1);
+ prevChar = c0;
+ outputOff++;
+ return c0;
+ }
+ }
+
+ @Override
+ public int read(char[] cbuf, int off, int len) throws IOException {
+ if (len <= 0) {
+ return 0;
+ }
+ int numRead = 0;
+ for (int i = off; i < off + len; i++) {
+ int c = read();
+ if (c == -1) {
+ break;
+ }
+ cbuf[i] = (char) c;
+ numRead++;
+ }
+ return numRead == 0 ? -1 : numRead;
+ }
+
+ private void fillBuffer() throws IOException {
+ while (!eof && inBufLen < 4) {
+ int ch = input.read();
+ if (ch == -1) {
+ eof = true;
+ break;
+ }
+ inputBuffer[inBufLen++] = (char) ch;
+ }
+ }
+
+ private void shiftBuffer(int count) {
+ int remaining = inBufLen - count;
+ if (remaining > 0) {
+ System.arraycopy(inputBuffer, count, inputBuffer, 0, remaining);
+ }
+ inBufLen = remaining;
+ }
+
+ private static boolean isThaiDiacritic(char c) {
+ return c == '\u0E31' || (c >= '\u0E34' && c <= '\u0E3A') || (c >= '\u0E47' && c <= '\u0E4E');
+ }
+
+ private static boolean isThaiTone(char c) {
+ return (c >= '\u0E48' && c <= '\u0E4B') || c == '\u0E4C';
+ }
+
+ private static boolean isThaiAboveBelowVowel(char c) {
+ return c == '\u0E31' || (c >= '\u0E34' && c <= '\u0E3A') || c == '\u0E47';
+ }
+}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiCharFilterFactory.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiCharFilterFactory.java
new file mode 100644
index 00000000000..dac127d4311
--- /dev/null
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiCharFilterFactory.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.Reader;
+import java.util.Map;
+import org.apache.lucene.analysis.CharFilterFactory;
+
+/**
+ * Factory for {@link ThaiCharFilter}.
+ *
+ * <pre class="prettyprint">
+ * <fieldType name="text_th" class="solr.TextField" positionIncrementGap="100">
+ * <analyzer>
+ * <charFilter name="thai"/>
+ * <tokenizer name="thai"/>
+ * </analyzer>
+ * </fieldType></pre>
+ *
+ * @since 11.0.0
+ * @lucene.spi {@value #NAME}
+ */
+public class ThaiCharFilterFactory extends CharFilterFactory {
+
+ /** SPI name */
+ public static final String NAME = "thai";
+
+ /** Default constructor for SPI */
+ public ThaiCharFilterFactory(Map<String, String> args) {
+ super(args);
+ if (!args.isEmpty()) {
+ throw new IllegalArgumentException("Unknown parameters: " + args);
+ }
+ }
+
+ /** Default constructor */
+ public ThaiCharFilterFactory() {
+ this(Map.of());
+ }
+
+ @Override
+ public Reader create(Reader input) {
+ return new ThaiCharFilter(input);
+ }
+
+ @Override
+ public Reader normalize(Reader input) {
+ return create(input);
+ }
+}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizationFilter.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizationFilter.java
new file mode 100644
index 00000000000..53f6c3dd471
--- /dev/null
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizationFilter.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.TokenFilter;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
+import org.apache.lucene.analysis.tokenattributes.KeywordAttribute;
+
+/**
+ * A {@link TokenFilter} that applies {@link ThaiNormalizer} to normalize Thai characters.
+ *
+ * @see ThaiNormalizer
+ */
+public final class ThaiNormalizationFilter extends TokenFilter {
+
+ private final ThaiNormalizer normalizer = new ThaiNormalizer();
+ private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
+ private final KeywordAttribute keywordAtt = addAttribute(KeywordAttribute.class);
+
+ public ThaiNormalizationFilter(TokenStream input) {
+ super(input);
+ }
+
+ @Override
+ public boolean incrementToken() throws IOException {
+ if (input.incrementToken()) {
+ if (!keywordAtt.isKeyword()) {
+ termAtt.setLength(normalizer.normalize(termAtt.buffer(), termAtt.length()));
+ }
+ return true;
+ }
+ return false;
+ }
+}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizationFilterFactory.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizationFilterFactory.java
new file mode 100644
index 00000000000..a5ab4a0cd29
--- /dev/null
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizationFilterFactory.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.util.Map;
+import org.apache.lucene.analysis.TokenFilterFactory;
+import org.apache.lucene.analysis.TokenStream;
+
+/**
+ * Factory for {@link ThaiNormalizationFilter}.
+ *
+ * <pre><code class="language-xml">
+ * <fieldType name="text_thnormal" class="solr.TextField" positionIncrementGap="100">
+ * <analyzer>
+ * <tokenizer class="solr.ThaiTokenizerFactory"/>
+ * <filter class="solr.ThaiNormalizationFilterFactory"/>
+ * </analyzer>
+ * </fieldType></code></pre>
+ *
+ * @since 11.0.0
+ * @lucene.spi {@value #NAME}
+ */
+public class ThaiNormalizationFilterFactory extends TokenFilterFactory {
+
+ /** SPI name */
+ public static final String NAME = "thaiNormalization";
+
+ /** Creates a new ThaiNormalizationFilterFactory */
+ public ThaiNormalizationFilterFactory(Map<String, String> args) {
+ super(args);
+ if (!args.isEmpty()) {
+ throw new IllegalArgumentException("Unknown parameters: " + args);
+ }
+ }
+
+ /** Default ctor for compatibility with SPI */
+ public ThaiNormalizationFilterFactory() {
+ throw defaultCtorException();
+ }
+
+ @Override
+ public TokenStream create(TokenStream input) {
+ return new ThaiNormalizationFilter(input);
+ }
+
+ @Override
+ public TokenStream normalize(TokenStream input) {
+ return create(input);
+ }
+}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizer.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizer.java
new file mode 100644
index 00000000000..5ce1c08fa7b
--- /dev/null
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiNormalizer.java
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import static org.apache.lucene.analysis.util.StemmerUtil.delete;
+
+/**
+ * Normalizer for Thai.
+ *
+ * <p>Normalizes Thai text to handle common orthographic variations, typos, and encoding anomalies:
+ *
+ * <ul>
+ * <li>Removes zero-width characters (ZWSP U+200B, ZWNJ U+200C).
+ * <li>Replaces double Sara E (U+0E40 U+0E40) with Sara Ae (U+0E41).
+ * <li>Recomposes decomposed Sara Am (Nikhahit U+0E4D + optional tone mark + Sara Aa U+0E32) into
+ * canonical Sara Am (U+0E33).
+ * <li>Normalizes Lakkhangyao (U+0E45) to Sara Aa (U+0E32) unless preceded by Ru (U+0E24) or Lu
+ * (U+0E26).
+ * <li>Reorders misplaced tone marks (U+0E48-U+0E4B) or Thanthakhat (U+0E4C) with above/below
+ * vowels (U+0E31, U+0E34-U+0E3A, U+0E47) to canonical Unicode order (vowel before tone mark).
+ * <li>Reorders follow vowels (e.g. Sara Aa U+0E32, Sara Am U+0E33) and misplaced trailing tone
+ * marks.
+ * <li>Deduplicates repeated vowels, tone marks, and diacritics.
+ * <li>Removes dangling combining marks at the start of a token.
+ * </ul>
+ */
+public class ThaiNormalizer {
+
+ /**
+ * Normalize an input buffer of Thai text.
+ *
+ * @param s input buffer
+ * @param len length of input buffer
+ * @return length of input buffer after normalization
+ */
+ public int normalize(char[] s, int len) {
+ if (len == 0) {
+ return 0;
+ }
+
+ // Step 1: Remove zero-width characters
+ for (int i = 0; i < len; i++) {
+ if (s[i] == '\u200B' || s[i] == '\u200C') {
+ len = delete(s, i, len);
+ i--;
+ }
+ }
+
+ // Remove leading dangling non-base marks at token start
+ while (len > 0 && isDanglingMark(s[0])) {
+ len = delete(s, 0, len);
+ }
+
+ // Step 2: Handle compositions (Double Sara E -> Sara Ae, Sara Am recomposition, Lakkhangyao)
+ for (int i = 0; i < len; i++) {
+ char c = s[i];
+
+ // Double Sara E -> Sara Ae
+ if (c == '\u0E40' && i + 1 < len && s[i + 1] == '\u0E40') {
+ s[i] = '\u0E41';
+ len = delete(s, i + 1, len);
+ continue;
+ }
+
+ // Nikhahit (U+0E4D) + optional tone mark + Sara Aa (U+0E32) -> optional tone mark + Sara Am
+ // (U+0E33)
+ if (c == '\u0E4D') {
+ if (i + 1 < len && s[i + 1] == '\u0E32') {
+ s[i] = '\u0E33';
+ len = delete(s, i + 1, len);
+ continue;
+ } else if (i + 2 < len && isToneMark(s[i + 1]) && s[i + 2] == '\u0E32') {
+ // Nikhahit + Tone + Sara Aa -> Tone + Sara Am
+ char tone = s[i + 1];
+ s[i] = tone;
+ s[i + 1] = '\u0E33';
+ len = delete(s, i + 2, len);
+ continue;
+ }
+ }
+
+ // Tone mark + Nikhahit + Sara Aa -> Tone + Sara Am
+ if (isToneMark(c) && i + 2 < len && s[i + 1] == '\u0E4D' && s[i + 2] == '\u0E32') {
+ s[i + 1] = '\u0E33';
+ len = delete(s, i + 2, len);
+ continue;
+ }
+
+ // Lakkhangyao (U+0E45) -> Sara Aa (U+0E32) unless preceded by Ru (U+0E24) or Lu (U+0E26)
+ if (c == '\u0E45') {
+ if (i == 0 || (s[i - 1] != '\u0E24' && s[i - 1] != '\u0E26')) {
+ s[i] = '\u0E32';
+ }
+ }
+ }
+
+ // Step 3: Canonical reordering
+ // Standard order: Consonant -> Above/Below Vowel -> Tone mark / Thanthakhat
+ // Also: Follow vowel (Sara Aa / Sara Am) + Tone mark -> Tone mark + Follow vowel
+ boolean changed = true;
+ while (changed) {
+ changed = false;
+ for (int i = 0; i < len - 1; i++) {
+ // Tone mark / Thanthakhat followed by Above/Below Vowel
+ if (isToneMarkOrThanthakhat(s[i]) && isAboveOrBelowVowel(s[i + 1])) {
+ char tmp = s[i];
+ s[i] = s[i + 1];
+ s[i + 1] = tmp;
+ changed = true;
+ }
+ // Follow vowel followed by Tone mark (e.g., Sara Am + Tone -> Tone + Sara Am)
+ else if (isFollowVowel(s[i]) && isToneMark(s[i + 1])) {
+ char tmp = s[i];
+ s[i] = s[i + 1];
+ s[i + 1] = tmp;
+ changed = true;
+ }
+ }
+ }
+
+ // Step 4: Deduplicate repeated vowels, tone marks, and diacritics
+ for (int i = 0; i < len - 1; i++) {
+ // Identical consecutive combining mark or vowel
+ if (isDeduplicable(s[i]) && s[i] == s[i + 1]) {
+ len = delete(s, i + 1, len);
+ i--;
+ }
+ // If two different consecutive tone marks appear, keep the last one
+ else if (isToneMark(s[i]) && isToneMark(s[i + 1])) {
+ len = delete(s, i, len);
+ i--;
+ }
+ }
+
+ return len;
+ }
+
+ private static boolean isToneMark(char c) {
+ return c >= '\u0E48' && c <= '\u0E4B';
+ }
+
+ private static boolean isToneMarkOrThanthakhat(char c) {
+ return isToneMark(c) || c == '\u0E4C';
+ }
+
+ private static boolean isAboveOrBelowVowel(char c) {
+ return c == '\u0E31' || (c >= '\u0E34' && c <= '\u0E3A') || c == '\u0E47';
+ }
+
+ private static boolean isFollowVowel(char c) {
+ return c == '\u0E30' || c == '\u0E32' || c == '\u0E33' || c == '\u0E45';
+ }
+
+ private static boolean isDanglingMark(char c) {
+ return isAboveOrBelowVowel(c) || isToneMarkOrThanthakhat(c) || c == '\u0E4D' || c == '\u0E4E';
+ }
+
+ private static boolean isDeduplicable(char c) {
+ return (c >= '\u0E30' && c <= '\u0E3A') || c == '\u0E45' || (c >= '\u0E47' && c <= '\u0E4E');
+ }
+}
diff --git a/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.CharFilterFactory b/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.CharFilterFactory
index c9f43ec2df7..2851b7f152a 100644
--- a/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.CharFilterFactory
+++ b/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.CharFilterFactory
@@ -18,3 +18,4 @@ org.apache.lucene.analysis.charfilter.MappingCharFilterFactory
org.apache.lucene.analysis.cjk.CJKWidthCharFilterFactory
org.apache.lucene.analysis.fa.PersianCharFilterFactory
org.apache.lucene.analysis.pattern.PatternReplaceCharFilterFactory
+org.apache.lucene.analysis.th.ThaiCharFilterFactory
diff --git a/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.TokenFilterFactory b/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.TokenFilterFactory
index 1b33f2603b3..b5b276803bf 100644
--- a/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.TokenFilterFactory
+++ b/lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.TokenFilterFactory
@@ -124,5 +124,6 @@ org.apache.lucene.analysis.synonym.word2vec.Word2VecSynonymFilterFactory
org.apache.lucene.analysis.core.FlattenGraphFilterFactory
org.apache.lucene.analysis.te.TeluguNormalizationFilterFactory
org.apache.lucene.analysis.te.TeluguStemFilterFactory
+org.apache.lucene.analysis.th.ThaiNormalizationFilterFactory
org.apache.lucene.analysis.tr.TurkishLowerCaseFilterFactory
org.apache.lucene.analysis.util.ElisionFilterFactory
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java
index bd8c9f8440a..9bca83c8f7b 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiAnalyzer.java
@@ -146,4 +146,23 @@ public class TestThaiAnalyzer extends BaseTokenStreamTestCase {
new int[] {4, 7, 9, 14, 19, 22, 25, 29, 33, 36, 39, 41});
analyzer.close();
}
+
+ public void testNormalization() throws Exception {
+ Analyzer analyzer = new ThaiAnalyzer(CharArraySet.EMPTY_SET);
+ // Double Sara E -> Sara Ae
+ assertAnalyzesTo(
+ analyzer, "\u0E40\u0E40\u0E1B\u0E25\u0E01", new String[] {"\u0E41\u0E1B\u0E25\u0E01"});
+ // Decomposed Sara Am -> Sara Am (pre-tokenization allows BreakIterator to properly segment ทำ
+ // and งาน)
+ assertAnalyzesTo(
+ analyzer,
+ "\u0E17\u0E4D\u0E32\u0E07\u0E32\u0E19",
+ new String[] {"\u0E17\u0E33", "\u0E07\u0E32\u0E19"});
+ // Sentence with double Sara E properly segmented into individual words thanks to ThaiCharFilter
+ assertAnalyzesTo(
+ analyzer, "ฉันรัก\u0E40\u0E40มวมาก", new String[] {"ฉัน", "รัก", "แมว", "มาก"});
+ // Duplicate vowels
+ assertAnalyzesTo(analyzer, "\u0E14\u0E35\u0E35", new String[] {"\u0E14\u0E35"});
+ analyzer.close();
+ }
}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiCharFilter.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiCharFilter.java
new file mode 100644
index 00000000000..c8901bf6b34
--- /dev/null
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiCharFilter.java
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.IOException;
+import java.io.Reader;
+import java.io.StringReader;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.CharFilter;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.Tokenizer;
+import org.apache.lucene.tests.analysis.BaseTokenStreamTestCase;
+import org.apache.lucene.tests.analysis.MockTokenizer;
+
+/** Test case for {@link ThaiCharFilter}. */
+public class TestThaiCharFilter extends BaseTokenStreamTestCase {
+
+ /** Double Sara E (เ + เ) normalized to Sara Ae (แ) with offset correction */
+ public void testDoubleSaraE() throws IOException {
+ CharFilter reader = new ThaiCharFilter(new StringReader("เเมว"));
+ TokenStream ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(ts, new String[] {"แมว"}, new int[] {0}, new int[] {4}, 4);
+ }
+
+ /** Sara Am recomposition (ํ + า -> ำ, and ํ + tone + า -> tone + ำ) */
+ public void testSaraAmRecomposition() throws IOException {
+ // น + ํ + ้ + า (4 chars) -> น้ำ (3 chars)
+ CharFilter reader = new ThaiCharFilter(new StringReader("น\u0E4D\u0E49\u0E32"));
+ TokenStream ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(ts, new String[] {"น้ำ"}, new int[] {0}, new int[] {4}, 4);
+
+ // ท + ํ + า (3 chars) -> ทำ (2 chars)
+ reader = new ThaiCharFilter(new StringReader("ท\u0E4D\u0E32"));
+ ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(ts, new String[] {"ทำ"}, new int[] {0}, new int[] {3}, 3);
+ }
+
+ /** Consecutive repeated diacritics / tone marks deduplication */
+ public void testDuplicateDiacritics() throws IOException {
+ // ด + ี + ี (3 chars) -> ดี (2 chars)
+ CharFilter reader = new ThaiCharFilter(new StringReader("ดีี"));
+ TokenStream ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(ts, new String[] {"ดี"}, new int[] {0}, new int[] {3}, 3);
+
+ // ไ + ม + ้ + ้ (4 chars) -> ไม้ (3 chars)
+ reader = new ThaiCharFilter(new StringReader("ไม้้"));
+ ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(ts, new String[] {"ไม้"}, new int[] {0}, new int[] {4}, 4);
+ }
+
+ /** Zero-width space and joiner removal */
+ public void testZeroWidth() throws IOException {
+ // สวัส + \u200B + ดี (7 chars in input) -> สวัสดี (6 chars in output, endOffset 7)
+ CharFilter reader = new ThaiCharFilter(new StringReader("สวัส\u200Bดี"));
+ TokenStream ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(ts, new String[] {"สวัสดี"}, new int[] {0}, new int[] {7}, 7);
+ }
+
+ /** Misplaced tone mark before above/below vowel swapped to canonical order */
+ public void testToneReordering() throws IOException {
+ // บ + ่ + ั + น -> บ + ั + ่ + น (บั่น)
+ CharFilter reader = new ThaiCharFilter(new StringReader("บ\u0E48\u0E31น"));
+ TokenStream ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(ts, new String[] {"บั่น"}, new int[] {0}, new int[] {4}, 4);
+ }
+
+ /** Lakkhangyao (ๅ) converted to Sara Aa (า) except after Ru (ฤ) or Lu (ฦ) */
+ public void testLakkhangyao() throws IOException {
+ CharFilter reader = new ThaiCharFilter(new StringReader("ฤๅ ฦๅ กๅ"));
+ TokenStream ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(
+ ts, new String[] {"ฤๅ", "ฦๅ", "กา"}, new int[] {0, 3, 6}, new int[] {2, 5, 8}, 8);
+ }
+
+ /**
+ * Demonstrates that pre-tokenization char filtering allows BreakIterator to segment words
+ * correctly that would otherwise be mistakenly merged into one huge token.
+ */
+ public void testBreakIteratorSegmentationWithCharFilter() throws IOException {
+ assumeTrue(
+ "JRE does not support Thai dictionary-based BreakIterator", ThaiTokenizer.DBBI_AVAILABLE);
+ // "ฉันรักเเมวมาก" (contains double Sara E in เเมว)
+ // Without ThaiCharFilter, BreakIterator lumps "รักเเมวมาก" into 1 token.
+ // With ThaiCharFilter, it normalizes to "ฉันรักแมวมาก" and segments all 4 words properly.
+ Reader reader = new ThaiCharFilter(new StringReader("ฉันรักเเมวมาก"));
+ Tokenizer tokenizer = new ThaiTokenizer();
+ tokenizer.setReader(reader);
+ assertTokenStreamContents(
+ tokenizer,
+ new String[] {"ฉัน", "รัก", "แมว", "มาก"},
+ new int[] {0, 3, 6, 10},
+ new int[] {3, 6, 10, 13},
+ 13);
+ }
+
+ public void testEmpty() throws IOException {
+ CharFilter reader = new ThaiCharFilter(new StringReader(""));
+ TokenStream ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(ts, new String[] {});
+ }
+
+ public void testRandom() throws IOException {
+ Analyzer analyzer =
+ new Analyzer() {
+ @Override
+ protected TokenStreamComponents createComponents(String fieldName) {
+ Tokenizer tokenizer = new MockTokenizer(MockTokenizer.WHITESPACE, false);
+ return new TokenStreamComponents(tokenizer, tokenizer);
+ }
+
+ @Override
+ protected Reader initReader(String fieldName, Reader reader) {
+ return new ThaiCharFilter(reader);
+ }
+ };
+ checkRandomData(random(), analyzer, 200 * RANDOM_MULTIPLIER);
+ analyzer.close();
+ }
+}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiCharFilterFactory.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiCharFilterFactory.java
new file mode 100644
index 00000000000..83974e93775
--- /dev/null
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiCharFilterFactory.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.Reader;
+import java.io.StringReader;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.tests.analysis.BaseTokenStreamFactoryTestCase;
+
+/** Simple tests to ensure the Thai char filter factory is working. */
+public class TestThaiCharFilterFactory extends BaseTokenStreamFactoryTestCase {
+
+ public void testCharFilter() throws Exception {
+ Reader reader = charFilterFactory("Thai").create(new StringReader("เเมว"));
+ TokenStream ts = whitespaceMockTokenizer(reader);
+ assertTokenStreamContents(ts, new String[] {"แมว"});
+ }
+
+ /** Test that bogus arguments result in exception */
+ public void testBogusArguments() throws Exception {
+ IllegalArgumentException expected =
+ expectThrows(
+ IllegalArgumentException.class,
+ () -> {
+ charFilterFactory("Thai", "bogusArg", "bogusValue");
+ });
+ assertTrue(expected.getMessage().contains("Unknown parameters"));
+ }
+}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiNormalizationFilterFactory.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiNormalizationFilterFactory.java
new file mode 100644
index 00000000000..594f3dd0500
--- /dev/null
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiNormalizationFilterFactory.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.Reader;
+import java.io.StringReader;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.Tokenizer;
+import org.apache.lucene.tests.analysis.BaseTokenStreamFactoryTestCase;
+import org.apache.lucene.tests.analysis.MockTokenizer;
+
+/** Simple tests to ensure the Thai normalization factory is working. */
+public class TestThaiNormalizationFilterFactory extends BaseTokenStreamFactoryTestCase {
+
+ public void testNormalization() throws Exception {
+ Reader reader =
+ new StringReader("\u0E40\u0E40\u0E1B\u0E25\u0E01 \u0E17\u0E4D\u0E32\u0E07\u0E32\u0E19");
+ TokenStream stream = new MockTokenizer(MockTokenizer.WHITESPACE, false);
+ ((Tokenizer) stream).setReader(reader);
+ stream = tokenFilterFactory("ThaiNormalization").create(stream);
+ assertTokenStreamContents(
+ stream, new String[] {"\u0E41\u0E1B\u0E25\u0E01", "\u0E17\u0E33\u0E07\u0E32\u0E19"});
+ }
+
+ /** Test that bogus arguments result in exception */
+ public void testBogusArguments() throws Exception {
+ IllegalArgumentException expected =
+ expectThrows(
+ IllegalArgumentException.class,
+ () -> {
+ tokenFilterFactory("ThaiNormalization", "bogusArg", "bogusValue");
+ });
+ assertTrue(expected.getMessage().contains("Unknown parameters"));
+ }
+}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiNormalizer.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiNormalizer.java
new file mode 100644
index 00000000000..8b760582446
--- /dev/null
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/TestThaiNormalizer.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.th;
+
+import java.io.IOException;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.TokenFilter;
+import org.apache.lucene.analysis.Tokenizer;
+import org.apache.lucene.analysis.core.KeywordTokenizer;
+import org.apache.lucene.tests.analysis.BaseTokenStreamTestCase;
+
+/** Test case for {@link ThaiNormalizer} and {@link ThaiNormalizationFilter}. */
+public class TestThaiNormalizer extends BaseTokenStreamTestCase {
+
+ public void testDoubleSaraE() throws IOException {
+ // Two Sara E (U+0E40 U+0E40) should become Sara Ae (U+0E41)
+ check("\u0E40\u0E40\u0E1B\u0E25\u0E01", "\u0E41\u0E1B\u0E25\u0E01"); // เเปลก -> แปลก
+ }
+
+ public void testSaraAmDecomposition() throws IOException {
+ // Nikhahit (U+0E4D) + Sara Aa (U+0E32) -> Sara Am (U+0E33)
+ check(
+ "\u0E17\u0E4D\u0E32\u0E07\u0E32\u0E19",
+ "\u0E17\u0E33\u0E07\u0E32\u0E19"); // ทํางาน -> ทำงาน
+
+ // Nikhahit + Tone (U+0E49) + Sara Aa -> Tone + Sara Am
+ check("\u0E19\u0E4D\u0E49\u0E32", "\u0E19\u0E49\u0E33"); // นํ้า -> น้ำ
+
+ // Tone + Nikhahit + Sara Aa -> Tone + Sara Am
+ check("\u0E19\u0E49\u0E4D\u0E32", "\u0E19\u0E49\u0E33"); // น้ำ
+ }
+
+ public void testSaraAmWithFollowedTone() throws IOException {
+ // Sara Am followed by Tone mark -> Tone mark + Sara Am
+ check("\u0E19\u0E33\u0E49", "\u0E19\u0E49\u0E33");
+ }
+
+ public void testReorderToneAndAboveVowel() throws IOException {
+ // Tone mark before Above Vowel (Mai Tho before Sara I) -> Sara I before Mai Tho
+ check("\u0E01\u0E49\u0E34", "\u0E01\u0E34\u0E49"); // ก้ิ -> กิ้
+ }
+
+ public void testDuplicateVowelsAndTones() throws IOException {
+ // Duplicate Sara Ii
+ check("\u0E14\u0E35\u0E35", "\u0E14\u0E35"); // ดีี -> ดี
+ // Duplicate Mai Tho
+ check("\u0E41\u0E21\u0E49\u0E49", "\u0E41\u0E21\u0E49"); // แม้้ -> แม้
+ // Duplicate Sara Aa
+ check("\u0E19\u0E32\u0E32", "\u0E19\u0E32"); // นาา -> นา
+ }
+
+ public void testConsecutiveDifferentTones() throws IOException {
+ // When multiple different tone marks are typed, keep the last one
+ check("\u0E44\u0E21\u0E48\u0E49", "\u0E44\u0E21\u0E49"); // ไม่้ -> ไม้
+ }
+
+ public void testZeroWidthCharacters() throws IOException {
+ check(
+ "\u0E20\u0E32\u0E29\u0E32\u200B\u0E44\u0E17\u0E22",
+ "\u0E20\u0E32\u0E29\u0E32\u0E44\u0E17\u0E22");
+ check(
+ "\u0E20\u0E32\u0E29\u0E32\u200C\u0E44\u0E17\u0E22",
+ "\u0E20\u0E32\u0E29\u0E32\u0E44\u0E17\u0E22");
+ }
+
+ public void testLakkhangyao() throws IOException {
+ // Preceded by Ru: preserve
+ check("\u0E24\u0E45", "\u0E24\u0E45");
+ // Not preceded by Ru or Lu: normalize to Sara Aa
+ check("\u0E01\u0E45", "\u0E01\u0E32");
+ }
+
+ public void testDanglingMark() throws IOException {
+ // Leading dangling tone mark without consonant
+ check("\u0E49\u0E01\u0E32", "\u0E01\u0E32");
+ }
+
+ public void testEmptyTerm() throws IOException {
+ Analyzer a =
+ new Analyzer() {
+ @Override
+ protected TokenStreamComponents createComponents(String fieldName) {
+ Tokenizer tokenizer = new KeywordTokenizer();
+ return new TokenStreamComponents(tokenizer, new ThaiNormalizationFilter(tokenizer));
+ }
+ };
+ checkOneTerm(a, "", "");
+ a.close();
+ }
+
+ private void check(String input, String output) throws IOException {
+ Tokenizer tokenizer = whitespaceMockTokenizer(input);
+ TokenFilter tf = new ThaiNormalizationFilter(tokenizer);
+ assertTokenStreamContents(tf, new String[] {output});
+ }
+}
(lucene) 06/06: docs: move Thai improvements to 10.6
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmuir pushed a commit to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit c2e4612e051be4a95a5b176791457a27e7d1c5c5
Author: Robert Muir <rm...@apache.org>
AuthorDate: Mon Sep 28 11:32:55 2026 -0400
docs: move Thai improvements to 10.6
---
lucene/CHANGES.txt | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index e75ade07b94..116ef3ba6ad 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -27,6 +27,15 @@ API Changes
New Features
---------------------
+* GITHUB#16722: Support user dictionary in ThaiTokenizer and ThaiTokenizerFactory.
+ (Kamthorn Krairaksa)
+
+* GITHUB#16717: Add ThaiCharFilter, ThaiCharFilterFactory, and ThaiNormalizationFilter for
+ pre-tokenization character normalization and token-level normalization in ThaiAnalyzer.
+ (Kamthorn Krairaksa)
+
+* GITHUB#16720: Add ThaiRepeatFilter and ThaiRepeatFilterFactory to expand Thai repetition
+ mark (Maiyamok) with exact OffsetAttribute and highlighter compatibility. (Kamthorn Krairaksa)
* GITHUB#16680: Add a "subword" type to the subword tokens generated by
the compound word token filters, so downstream filters can distinguish
@@ -62,6 +71,8 @@ New Features
Improvements
---------------------
+* GITHUB#16718: Modernize ThaiAnalyzer default stopwords list based on contemporary Thai corpus
+ frequency analysis and linguistically validated stop words. (Kamthorn Krairaksa)
* GITHUB#16298: Store doc values DISI data into a separate file. (Sagar Upadhyaya)
@@ -181,6 +192,8 @@ Bug Fixes
---------------------
* GITHUB#16546: Prevent RangeBulkScorer from passing empty ranges to LeafCollector. (jxy)
+* GITHUB#10153: Allow configurable buffer size in SegmentingTokenizerBase and recognize whitespace as safe boundary in ThaiTokenizer to avoid splitting words across buffer boundaries. (Kamthorn Krairaksa)
+
* GITHUB#16620: Make HNSW merges abortable during graph initialization and repair, so that the
copy, repair and rebalance phases of an initialized graph build can be promptly aborted when the
surrounding merge is cancelled. (Tanguy Leroux)