You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "kamthorn (via GitHub)" <gi...@apache.org> on 2026/09/26 07:38:41 UTC
[I] Add ThaiRepeatFilter to handle Thai Maiyamok (ๆ) reduplication [lucene]
kamthorn opened a new issue, #16719:
URL: https://github.com/apache/lucene/issues/16719
### Description
In the Thai language, the character Maiyamok (`ๆ` / `U+0E46`) denotes reduplication (repetition) of the preceding word or phrase to convey emphasis, plurality, or frequency:
- `"เร็วๆ"` (or `"เร็ว ๆ"`) means `"เร็ว เร็ว"` (very fast)
- `"มากๆ"` (or `"มาก ๆ"`) means `"มาก มาก"` (very much)
- `"เด็กๆ"` means `"เด็ก เด็ก"` (children)
- `"บ่อยๆ"` means `"บ่อย บ่อย"` (frequently)
#### Current Problem:
When Thai text is tokenized (e.g. using `ThaiTokenizer` with `java.text.BreakIterator`, or `ICUTokenizer`), `ๆ` is classified in Unicode as `Character.MODIFIER_LETTER` and emitted as an isolated token:
`"เร็วๆ"` -> `["เร็ว", "ๆ"]`
This leads to several search retrieval issues:
1. **Term Frequency distortion**: The repeated word `"เร็ว"` only has a term frequency of 1 instead of 2.
2. **Phrase search failures**: A phrase query for `"เร็ว เร็ว"` will fail to match a document containing `"เร็วๆ"` because position 2 is occupied by `"ๆ"` instead of `"เร็ว"`.
3. **Punctuation noise**: The symbol `"ๆ"` is indexed as a searchable token despite having no independent lexical meaning.
4. **Standalone / attached token inconsistency**: Different tokenizers or input texts either emit `ๆ` as a separate token (`["เร็ว", "ๆ"]`) or attached (`["เร็วๆ"]`).
#### Proposed Solution:
We propose adding:
- `ThaiRepeatFilter`: A `TokenFilter` that duplicates the preceding term upon encountering Maiyamok (`ๆ`), handling both standalone tokens (`["เร็ว", "ๆ"]`) and tokens with trailing Maiyamok (`["เร็วๆ"]`).
- `ThaiRepeatFilterFactory`: SPI factory with name `thaiRepeat`.
- Integrate `ThaiRepeatFilter` into `ThaiAnalyzer`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Add ThaiRepeatFilter to handle Thai Maiyamok (ๆ) reduplication [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir closed issue #16719: Add ThaiRepeatFilter to handle Thai Maiyamok (ๆ) reduplication
URL: https://github.com/apache/lucene/issues/16719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org