You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@opennlp.apache.org by "Kristian Rickert (Jira)" <ji...@apache.org> on 2026/07/16 01:46:00 UTC

[jira] [Commented] (OPENNLP-1883) Thread-safe stemmers: StemmerFactory, thread-safe SnowballStemmer, per-thread stem caching

    [ https://issues.apache.org/jira/browse/OPENNLP-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096646#comment-18096646 ] 

Kristian Rickert commented on OPENNLP-1883:
-------------------------------------------

PR 1166 (light-stemmers, OPENNLP-1886) is built on top of PR 1163 (stemmer-factory, OPENNLP-1883) in git, same as the wordnet pair. Adding the link once from 1883 automatically shows as is blocked by on 1886, so when you get to 1886 it only needs its epic link.

> Thread-safe stemmers: StemmerFactory, thread-safe SnowballStemmer, per-thread stem caching
> ------------------------------------------------------------------------------------------
>
>                 Key: OPENNLP-1883
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1883
>             Project: OpenNLP
>          Issue Type: Improvement
>          Components: Stemmer
>    Affects Versions: 3.0.0-M4
>            Reporter: Kristian Rickert
>            Assignee: Kristian Rickert
>            Priority: Major
>             Fix For: 3.0.0-M5
>
>          Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> {{SnowballStemmer}} holds mutable per-call buffers in the generated engine, so sharing an instance across threads silently corrupts results — no exception, just wrong stems. With 3.0.0 unreleased, now is the time to align stemmers with the thread-safe {{*ME}} components.
>  - {{SnowballStemmer}} routes each call to a per-thread generated engine via {{OwnerOrPerThreadState}} (same pattern as the {{*ME}} classes); the generated Snowball code is untouched. One instance is now safe to share; {{@ThreadSafe}}.
>  - New {{StemmerFactory}} (opennlp-api): an immutable, thread-safe capture of stemmer configuration, with {{SnowballStemmerFactory}} and {{PorterStemmerFactory}} implementations and a {{SharingStemmer}} adapter for engines that are not thread-safe themselves (e.g. {{PorterStemmer}}).
>  - New {{CachingStemmer}}: bounded per-thread LRU memoization of word→stem mappings. Natural text is Zipf-distributed, so most tokens are repeats; JMH shows ~34x throughput on a Zipf workload and break-even on a cache-hostile one.{{TermAnalyzer.Builder.stem(StemmerFactory)}} and {{NormalizationProfile.matchingAnalyzer()}} use it, so profile analyzers are shareable and cached by default.
>  - Tests: concurrency unit tests for all 21 algorithms (fixed pools, owner-transition, virtual threads), {{MultiThreadedStemmerEval}} mirroring {{MultiThreadedToolsEval}}, JMH benchmarks with results in BENCHMARKS.md.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)