[jira] [Updated] (OPENNLP-1883) Make stemmers thread-safe, add StemmerFactory and per-thread stem caching

["Martin Wiesner (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Wiesner updated OPENNLP-1883:
------------------------------------
    Summary: Make stemmers thread-safe, add StemmerFactory and per-thread stem caching  (was: Thread-safe stemmers: StemmerFactory, thread-safe SnowballStemmer, per-thread stem caching)

> Make stemmers thread-safe, add StemmerFactory and per-thread stem caching
> -------------------------------------------------------------------------
>
>                 Key: OPENNLP-1883
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1883
>             Project: OpenNLP
>          Issue Type: Improvement
>          Components: Stemmer
>    Affects Versions: 3.0.0-M4
>            Reporter: Kristian Rickert
>            Assignee: Kristian Rickert
>            Priority: Major
>             Fix For: 3.0.0-M5
>
>          Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> {{SnowballStemmer}} holds mutable per-call buffers in the generated engine, so sharing an instance across threads silently corrupts results — no exception, just wrong stems. With 3.0.0 unreleased, now is the time to align stemmers with the thread-safe {{*ME}} components.
>  - {{SnowballStemmer}} routes each call to a per-thread generated engine via {{OwnerOrPerThreadState}} (same pattern as the {{*ME}} classes); the generated Snowball code is untouched. One instance is now safe to share; {{@ThreadSafe}}.
>  - New {{StemmerFactory}} (opennlp-api): an immutable, thread-safe capture of stemmer configuration, with {{SnowballStemmerFactory}} and {{PorterStemmerFactory}} implementations and a {{SharingStemmer}} adapter for engines that are not thread-safe themselves (e.g. {{PorterStemmer}}).
>  - New {{CachingStemmer}}: bounded per-thread LRU memoization of word→stem mappings. Natural text is Zipf-distributed, so most tokens are repeats; JMH shows ~34x throughput on a Zipf workload and break-even on a cache-hostile one.{{TermAnalyzer.Builder.stem(StemmerFactory)}} and {{NormalizationProfile.matchingAnalyzer()}} use it, so profile analyzers are shareable and cached by default.
>  - Tests: concurrency unit tests for all 21 algorithms (fixed pools, owner-transition, virtual threads), {{MultiThreadedStemmerEval}} mirroring {{MultiThreadedToolsEval}}, JMH benchmarks with results in BENCHMARKS.md.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)