[jira] [Commented] (LUCENE-9930) UkrainianMorfologikAnalyzer reloads its Dictionary for every new TokenStreamComponents instance

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17334710#comment-17334710 ] 

ASF subversion and git services commented on LUCENE-9930:
---------------------------------------------------------

Commit 90d363ece7116954c530a74a014487fcbdee7610 in lucene's branch refs/heads/main from Alan Woodward
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=90d363e ]

LUCENE-9930: Only load Ukrainian morfologik dictionary once per JVM (#109)

The UkrainianMorfologikAnalyzer was reloading its dictionary every
time it created a new TokenStreamComponents, which meant that
while the analyzer was open it would hold onto one copy of the
dictionary per thread.

This commit loads the dictionary in a lazy static initializer, alongside
its stopword set. It also makes the normalizer charmap a singleton
so that we do not rebuild the same immutable object on every call
to initReader.

> UkrainianMorfologikAnalyzer reloads its Dictionary for every new TokenStreamComponents instance
> -----------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-9930
>                 URL: https://issues.apache.org/jira/browse/LUCENE-9930
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Alan Woodward
>            Assignee: Alan Woodward
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Large static data structures should be loaded in Analyzer constructors and shared between threads, but the UkrainianMorfologikAnalyzer is loading its dictionary in `createComponents`, which means it is reloaded and stored on every new analysis thread.  If you have a large dictionary and highly concurrent indexing then this can lead to you running out of memory as multiple copies of the dictionary are held in thread locals.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org