You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Bruno Roustant (Jira)" <ji...@apache.org> on 2019/12/05 13:55:00 UTC
[jira] [Commented] (LUCENE-9064) Can we remove the FST cache in Kuromoji and Nori analyzers?
[ https://issues.apache.org/jira/browse/LUCENE-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988828#comment-16988828 ]
Bruno Roustant commented on LUCENE-9064:
----------------------------------------
So I benchmarked Kuromoji analyzer with TestJapaneseTokenizer.testWikipedia. I clearly saw that if I removed the cache, then the token throughput was reduced by half, in a repeatable way.
*Conclusion*: we must not remove the root arcs cache for those analysers.
This made me think more about the recent removal of the FST root arcs cache. I wanted to verify/understand the difference. The Kuromoji tokenizer calls directly and heavily FST.findTargetArc(). So to benchmark FST in a similar way, I called FSTEnum.seekExact() in loops, for terms matching/not matching. And indeed the FST root arcs cache brings perf improvement, but less visibly than for the above analyzers (that's why we don't see it in luceneutil benchmark).
* FST root arcs cache brings around 5% perf improvement on seekExact (between 4% and 9% in my tests) (it was more before direct-addressing).
* This cache size is around 5KB (0.5% for 1MB FST).
Given that direct-addressing reduced the FST size by nearly 2%, we could easily restore the FST root arcs cache to get back some perf. But on the other hand we currently have FST size reduced by around 2.5% and still a nice perf, so I'm inclined to keep the FST without cache. What is your opinion on this [~sokolov] [~rcmuir] [~jpountz]?
> Can we remove the FST cache in Kuromoji and Nori analyzers?
> -----------------------------------------------------------
>
> Key: LUCENE-9064
> URL: https://issues.apache.org/jira/browse/LUCENE-9064
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Bruno Roustant
> Assignee: Bruno Roustant
> Priority: Minor
>
> Is the ~30k han cache in kuromoji redundant after LUCENE-8920?
> [https://github.com/apache/lucene-solr/blob/813ca77250db29116812bc949e2a466a70f969a3/lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/dict/TokenInfoFST.java#L35-L38])
> The entire linked file's purpose is all around this caching, so if its not needed anymore it would be a nice cleanup. But it was definitely needed for good performance before, so we shoudl be careful. Nori analyzer has the exact same thing (file has the same name) for ~10k hangul syllables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org