You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "reugn (via GitHub)" <gi...@apache.org> on 2026/10/01 09:52:13 UTC
[PR] Expose ramBytesUsed on Kuromoji and Nori UserDictionary [lucene]
reugn opened a new pull request, #16754:
URL: https://github.com/apache/lucene/pull/16754
Kuromoji and Nori accept a user dictionary of any size and compile it into an FST that lives as long as the analyzer. The FST already computes its own size, but nothing exposed it, so callers could not measure or bound what a user dictionary costs.
This makes `UserDictionary` in both modules, and the shared `morph.TokenInfoFST` they build on, implement `Accountable`. `ramBytesUsed()` covers the FST plus the per-entry data held next to it.
`morph.TokenInfoFST` also backs the bundled system dictionaries, so `TokenInfoDictionary.getInstance().getFST().ramBytesUsed()` now reports those too: about 2.2 MB for Japanese and 5.7 MB for Korean. The other dictionary classes are out of scope here, since their bundled instances are fixed in size and shared per JVM rather than built from caller input.
Additive only, no changes to existing behavior or signatures. The size is computed once at construction, since the state is immutable. Tests pin the reported sizes against independently computed values, including a `RamUsageTester` cross-check, and add the first test for `morph.TokenInfoFST`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Expose ramBytesUsed on Kuromoji and Nori UserDictionary [lucene]
Posted by "reugn (via GitHub)" <gi...@apache.org>.
reugn commented on PR #16754:
URL: https://github.com/apache/lucene/pull/16754#issuecomment-5935889229
The accounting circuit breaker that would have consumed this was removed in Elasticsearch. Closing this since there is no consumer for the number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Expose ramBytesUsed on Kuromoji and Nori UserDictionary [lucene]
Posted by "reugn (via GitHub)" <gi...@apache.org>.
reugn closed pull request #16754: Expose ramBytesUsed on Kuromoji and Nori UserDictionary
URL: https://github.com/apache/lucene/pull/16754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org