You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Simon Willnauer (JIRA)" <ji...@apache.org> on 2014/05/28 12:39:01 UTC
[jira] [Updated] (LUCENE-4556) FuzzyTermsEnum creates tons of
objects
[ https://issues.apache.org/jira/browse/LUCENE-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4556:
------------------------------------
Assignee: Michael McCandless (was: Simon Willnauer)
> FuzzyTermsEnum creates tons of objects
> --------------------------------------
>
> Key: LUCENE-4556
> URL: https://issues.apache.org/jira/browse/LUCENE-4556
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search, modules/spellchecker
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Michael McCandless
> Priority: Critical
> Fix For: 4.9, 5.0
>
> Attachments: LUCENE-4556.patch, LUCENE-4556.patch
>
>
> I ran into this problem in production using the DirectSpellchecker. The number of objects created by the spellchecker shoot through the roof very very quickly. We ran about 130 queries and ended up with > 2M transitions / states. We spend 50% of the time in GC just because of transitions. Other parts of the system behave just fine here.
> I talked quickly to robert and gave a POC a shot providing a LevenshteinAutomaton#toRunAutomaton(prefix, n) method to optimize this case and build a array based strucuture converted into UTF-8 directly instead of going through the object based APIs. This involved quite a bit of changes but they are all package private at this point. I have a patch that still has a fair set of nocommits but its shows that its possible and IMO worth the trouble to make this really useable in production. All tests pass with the patch - its a start....
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org