[jira] Updated: (LUCENE-2391) Spellchecker uses default IW mergefactor/ramMB settings of 300/10

["Robert Muir (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2391:
--------------------------------

    Attachment: LUCENE-2391.patch

Here's a patch to speed up the spellchecker build.

* i wired the default RamMB to IWConfig's default
* i didnt mess with the mergefactor for now (because the default is still to optimize)
* but i added an additional 'optimize' parameter so you can update your spellcheck index without re-optimizing.
* when updating, i changed the exists() to work per-segment, so its reasonable if the index isn't optimized.
* the exists() check now bypasses the term dictionary cache, which is stupid and just slows it down.
* we don't do any of the exists() logic if the index is empty (this is the case for i think solr which completely rebuilds
  and doesnt do an incremental update)
* the startXXX, endXXX, and word fields can only contain one term per document. I turned off norms, positions,
  and tf for these.
* the gramXXX field is unchanged, i didnt want to change spellchecker scoring in any way. But we could
  reasonably in the future likely omit norms here too since i think its gonna be very short.

{noformat}
trunk:
scratch build time: 229,803ms
index size: 214,322,200 bytes
no-op update time (updating but there is no new terms to add): 4,619ms

patch:
scratch build time: 99,214ms
index size: 177,781,273 bytes
no-op update time: 2,504ms
{noformat}

i still left the optimize default on, but really i think for most users (e.g. solr) they should set 
mergefactor to be maybe a bit more reasonable, set optimize to false, and the scratch build 
is then must faster (60,000 ms), but the no-op update time is heavier (eg 16,000ms). Still, 
if you are rebuilding on every commit for smallish updates something like 20-30 seconds 
is a lot better than 100seconds, but for now I kept the defaults as is (optimizing every time).


> Spellchecker uses default IW mergefactor/ramMB settings of 300/10
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2391
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2391
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: contrib/spellchecker
>            Reporter: Mark Miller
>            Priority: Trivial
>         Attachments: LUCENE-2391.patch
>
>
> These settings seem odd - I'd like to investigate what makes most sense here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org