[jira] [Commented] (LUCENE-7834) BlockTree's terms index should be loaded into memory lazily

["Robert Muir (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014608#comment-16014608 ] 

Robert Muir commented on LUCENE-7834:
-------------------------------------

I am not sure we should do this lazily, it can create a false sense of security.

Someone really does not have enough memory for their data, seems fine, then they hit merge (or calls addindexes) and all the fields are loaded and they get OOM.

> BlockTree's terms index should be loaded into memory lazily
> -----------------------------------------------------------
>
>                 Key: LUCENE-7834
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7834
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/codecs
>            Reporter: David Smiley
>            Assignee: David Smiley
>         Attachments: LUCENE_7834_BlockTreeLazyFST.patch
>
>
> I propose that BlockTree delay loading the FST prefix terms index into memory until {{terms(String)}} is first called.  This seems like how it should work since if someone wants to eager load then they can use {{IndexReaderWarmer}}.  By making the FST lazy load, we can be more NRT friendly (if some fields are rarely used), and also save memory (if some fields are rarely used).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org