[jira] [Updated] (LUCENE-7007) Reduce block-tree GC/CPU cost when flushing or merging postings

["Michael McCandless (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-7007:
---------------------------------------
    Attachment: LUCENE-7007.patch

Here's the patch (NOT committable!).

But I will commit the "XXX msec to flush YYY part of index" ... I think that can be helpful.

> Reduce block-tree GC/CPU cost when flushing or merging postings
> ---------------------------------------------------------------
>
>                 Key: LUCENE-7007
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7007
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>         Attachments: LUCENE-7007.patch
>
>
> Writing postings is a GC and CPU heavy operation now, in part because of how
> block tree recursively builds up the tree structure, by creating many
> tiny FSTs which it inefficiently merges together as it walks up the
> tree eventually to the root block.
> So I tried a quick prototype (patch attached) to use a
> less-RAM-efficient, but much fewer tiny FST related objects, when
> writing postings.
> But in some quick indexing performance tests (luceneutil), it makes no
> measurable improvements to indexing performance.
> So I'm putting my patch up here for posterity ... I don't intend to
> commit it unless we can iterate it further.  It adds code complexity,
> it's not committable as-is (we need to conditionalize it so it
> sometimes does use FSTs, for segments with many terms), etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org