[jira] [Created] (LUCENE-7258) Tune Spatial RPT Intersects allocation rate

["Jeff Wartes (JIRA)" <ji...@apache.org>]

Jeff Wartes created LUCENE-7258:
-----------------------------------

             Summary: Tune Spatial RPT Intersects allocation rate
                 Key: LUCENE-7258
                 URL: https://issues.apache.org/jira/browse/LUCENE-7258
             Project: Lucene - Core
          Issue Type: Improvement
          Components: modules/spatial
            Reporter: Jeff Wartes


LUCENE-7211 converted IntersectsPrefixTreeQuery to use DocIdSetBuilder, but didn't actually reduce garbage generation for my Solr index.

Since something like 40% of my garbage (by space) is now attributed to DocIdSetBuilder.growBuffer, I charted a few different allocation strategies to see if I could tune things more. 

See here: http://i.imgur.com/7sXLAYv.jpg 
The jump-then-flatline at the right would be where DocIdSetBuilder gives up and allocates a FixedBitSet for a 100M-doc index. (The 1M-doc index curve/cutoff looked similar)

Perhaps unsurprisingly, the 1/8th growth factor in ArrayUtil.oversize is terrible from an allocation standpoint if you're doing a lot of expansions, and is especially terrible when used to build a short-lived data structure like this one.
By the time it goes with the FBS, it's allocated around twice as much memory for the buffer as it would have needed for just the FBS.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org