[GitHub] [lucene] mikemccand commented on issue #10520: Improve FST performance created by Builder [LUCENE-9481]

["mikemccand (via GitHub)" <gi...@apache.org>]

mikemccand commented on issue #10520:
URL: https://github.com/apache/lucene/issues/10520#issuecomment-1737708991

   Coming back to this fun FST issue ...
   
   Given that FSTs are now readable off-heap, if we simply [fixed FST writing to also be off-heap](https://github.com/apache/lucene/issues/12543) (this should not be so hard since the writing is already append-only), then we wouldn't have these silly performance problems because we are re-inventing an on-heap filesystem (paged byte[]), badly.
   
   But a less radical fix here would be good too (progress not perfection) -- growing the `byte[]` maybe (though this is maybe problematic for larger FSTs since you need at least 2X the final size of the FST max heap size).  Or, when the FST is done being built, allocate a single byte[] and copy the pages to it?  (But the same 2X+ heap size requirement is there too).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org