You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Andras Salamon (Jira)" <ji...@apache.org> on 2021/04/13 12:24:00 UTC
[jira] [Created] (LUCENE-9927) Configurable BKDWriter maximum heap
size
Andras Salamon created LUCENE-9927:
--------------------------------------
Summary: Configurable BKDWriter maximum heap size
Key: LUCENE-9927
URL: https://issues.apache.org/jira/browse/LUCENE-9927
Project: Lucene - Core
Issue Type: Improvement
Reporter: Andras Salamon
With Lucene's default codec, when writing dimensional points, we only give {{BKDWriter}} 16 MB heap to use for sorting. If we reach this limit the codec will change to [OfflinePointWriter|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/bkd/OfflinePointWriter.java].
This is specially bad for Solr versions not containing SOLR-14242, because we get the following error:
{noformat}
Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot complete forceMerge
at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1998)
at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1940)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:662)
at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:102)
at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1079)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1066)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:72)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
... 15 more
Caused by: java.lang.UnsupportedOperationException
at org.apache.solr.store.hdfs.HdfsDirectory.createTempOutput(HdfsDirectory.java:119)
at org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:79)
at org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:79)
at org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:51)
at org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:51)
at org.apache.lucene.util.bkd.OfflinePointWriter.<init>(OfflinePointWriter.java:45)
at org.apache.lucene.util.bkd.BKDWriter.initPointWriter(BKDWriter.java:223)
at org.apache.lucene.util.bkd.BKDWriter.add(BKDWriter.java:241)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter$1.visit(Lucene60PointsWriter.java:123)
at org.apache.lucene.codecs.PointsWriter$1$1$1.visit(PointsWriter.java:118)
at org.apache.lucene.util.bkd.BKDReader.visitCompressedDocValues(BKDReader.java:735)
at org.apache.lucene.util.bkd.BKDReader.visitDocValuesWithCardinality(BKDReader.java:684)
at org.apache.lucene.util.bkd.BKDReader.visitDocValues(BKDReader.java:590)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:790)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:506)
at org.apache.lucene.codecs.PointsWriter$1$1.intersect(PointsWriter.java:106)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.writeField(Lucene60PointsWriter.java:116)
at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:63)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:227)
at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:206)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:164)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4467)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4059)
at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:201)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662){noformat}
It would be great to make this 16MB value configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org