You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by John Wilson <sa...@gmail.com> on 2018/12/27 06:18:32 UTC
ByteBuffersDirectory throws Exception
Hi,
I'm getting the below error message while running a simple multi-threaded
indexing using ByteBuffersDirectory. Any suggestions?
Exception in thread "Lucene Merge Thread #879"
org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.store.AlreadyClosedException: refusing to delete any
files: this IndexWriter hit an unrecoverable exception
at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to
delete any files: this IndexWriter hit an unrecoverable exception
at
org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)
at
org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)
at
org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)
at
org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)
Caused by: java.lang.IllegalArgumentException: cannot write negative vLong
(got: -4294878395)
at
org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
at
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)
at
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
at
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
at
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
at
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
at
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
... 3 more
Re: ByteBuffersDirectory throws Exception
Posted by Dawid Weiss <da...@gmail.com>.
Looks like an int overflow to me, but I'd have to take a look at the
code to see where it can originate. Please file a Jira issue, attach
the full stack trace and name it "int overflow in
Lucene50SkipWriter.writeSkipData" because that's where the problem is.
We'll definitely take a look.
D.
On Thu, Dec 27, 2018 at 10:34 AM John Wilson <sa...@gmail.com> wrote:
>
> The data set I'm experimenting with is ~100G. If I break it up into four 25G data sets and index is individually, it works just fine. But if I try to index the entire 100G it breaks. I'm not sure if there is a size limitation or sth (FYI, I have a really large heap of size 256G).
>
> Thanks,
>
> On Thu, Dec 27, 2018 at 12:27 AM Dawid Weiss <da...@gmail.com> wrote:
>>
>> This looks like a bug, the underlying cause is:
>>
>> Caused by: java.lang.IllegalArgumentException: cannot write negative
>> vLong (got: -4294878395)
>> at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
>> at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)
>>
>> Can you reproduce it on a smaller example and file a Jira issue?
>>
>> Dawid
>>
>> On Thu, Dec 27, 2018 at 7:18 AM John Wilson <sa...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I'm getting the below error message while running a simple multi-threaded indexing using ByteBuffersDirectory. Any suggestions?
>> >
>> > Exception in thread "Lucene Merge Thread #879" org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
>> > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
>> > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
>> > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
>> > at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)
>> > at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)
>> > at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)
>> > at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)
>> > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)
>> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
>> > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
>> > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)
>> > Caused by: java.lang.IllegalArgumentException: cannot write negative vLong (got: -4294878395)
>> > at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
>> > at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)
>> > at org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
>> > at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
>> > at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
>> > at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
>> > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
>> > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
>> > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
>> > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
>> > at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
>> > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
>> > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
>> > ... 3 more
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: ByteBuffersDirectory throws Exception
Posted by John Wilson <sa...@gmail.com>.
The data set I'm experimenting with is ~100G. If I break it up into four
25G data sets and index is individually, it works just fine. But if I try
to index the entire 100G it breaks. I'm not sure if there is a size
limitation or sth (FYI, I have a really large heap of size 256G).
Thanks,
On Thu, Dec 27, 2018 at 12:27 AM Dawid Weiss <da...@gmail.com> wrote:
> This looks like a bug, the underlying cause is:
>
> Caused by: java.lang.IllegalArgumentException: cannot write negative
> vLong (got: -4294878395)
> at
> org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
> at
> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)
>
> Can you reproduce it on a smaller example and file a Jira issue?
>
> Dawid
>
> On Thu, Dec 27, 2018 at 7:18 AM John Wilson <sa...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I'm getting the below error message while running a simple
> multi-threaded indexing using ByteBuffersDirectory. Any suggestions?
> >
> > Exception in thread "Lucene Merge Thread #879"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.store.AlreadyClosedException: refusing to delete any
> files: this IndexWriter hit an unrecoverable exception
> > at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
> > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
> > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to
> delete any files: this IndexWriter hit an unrecoverable exception
> > at
> org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)
> > at
> org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)
> > at
> org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)
> > at
> org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)
> > at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)
> > at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
> > at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
> > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)
> > Caused by: java.lang.IllegalArgumentException: cannot write negative
> vLong (got: -4294878395)
> > at
> org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
> > at
> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)
> > at
> org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
> > at
> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
> > at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
> > at
> org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
> > at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
> > at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
> > at
> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
> > at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
> > at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
> > at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
> > at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
> > ... 3 more
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: ByteBuffersDirectory throws Exception
Posted by Dawid Weiss <da...@gmail.com>.
This looks like a bug, the underlying cause is:
Caused by: java.lang.IllegalArgumentException: cannot write negative
vLong (got: -4294878395)
at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)
Can you reproduce it on a smaller example and file a Jira issue?
Dawid
On Thu, Dec 27, 2018 at 7:18 AM John Wilson <sa...@gmail.com> wrote:
>
> Hi,
>
> I'm getting the below error message while running a simple multi-threaded indexing using ByteBuffersDirectory. Any suggestions?
>
> Exception in thread "Lucene Merge Thread #879" org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
> Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
> at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)
> at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)
> at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)
> at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)
> Caused by: java.lang.IllegalArgumentException: cannot write negative vLong (got: -4294878395)
> at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
> at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)
> at org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
> at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
> at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
> at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
> ... 3 more
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org