You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Yonghui Zhao <zh...@gmail.com> on 2017/03/03 02:19:47 UTC
A flush exception in lucene 4.10.0
Hi all,
Anyone see this exception before? Is it a lucene bug or something wrong in
my code?
Exception in thread "Thread-14" java.lang.IllegalArgumentException:
maxValue must be non-negative (got: -1)
at
org.apache.lucene.util.packed.PackedInts.bitsRequired(PackedInts.java:1141)
at
org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(ForUtil.java:253)
at
org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(ForUtil.java:174)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.addPosition(Lucene41PostingsWriter.java:377)
at
org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:486)
at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:80)
at
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:114)
at
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:441)
at
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:510)
at
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:621)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:414)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:292)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:267)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:257)
at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:171)
Re: A flush exception in lucene 4.10.0
Posted by Yonghui Zhao <zh...@gmail.com>.
The exception in my case is related to one index field. The field is string
type, indexed, tokenized and stored, index option
is docsAndFreqsAndPositionsAndOffsets.
Each doc may index this field more than one times. For example doc 0 may
index it 3 times, 1st value is "A B C", 2nd is "D E F"，3rd is "G H I".
If I concatenate these values with space, "A B C D E F G H I" , so each doc
only index this field one 1 time, but has the same effect, no exception
occurs.
2017-03-10 13:43 GMT+08:00 Yonghui Zhao <zh...@gmail.com>:
> My version is 4.10.0 which is later than 4.5, but I didn't find the fix
> like https://issues.apache.org/jira/secure/attachment/
> 12593180/LUCENE-5116.patch in indexwriter, neither in 4.10.4. Not sure
> which version has fixed it.
>
>
> public void addIndexes(IndexReader... readers) throws IOException {
> ensureOpen();
> int numDocs = 0;
>
> try {
> if (infoStream.isEnabled("IW")) {
> infoStream.message("IW", "flush at addIndexes(IndexReader...)");
> }
> flush(false, true);
>
> String mergedName = newSegmentName();
> final List<AtomicReader> mergeReaders = new ArrayList<>();
> for (IndexReader indexReader : readers) {
> numDocs += indexReader.numDocs();
> for (AtomicReaderContext ctx : indexReader.leaves()) {
> mergeReaders.add(ctx.reader());
> }
> }
>
> // Make sure adding the new documents to this index won't
> // exceed the limit:
> reserveDocs(numDocs);
>
> final IOContext context = new IOContext(new MergeInfo(numDocs, -1,
> true, -1));
>
> 2017-03-09 22:44 GMT+08:00 Steve Rowe <sa...@gmail.com>:
>
>> Maybe <https://issues.apache.org/jira/browse/LUCENE-5116> (though it was
>> committed in Lucene 4.5)? Robert Muir pointed to this issue as fixing <
>> https://issues.apache.org/jira/browse/SOLR-5144>, which contains a
>> similar stack track to yours.
>>
>> --
>> Steve
>> www.lucidworks.com
>>
>> > On Mar 9, 2017, at 6:02 AM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>> >
>> > This seems likely to be a Lucene bug, and it seems vaguely familiar. I
>> > tried to find the issue / commit that may have fixed it, but so far
>> failed.
>> >
>> > But 4.10.0 is truly ancient; you should at least try upgrading to
>> 4.10.4?
>> >
>> > Mike McCandless
>> >
>> > http://blog.mikemccandless.com
>> >
>> > On Wed, Mar 8, 2017 at 6:15 AM, Yonghui Zhao <zh...@gmail.com>
>> wrote:
>> >
>> >> Seems it is related to an empty segment, all docs in this segment are
>> >> deleted before commit.
>> >> Any one can confirm it? Maybe I need upgrade my lucene version.
>> >>
>> >>
>> >> 2017-03-03 10:19 GMT+08:00 Yonghui Zhao <zh...@gmail.com>:
>> >>
>> >>> Hi all,
>> >>>
>> >>> Anyone see this exception before? Is it a lucene bug or something
>> wrong
>> >>> in my code?
>> >>>
>> >>>
>> >>> Exception in thread "Thread-14" java.lang.IllegalArgumentException:
>> >>> maxValue must be non-negative (got: -1)
>> >>> at org.apache.lucene.util.packed.PackedInts.bitsRequired(
>> >>> PackedInts.java:1141)
>> >>> at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
>> >>> ForUtil.java:253)
>> >>> at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
>> >>> ForUtil.java:174)
>> >>> at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
>> >>> addPosition(Lucene41PostingsWriter.java:377)
>> >>> at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
>> >>> FreqProxTermsWriterPerField.java:486)
>> >>> at org.apache.lucene.index.FreqProxTermsWriter.flush(
>> >>> FreqProxTermsWriter.java:80)
>> >>> at org.apache.lucene.index.DefaultIndexingChain.flush(
>> >>> DefaultIndexingChain.java:114)
>> >>> at org.apache.lucene.index.DocumentsWriterPerThread.flush(
>> >>> DocumentsWriterPerThread.java:441)
>> >>> at org.apache.lucene.index.DocumentsWriter.doFlush(
>> >>> DocumentsWriter.java:510)
>> >>> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
>> >>> DocumentsWriter.java:621)
>> >>> at org.apache.lucene.index.IndexWriter.getReader(
>> >>> IndexWriter.java:414)
>> >>> at org.apache.lucene.index.StandardDirectoryReader.
>> >>> doOpenFromWriter(StandardDirectoryReader.java:292)
>> >>> at org.apache.lucene.index.StandardDirectoryReader.
>> >>> doOpenIfChanged(StandardDirectoryReader.java:267)
>> >>> at org.apache.lucene.index.StandardDirectoryReader.
>> >>> doOpenIfChanged(StandardDirectoryReader.java:257)
>> >>> at org.apache.lucene.index.DirectoryReader.openIfChanged(
>> >>> DirectoryReader.java:171)
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: A flush exception in lucene 4.10.0
Posted by Yonghui Zhao <zh...@gmail.com>.
My version is 4.10.0 which is later than 4.5, but I didn't find the fix
like
https://issues.apache.org/jira/secure/attachment/12593180/LUCENE-5116.patch in
indexwriter, neither in 4.10.4. Not sure which version has fixed it.
public void addIndexes(IndexReader... readers) throws IOException {
ensureOpen();
int numDocs = 0;
try {
if (infoStream.isEnabled("IW")) {
infoStream.message("IW", "flush at addIndexes(IndexReader...)");
}
flush(false, true);
String mergedName = newSegmentName();
final List<AtomicReader> mergeReaders = new ArrayList<>();
for (IndexReader indexReader : readers) {
numDocs += indexReader.numDocs();
for (AtomicReaderContext ctx : indexReader.leaves()) {
mergeReaders.add(ctx.reader());
}
}
// Make sure adding the new documents to this index won't
// exceed the limit:
reserveDocs(numDocs);
final IOContext context = new IOContext(new MergeInfo(numDocs, -1,
true, -1));
2017-03-09 22:44 GMT+08:00 Steve Rowe <sa...@gmail.com>:
> Maybe <https://issues.apache.org/jira/browse/LUCENE-5116> (though it was
> committed in Lucene 4.5)? Robert Muir pointed to this issue as fixing <
> https://issues.apache.org/jira/browse/SOLR-5144>, which contains a
> similar stack track to yours.
>
> --
> Steve
> www.lucidworks.com
>
> > On Mar 9, 2017, at 6:02 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
> >
> > This seems likely to be a Lucene bug, and it seems vaguely familiar. I
> > tried to find the issue / commit that may have fixed it, but so far
> failed.
> >
> > But 4.10.0 is truly ancient; you should at least try upgrading to 4.10.4?
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> > On Wed, Mar 8, 2017 at 6:15 AM, Yonghui Zhao <zh...@gmail.com>
> wrote:
> >
> >> Seems it is related to an empty segment, all docs in this segment are
> >> deleted before commit.
> >> Any one can confirm it? Maybe I need upgrade my lucene version.
> >>
> >>
> >> 2017-03-03 10:19 GMT+08:00 Yonghui Zhao <zh...@gmail.com>:
> >>
> >>> Hi all,
> >>>
> >>> Anyone see this exception before? Is it a lucene bug or something wrong
> >>> in my code?
> >>>
> >>>
> >>> Exception in thread "Thread-14" java.lang.IllegalArgumentException:
> >>> maxValue must be non-negative (got: -1)
> >>> at org.apache.lucene.util.packed.PackedInts.bitsRequired(
> >>> PackedInts.java:1141)
> >>> at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
> >>> ForUtil.java:253)
> >>> at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
> >>> ForUtil.java:174)
> >>> at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
> >>> addPosition(Lucene41PostingsWriter.java:377)
> >>> at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
> >>> FreqProxTermsWriterPerField.java:486)
> >>> at org.apache.lucene.index.FreqProxTermsWriter.flush(
> >>> FreqProxTermsWriter.java:80)
> >>> at org.apache.lucene.index.DefaultIndexingChain.flush(
> >>> DefaultIndexingChain.java:114)
> >>> at org.apache.lucene.index.DocumentsWriterPerThread.flush(
> >>> DocumentsWriterPerThread.java:441)
> >>> at org.apache.lucene.index.DocumentsWriter.doFlush(
> >>> DocumentsWriter.java:510)
> >>> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
> >>> DocumentsWriter.java:621)
> >>> at org.apache.lucene.index.IndexWriter.getReader(
> >>> IndexWriter.java:414)
> >>> at org.apache.lucene.index.StandardDirectoryReader.
> >>> doOpenFromWriter(StandardDirectoryReader.java:292)
> >>> at org.apache.lucene.index.StandardDirectoryReader.
> >>> doOpenIfChanged(StandardDirectoryReader.java:267)
> >>> at org.apache.lucene.index.StandardDirectoryReader.
> >>> doOpenIfChanged(StandardDirectoryReader.java:257)
> >>> at org.apache.lucene.index.DirectoryReader.openIfChanged(
> >>> DirectoryReader.java:171)
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: A flush exception in lucene 4.10.0
Posted by Steve Rowe <sa...@gmail.com>.
Maybe <https://issues.apache.org/jira/browse/LUCENE-5116> (though it was committed in Lucene 4.5)? Robert Muir pointed to this issue as fixing <https://issues.apache.org/jira/browse/SOLR-5144>, which contains a similar stack track to yours.
--
Steve
www.lucidworks.com
> On Mar 9, 2017, at 6:02 AM, Michael McCandless <lu...@mikemccandless.com> wrote:
>
> This seems likely to be a Lucene bug, and it seems vaguely familiar. I
> tried to find the issue / commit that may have fixed it, but so far failed.
>
> But 4.10.0 is truly ancient; you should at least try upgrading to 4.10.4?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Mar 8, 2017 at 6:15 AM, Yonghui Zhao <zh...@gmail.com> wrote:
>
>> Seems it is related to an empty segment, all docs in this segment are
>> deleted before commit.
>> Any one can confirm it? Maybe I need upgrade my lucene version.
>>
>>
>> 2017-03-03 10:19 GMT+08:00 Yonghui Zhao <zh...@gmail.com>:
>>
>>> Hi all,
>>>
>>> Anyone see this exception before? Is it a lucene bug or something wrong
>>> in my code?
>>>
>>>
>>> Exception in thread "Thread-14" java.lang.IllegalArgumentException:
>>> maxValue must be non-negative (got: -1)
>>> at org.apache.lucene.util.packed.PackedInts.bitsRequired(
>>> PackedInts.java:1141)
>>> at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
>>> ForUtil.java:253)
>>> at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
>>> ForUtil.java:174)
>>> at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
>>> addPosition(Lucene41PostingsWriter.java:377)
>>> at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
>>> FreqProxTermsWriterPerField.java:486)
>>> at org.apache.lucene.index.FreqProxTermsWriter.flush(
>>> FreqProxTermsWriter.java:80)
>>> at org.apache.lucene.index.DefaultIndexingChain.flush(
>>> DefaultIndexingChain.java:114)
>>> at org.apache.lucene.index.DocumentsWriterPerThread.flush(
>>> DocumentsWriterPerThread.java:441)
>>> at org.apache.lucene.index.DocumentsWriter.doFlush(
>>> DocumentsWriter.java:510)
>>> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
>>> DocumentsWriter.java:621)
>>> at org.apache.lucene.index.IndexWriter.getReader(
>>> IndexWriter.java:414)
>>> at org.apache.lucene.index.StandardDirectoryReader.
>>> doOpenFromWriter(StandardDirectoryReader.java:292)
>>> at org.apache.lucene.index.StandardDirectoryReader.
>>> doOpenIfChanged(StandardDirectoryReader.java:267)
>>> at org.apache.lucene.index.StandardDirectoryReader.
>>> doOpenIfChanged(StandardDirectoryReader.java:257)
>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(
>>> DirectoryReader.java:171)
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: A flush exception in lucene 4.10.0
Posted by Michael McCandless <lu...@mikemccandless.com>.
This seems likely to be a Lucene bug, and it seems vaguely familiar. I
tried to find the issue / commit that may have fixed it, but so far failed.
But 4.10.0 is truly ancient; you should at least try upgrading to 4.10.4?
Mike McCandless
http://blog.mikemccandless.com
On Wed, Mar 8, 2017 at 6:15 AM, Yonghui Zhao <zh...@gmail.com> wrote:
> Seems it is related to an empty segment, all docs in this segment are
> deleted before commit.
> Any one can confirm it? Maybe I need upgrade my lucene version.
>
>
> 2017-03-03 10:19 GMT+08:00 Yonghui Zhao <zh...@gmail.com>:
>
> > Hi all,
> >
> > Anyone see this exception before? Is it a lucene bug or something wrong
> > in my code?
> >
> >
> > Exception in thread "Thread-14" java.lang.IllegalArgumentException:
> > maxValue must be non-negative (got: -1)
> > at org.apache.lucene.util.packed.PackedInts.bitsRequired(
> > PackedInts.java:1141)
> > at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
> > ForUtil.java:253)
> > at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
> > ForUtil.java:174)
> > at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
> > addPosition(Lucene41PostingsWriter.java:377)
> > at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
> > FreqProxTermsWriterPerField.java:486)
> > at org.apache.lucene.index.FreqProxTermsWriter.flush(
> > FreqProxTermsWriter.java:80)
> > at org.apache.lucene.index.DefaultIndexingChain.flush(
> > DefaultIndexingChain.java:114)
> > at org.apache.lucene.index.DocumentsWriterPerThread.flush(
> > DocumentsWriterPerThread.java:441)
> > at org.apache.lucene.index.DocumentsWriter.doFlush(
> > DocumentsWriter.java:510)
> > at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
> > DocumentsWriter.java:621)
> > at org.apache.lucene.index.IndexWriter.getReader(
> > IndexWriter.java:414)
> > at org.apache.lucene.index.StandardDirectoryReader.
> > doOpenFromWriter(StandardDirectoryReader.java:292)
> > at org.apache.lucene.index.StandardDirectoryReader.
> > doOpenIfChanged(StandardDirectoryReader.java:267)
> > at org.apache.lucene.index.StandardDirectoryReader.
> > doOpenIfChanged(StandardDirectoryReader.java:257)
> > at org.apache.lucene.index.DirectoryReader.openIfChanged(
> > DirectoryReader.java:171)
> >
>
Re: A flush exception in lucene 4.10.0
Posted by Yonghui Zhao <zh...@gmail.com>.
Seems it is related to an empty segment, all docs in this segment are
deleted before commit.
Any one can confirm it? Maybe I need upgrade my lucene version.
2017-03-03 10:19 GMT+08:00 Yonghui Zhao <zh...@gmail.com>:
> Hi all,
>
> Anyone see this exception before? Is it a lucene bug or something wrong
> in my code?
>
>
> Exception in thread "Thread-14" java.lang.IllegalArgumentException:
> maxValue must be non-negative (got: -1)
> at org.apache.lucene.util.packed.PackedInts.bitsRequired(
> PackedInts.java:1141)
> at org.apache.lucene.codecs.lucene41.ForUtil.bitsRequired(
> ForUtil.java:253)
> at org.apache.lucene.codecs.lucene41.ForUtil.writeBlock(
> ForUtil.java:174)
> at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.
> addPosition(Lucene41PostingsWriter.java:377)
> at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(
> FreqProxTermsWriterPerField.java:486)
> at org.apache.lucene.index.FreqProxTermsWriter.flush(
> FreqProxTermsWriter.java:80)
> at org.apache.lucene.index.DefaultIndexingChain.flush(
> DefaultIndexingChain.java:114)
> at org.apache.lucene.index.DocumentsWriterPerThread.flush(
> DocumentsWriterPerThread.java:441)
> at org.apache.lucene.index.DocumentsWriter.doFlush(
> DocumentsWriter.java:510)
> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(
> DocumentsWriter.java:621)
> at org.apache.lucene.index.IndexWriter.getReader(
> IndexWriter.java:414)
> at org.apache.lucene.index.StandardDirectoryReader.
> doOpenFromWriter(StandardDirectoryReader.java:292)
> at org.apache.lucene.index.StandardDirectoryReader.
> doOpenIfChanged(StandardDirectoryReader.java:267)
> at org.apache.lucene.index.StandardDirectoryReader.
> doOpenIfChanged(StandardDirectoryReader.java:257)
> at org.apache.lucene.index.DirectoryReader.openIfChanged(
> DirectoryReader.java:171)
>