You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2014/05/11 00:13:34 UTC
[jira] [Comment Edited] (LUCENE-5658) IllegalArgumentException from
ByteBufferIndexInput.buildSlice
[ https://issues.apache.org/jira/browse/LUCENE-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993117#comment-13993117 ]
Uwe Schindler edited comment on LUCENE-5658 at 5/8/14 9:15 PM:
---------------------------------------------------------------
bq. Perhaps there is a missing bounds check in the slicing logic!
Actually, this bug does not happen while slicing. It calls openFullSlice, which delegates to clone() on the indexinput, a well tested standard function. Clone() then duplicates all buffers after checking the bounds already:
{code}
@Override
public final ByteBufferIndexInput clone() {
final ByteBufferIndexInput clone = buildSlice(0L, this.length);
// ...
private ByteBufferIndexInput buildSlice(long offset, long length) {
if (buffers == null) {
throw new AlreadyClosedException("Already closed: " + this);
}
if (offset < 0 || length < 0 || offset+length > this.length) {
throw new IllegalArgumentException("slice() " + sliceDescription + " out of bounds: offset=" + offset + ",length=" + length + ",fileLength=" + this.length + ": " + this);
}
{code}
In that case the bounds are correct. It then calls the slicing logic which sets the limit on the last buffer. And this seems to get out of bounds because this.length and last file buffers limit are not synchronous.
This can only happen if the underlying JVM has a bug (J9???) or the underlying filesystem reports another file length after opening than it really is (maybe file was truncated).
We have a good slicer test already that opens a smaller file and uses a smaller buffer size for the byte buffers and randomly tests all stuff including opening a full slice. This code is unchanged since more than a year and we never had any bug, here, this is really strange.
was (Author: thetaphi):
Its not part of slicing. It calls openFullSlice, which calls clone() on the indexinput. Clone then duplicates all slices after checking the bounds already:
{code}
@Override
public final ByteBufferIndexInput clone() {
final ByteBufferIndexInput clone = buildSlice(0L, this.length);
// ...
private ByteBufferIndexInput buildSlice(long offset, long length) {
if (buffers == null) {
throw new AlreadyClosedException("Already closed: " + this);
}
if (offset < 0 || length < 0 || offset+length > this.length) {
throw new IllegalArgumentException("slice() " + sliceDescription + " out of bounds: offset=" + offset + ",length=" + length + ",fileLength=" + this.length + ": " + this);
}
{code}
In that case the bounds are correct. It then calls the slicing logic which sets the limit on the last buffer. And this seems to get out of bounds because this.length and last file buffers limit are not synchronous.
This can only happen if the underlying JVM has a bug (J9???) or the underlying filesystem reports another file length after opening than it really is (maybe file was truncated).
We have a good slicer test already that opens a smaller file and uses a smaller buffer size for the byte buffers and randomly tests all stuff including opening a full slice. This code is unchanged since more than a year and we never had any bug, here, this is really strange.
> IllegalArgumentException from ByteBufferIndexInput.buildSlice
> -------------------------------------------------------------
>
> Key: LUCENE-5658
> URL: https://issues.apache.org/jira/browse/LUCENE-5658
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
> Affects Versions: 4.8
> Reporter: Shai Erera
>
> I've received an email with the following stacktrace:
> {noformat}
> Exception in thread "Lucene Merge Thread #73" org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalArgumentException
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
> Caused by: java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:278)
> at org.apache.lucene.store.ByteBufferIndexInput.buildSlice(ByteBufferIndexInput.java:259)
> at org.apache.lucene.store.ByteBufferIndexInput.buildSlice(ByteBufferIndexInput.java:230)
> at org.apache.lucene.store.ByteBufferIndexInput.clone(ByteBufferIndexInput.java:187)
> at org.apache.lucene.store.MMapDirectory$1.openFullSlice(MMapDirectory.java:211)
> at org.apache.lucene.store.CompoundFileDirectory.readEntries(CompoundFileDirectory.java:138)
> at org.apache.lucene.store.CompoundFileDirectory.<init>(CompoundFileDirectory.java:105)
> at org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:209)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:99)
> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:142)
> at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:624)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4068)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3728)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
> {noformat}
> According to the email, it happens randomly while indexing Wikipedia, on 4.8.0. As far as I understood, the indexer creates 4 indexes in parallel, by a total of 48 threads. Each index is created in a separate directory, and there's no sharing of MP or MS instances between the writers (in fact, default settings are used). This could explain the {{Lucene Merge Thread #73}}. The indexing process ends w/ a {{forceMerge(1)}}. If that call is omitted, the exception doesn't reproduce. Also, since it doesn't happen always, there's no simple testcase which reproduces.
> I've asked the reporter to add more info to the issue, but opening the issue now so we could check and hopefully fix before 4.8.1.
> I checked the stacktrace against trunk, but not all the lines align (e.g. {{at org.apache.lucene.store.MMapDirectory$1.openFullSlice(MMapDirectory.java:211)}} is only in 4.8), but I haven't dug deeper yet...
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org