You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by Will Palmeri <wi...@imeem.com> on 2008/06/28 03:22:41 UTC
IndexOutOfRangeException when adding Document to index
I just checked out the 2.3.1 code from svn and built a fresh dll to try
out with our existing codebase. (We had been using 2.0.5). A couple of
my indexes are working just fine, but for one of them when I get to
around 1000 documents, I get sporadic IndexOutOfRangeExceptions when
calling IndexWriter.AddDocument(Document)
I stepped through the code and drilled down to where the exception is
getting thrown. When processing a single term in a given field, the
method AllocSlice of the internal class ByteBlockPool in
DocumentsWriter.cs is called.
In AllocSlice, the variable level is computed by AND'ing a value from
slice array with 15 (0xf). However, in the next line 'level' is used to
index into DocumentsWriter.nextLevelArray. At runtime, this array was
only of length 10, so occasionally when level was >= 10, the
IndexOutOfRangeException would be thrown.
On line 3410, nextLevelArray is initialized:
internal static readonly int[] nextLevelArray = new int[]{1, 2, 3, 4, 5,
6, 7, 8, 9, 9};
I'm not familiar with the codebase, so I have no idea what this code
does specifically. But it seems to me that line 3510 should be:
int level = slice[upto] & DocumentsWriter.nextLevelArray.Length
I'm looking forward to using 2.3.1, hope this helps!
-Will
imeem.com
Re: IndexOutOfRangeException when adding Document to index
Posted by Eran Sevi <er...@gmail.com>.
Hi,
I have exactly the same problem. I'm trying to create an index with 3
fields. the field with the actual text contains a termpositionvector and
also payload for each token is the one causing the problem.
In a specific document, on a specific token, I get the same exception from
the same line. the slice value is 10 and thus 10&15 equals 10 and the
nextLevelArray throws the out of bounds exception.
Any suggestions would be appreciated,
Eran.
On Sat, Jun 28, 2008 at 5:55 AM, Will Palmeri <wi...@imeem.com> wrote:
> My earlier suggestion obviously wouldn't work. 15 is a bit mask, not the
> length :)
>
> Either nextLevelArray should be initialized with length 16, or a Min
> could be taken:
>
> int level = Math.Min(slice[upto] & 15,
> DocumentsWriter.nextLevelArray.Length);
>
> -Will
>
> -----Original Message-----
> From: Will Palmeri [mailto:will@imeem.com]
> Sent: Friday, June 27, 2008 6:23 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: IndexOutOfRangeException when adding Document to index
>
> I just checked out the 2.3.1 code from svn and built a fresh dll to try
> out with our existing codebase. (We had been using 2.0.5). A couple of
> my indexes are working just fine, but for one of them when I get to
> around 1000 documents, I get sporadic IndexOutOfRangeExceptions when
> calling IndexWriter.AddDocument(Document)
>
>
>
> I stepped through the code and drilled down to where the exception is
> getting thrown. When processing a single term in a given field, the
> method AllocSlice of the internal class ByteBlockPool in
> DocumentsWriter.cs is called.
>
>
>
> In AllocSlice, the variable level is computed by AND'ing a value from
> slice array with 15 (0xf). However, in the next line 'level' is used to
> index into DocumentsWriter.nextLevelArray. At runtime, this array was
> only of length 10, so occasionally when level was >= 10, the
> IndexOutOfRangeException would be thrown.
>
>
>
> On line 3410, nextLevelArray is initialized:
>
> internal static readonly int[] nextLevelArray = new int[]{1, 2, 3, 4, 5,
> 6, 7, 8, 9, 9};
>
>
>
> I'm not familiar with the codebase, so I have no idea what this code
> does specifically. But it seems to me that line 3510 should be:
>
> int level = slice[upto] & DocumentsWriter.nextLevelArray.Length
>
>
>
>
>
> I'm looking forward to using 2.3.1, hope this helps!
>
>
>
> -Will
>
> imeem.com
>
>
RE: IndexOutOfRangeException when adding Document to index
Posted by Will Palmeri <wi...@imeem.com>.
My earlier suggestion obviously wouldn't work. 15 is a bit mask, not the
length :)
Either nextLevelArray should be initialized with length 16, or a Min
could be taken:
int level = Math.Min(slice[upto] & 15,
DocumentsWriter.nextLevelArray.Length);
-Will
-----Original Message-----
From: Will Palmeri [mailto:will@imeem.com]
Sent: Friday, June 27, 2008 6:23 PM
To: lucene-net-dev@incubator.apache.org
Subject: IndexOutOfRangeException when adding Document to index
I just checked out the 2.3.1 code from svn and built a fresh dll to try
out with our existing codebase. (We had been using 2.0.5). A couple of
my indexes are working just fine, but for one of them when I get to
around 1000 documents, I get sporadic IndexOutOfRangeExceptions when
calling IndexWriter.AddDocument(Document)
I stepped through the code and drilled down to where the exception is
getting thrown. When processing a single term in a given field, the
method AllocSlice of the internal class ByteBlockPool in
DocumentsWriter.cs is called.
In AllocSlice, the variable level is computed by AND'ing a value from
slice array with 15 (0xf). However, in the next line 'level' is used to
index into DocumentsWriter.nextLevelArray. At runtime, this array was
only of length 10, so occasionally when level was >= 10, the
IndexOutOfRangeException would be thrown.
On line 3410, nextLevelArray is initialized:
internal static readonly int[] nextLevelArray = new int[]{1, 2, 3, 4, 5,
6, 7, 8, 9, 9};
I'm not familiar with the codebase, so I have no idea what this code
does specifically. But it seems to me that line 3510 should be:
int level = slice[upto] & DocumentsWriter.nextLevelArray.Length
I'm looking forward to using 2.3.1, hope this helps!
-Will
imeem.com